yumoqing/pipeline_core

Fork 0

yumoqing 1abaa4ea2e feat: 技能库(skills_library 213个Hermes技能)+skill_pack技能集安装能力+ocai-h5-dev技能集(37个应用/模块开发技能)

2026-08-16 15:17:43 +08:00

25 KiB

Raw Blame History

name

version

description

trigger_conditions

integrated-crm-app

1.1.0

Complete integrated CRM application combining customer management, contract management, opportunity management, financial management, appbase foundation, and RBAC security modules into a unified web application.

User requests to create an integrated CRM system

Task involves combining multiple business modules into a single application

Need unified interface for customer, contract, opportunity, and financial management

Require RBAC security and appbase foundation integration

Integrated CRM Application

Overview

This skill provides a complete, production-ready integrated CRM application that seamlessly combines eight core modules into a unified web interface:

Customer Management - Comprehensive client lifecycle management
Opportunity Management - Sales pipeline and revenue forecasting
Contract Management - Contract lifecycle with milestone tracking
Financial Management - Order-level receivables and payments management
Workflow Approval - Cross-module approval workflow management
Unified Dashboard - Real-time business intelligence and reporting
AppBase - Foundation module for code and parameter management
RBAC - Role-based access control and multi-tenant security

The application follows all established module development specifications and provides a cohesive user experience through a tab-based navigation interface.

Core Features

Unified Interface

TabPanel Navigation: Single-page application with six main tabs
Responsive Design: Adapts to different screen sizes and devices
Consistent UX: Uniform styling and interaction patterns across all modules
Integrated Login: Centralized authentication using RBAC module

Module Integration Points

Customer ↔ Opportunity Integration

Customer 360° view includes associated opportunities
Opportunity creation can reference existing customers
Handover workflows include both customer and opportunity data

Opportunity ↔ Contract Integration

One-click contract generation from opportunities
Contract status automatically updates opportunity stage
Revenue forecasting considers both open opportunities and active contracts

Contract ↔ Financial Integration

Automatic receivable creation from contract payment milestones
Order-level financial tracking linked to contract fulfillment
Payment completion triggers contract milestone updates

Financial ↔ Customer Integration

Customer financial summary shows total receivables/payments
Overdue notifications sent to both sales and finance teams
Customer credit limits enforced during opportunity/contract creation

Cross-Module Approval Integration

Customer: Handover approvals, critical data changes
Opportunity: High-value creation, stage transitions
Contract: Creation/modification, special terms approval
Financial: Large expenses, exceptional payments
Unified Interface: Single approval center for all modules

Unified Dashboard Integration

Executive View: Aggregated KPIs across all modules
Sales View: Opportunity pipeline and conversion metrics
Finance View: Receivables, revenue, and financial health
Customer View: Portfolio analysis and engagement metrics
Real-time Data: Live aggregation from all integrated modules

Security and Multi-tenancy

Organization Isolation: All data separated by org_id
RBAC Permissions: Fine-grained access control per module and function — see references/rbac-permission-matrix.md for the 17-role / 4-department matrix
Audit Trail: Comprehensive logging of all critical operations
API Key Management: Programmatic access via userapp table

Technical Architecture

Frontend Architecture

Bricks Framework: JSON-driven component system
Main Layout: base.ui with TabPanel organizing all modules
Authentication: login.ui + login.dspy using RBAC functions
Module Integration: Frame components loading individual module UIs

Backend Architecture

Module Loader: init.py loads all six modules in correct order
Dependency Order: AppBase → RBAC → Business Modules
Function Exposure: ServerEnv exposes all required functions
Async Design: Proper awaitify usage for synchronous functions

Database Architecture

Shared Schema: All modules use same database with org_id isolation
Referential Integrity: Foreign keys maintain data consistency
Performance Optimization: Strategic indexing on frequently queried fields
DDL Generation: Automated schema creation from JSON/XLSX definitions

Directory Structure

integrated_crm_app/
├── integrated_crm_app/         # Python package
│   ├── __init__.py            # Package marker
│   └── init.py                # Main module loader
├── wwwroot/                   # Main application frontend
│   ├── base.ui               # Unified layout with TabPanel
│   ├── login.ui              # Centralized login form  
│   └── login.dspy            # RBAC-integrated auth handler
├── build.sh                  # Build script for all modules
├── pyproject.toml            # Package configuration
└── README.md                 # Comprehensive documentation

Implementation Workflow

Step 1: Module Preparation

Ensure all eight modules exist in ~/repos/ directory
Verify each module follows development specifications
Confirm database table definitions are complete

Step 2: Main Application Setup

Create main application directory structure
Implement unified init.py module loader (8 modules)
Design base.ui TabPanel layout with approval and dashboard tabs
Create centralized authentication flow

Build Integration

Implement build.sh script to process all eight modules
Critical: The build script must check for mysql.ddl.sql files in each module directory and merge them into integrated_crm_app_schema.sql
Generate DDL scripts for database schema by concatenating all module DDL files
Create symbolic links for frontend resources
Test module loading and function exposure
Verification: Always run the build script after any module changes to ensure integration compatibility

Deployment Process

Prerequisites

MariaDB/MySQL installed and running
Python 3.10+ with venv support
All eight modules present in ~/repos/ directory

Step-by-Step Deployment

Database Setup:

mysql -u hermes -p'hermes123' -e "CREATE DATABASE crm_db CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;"

Build Script Configuration:
- The build.sh script in the integrated app directory references ~/repos/* modules directly
- Verify each module path is correct in build.sh before running
- Run: cd ~/repos/integrated_crm_app && ./build.sh
Database Schema Import:
```
mysql -u hermes -p'hermes123' crm_db < build/integrated_crm_app_schema.sql
```
- The build script merges all module mysql.ddl.sql files into one schema file
Dependency Installation:
- The build script creates a venv and installs dependencies
- Required packages: aiohttp, aiohttp-auth, PyMySQL, jinja2, cryptography, aiomysql, bcrypt, etc.
- If installation fails, install missing packages manually: source py3/bin/activate && pip install <package>
Configuration (see Critical config.json Format Requirements below):
- Edit conf/config.json: Set database host, port, username, password (AES-encrypted)
- Ensure paths, processors, and URL prefix follow strict format requirements
- Ensure database.dbname matches the created database (e.g., crm_db)
RBAC Permission Initialization (see RBAC Permission Setup below):
- Create perm_config.py defining ROLES, PERMISSION_MATRIX, and CRUD_TABLES
- Run init_permissions.py to register roles, expand wildcards, and grant permissions
- See references/rbac-permission-matrix.md for the 4-department role design
- Critical: Permission cache is in-memory — app MUST be restarted after init

Start Application:

source py3/bin/activate
python app/integrated_crm_app.py

Common Integration Issues and Solutions

Dependency Resolution Issues

pyproject.toml dependency names must match actual package names from setup.cfg:

sqlor — NOT sqlor-database-module
bricks_for_python — NOT bricks-framework (package name from setup.cfg, repo is bricks-for-python)
apppublic — installed from git, NOT declared as dependency
ahserver — installed from git, NOT declared as dependency
rbac — installed from git, NOT declared as dependency

When running pip install . on a module, pip tries to resolve ALL dependencies from PyPI. If a dependency like sqlor-database-module doesn't exist on PyPI, installation fails with No matching distribution found. The solution is to remove local-only dependencies from pyproject.toml dependencies section, since build.sh installs them in order beforehand.

getConfig import path:

CORRECT: from appPublic.jsonConfig import getConfig
WRONG: from appPublic.Config import getConfig — causes ModuleNotFoundError
Missing DDL files: If the integrated schema is empty, verify that each module has a non-empty mysql.ddl.sql file
Module loading order: Ensure dependency order is correct (AppBase → RBAC → Business Modules) to avoid import errors
Missing __init__.py: Some modules (like workflow_approval, unified_dashboard) may be missing their package __init__.py files — create them: mkdir -p module/module && touch module/module/__init__.py
Missing UI files: The main app requires wwwroot/login.ui, wwwroot/base.ui, and wwwroot/login.dspy. Individual modules may need their own UI files (e.g., wwwroot/index.ui, wwwroot/api/*.dspy)
DDL generation issues: DECIMAL types need proper syntax DECIMAL(18,2), not DECIMAL(18,). Index field names must exist in the table definition
Permission table permtype column too short: When rbac's permission.xlsx generates DDL via xls2ddl, the permtype column may be defined with insufficient length (e.g., VARCHAR(4)). Inserting values like 'module' (6 chars) fails with DataError: Data too long for column 'permtype'. Fix: Add ALTER TABLE permission MODIFY COLUMN permtype VARCHAR(255) after schema import in build.sh. The build.sh already includes this fix.
Git repository conflicts: When multiple developers work on different modules, use git pull --rebase to handle remote updates
Symbolic link problems: Exclude wwwroot/wwwroot symlinks via .gitignore to prevent circular references in repositories
Build script failures: Check that all required modules exist in ~/repos/ before running the build script
Wrong module name in init.py: Ensure all module names in init.py match the actual directory names (e.g., financial_management not accounting)
Missing Python packages: aiohttp-auth is required for authentication — install via pip install aiohttp-auth

API Endpoint Audit & Fix Workflow

After deployment or when adding new modules, all API endpoint files (.dspy) must be audited against the DDL schema. This is the most common source of 500 errors.

Step 1: Identify all .dspy API files

find ~/repos -path "*/wwwroot/api/*.dspy" -not -path "*/py3/*"

Step 2: For each .dspy file, verify SQL columns match DDL

Read the .dspy file and extract all SQL SELECT statements
Read the corresponding mysql.ddl.sql for that module
Run DESCRIBE table_name in MySQL to confirm actual schema
Fix any column name mismatches

Common column mismatch patterns found:

customers: DDL has customer_name, not contact_person; has customer_type, industry, customer_level, region — not generic address-only queries
customer_pool: DDL uses recycle_reason not reason; pool_status not status; has original_owner_id, inactive_days, recycled_at
customer_handover: DDL uses from_owner_id not from_user_id; to_owner_id not to_user_id; current_stage not status; handover_reason not reason
receivables: Actual DB columns are: id, order_id, contract_id, customer_id, receivable_amount, received_amount, due_date, status, description, org_id, created_at, updated_at. Does NOT have credit_period, sales_owner_id, or receivable_date despite what DDL/schema files may say

Step 3: Create missing API endpoints

If a module's UI references an API that doesn't exist, create it:

module_name/wwwroot/api/{table}_list.dspy

Standard template (uses global variables, NOT imports):

#!/usr/bin/env python3
import json
result = {'success': False, 'rows': [], 'total': 0}
try:
    dbname = get_module_dbname('module_name')
    ns = {
        'page': int(params_kw.get('page', 1)),
        'rows': int(params_kw.get('rows', 20)),
        'sort': 'created_at desc'
    }
    sql = "SELECT col1, col2, ... FROM table_name"
    async with DBPools().sqlorContext(dbname) as sor:
        data = await sor.sqlExe(sql, ns)
        if isinstance(data, dict):
            result['total'] = data.get('total', 0)
            result['rows'] = [dict(r) for r in data.get('rows', [])]
        else:
            result['rows'] = [dict(r) for r in (data or [])]
            result['total'] = len(result['rows'])
        result['success'] = True
except Exception as e:
    result['error'] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)

CRITICAL: .dspy file conventions:

DO NOT import DBPools, ServerEnv, etc. — they are pre-registered as globals
DO NOT do env = ServerEnv(); env.get_module_dbname(...) — use get_module_dbname('mod') directly
DO NOT use manual LIMIT/OFFSET — use sqlExe's built-in pagination via ns={'page': N, 'rows': N}
DO NOT use print() — always return json.dumps(...) — returning None causes 500 error
POST/GET params accessed via params_kw (DictObject)

Step 4: Fix .ui file URL references

Replace broken {{entire_url('xxx.json')}} patterns with direct paths:

"url": "/main/module_name/api/endpoint_list.dspy"

Step 5: Verify all endpoints

# Login first
curl -s -c /tmp/crm_cookies.txt "http://localhost:8080/main/login.dspy?username=admin&password=admin123"
# Test each API
curl -s -b /tmp/crm_cookies.txt "http://localhost:8080/main/module_name/api/endpoint_list.dspy?page=1&rows=20"

Step 6: Comprehensive Health Check

Run all endpoints in a loop to verify:

echo "=== UI Pages ===" && for path in \
  "main/base.ui" \
  "main/customer_management/base.ui" \
  "main/opportunity_management/opportunity_management.ui" \
  "main/contract_management/contract_list.ui" \
  "main/financial_management/index.ui" \
  "main/workflow_approval/approval_task_detail.ui" \
  "main/unified_dashboard/mobile_dashboard.ui" \
  "main/rbac/admin_menu.ui"; do
  code=$(curl -s -b /tmp/crm_cookies.txt -o /dev/null -w "%{http_code}" "http://localhost:8080/$path")
  echo "$code $path"
done && echo "" && echo "=== API Endpoints ===" && for path in \
  "main/customer_management/api/customers_list.dspy?page=1&rows=20" \
  "main/opportunity_management/api/opportunities_list.dspy?page=1&rows=20" \
  "main/contract_management/api/contract_list.dspy?page=1&rows=20" \
  "main/financial_management/api/receivables.dspy?page=1&rows=20"; do
  code=$(curl -s -b /tmp/crm_cookies.txt -o /dev/null -w "%{http_code}" "http://localhost:8080/$path")
  echo "$code $path"
done

Step 7: Insert Test Data

Use Python with aiomysql to insert test data across modules:

import asyncio
from aiomysql import create_pool
from appPublic.uniqueID import getID

async def insert_test_data():
    pool = await create_pool(host='localhost', port=3306,
        user='hermes', password='hermes123', db='crm_db', charset='utf8mb4')
    async with pool.acquire() as conn:
        async with conn.cursor() as cur:
            # Insert sales stages, customers, opportunities, contracts, etc.
            cust_id = getID()
            await cur.execute("INSERT INTO customers (...) VALUES (...)", (...))
            await conn.commit()
    pool.close()
    await pool.wait_closed()

asyncio.run(insert_test_data())

Step 8: Fix Missing API Endpoints

When UI pages reference endpoints that don't exist:

Create module_name/wwwroot/api/ directory if missing
Create {table}_list.dspy files following the standard template
Verify columns match DDL schema using DESCRIBE table_name
Test endpoint returns valid JSON with success: true

Step 9: Fix Missing UI Sub-pages

When index pages reference .ui files that don't exist (causing 500 errors):

Create stub pages in module_name/wwwroot/ for missing files
Use standard "Feature under development" template
Verify all referenced .ui files return 200

Critical config.json Format Requirements

The conf/config.json file has strict format requirements that cause silent startup failures if incorrect:

1. `paths` must be list of `[filepath, url_prefix]` tuples (NOT strings)

"paths": [
  ["$[workdir]$/wwwroot", "/main"]
]

NOT: ["/main/login.ui"] or ["$[workdir]$/wwwroot"]

2. URL prefix cannot end with `/`

CORRECT: "/main"
WRONG: "/main/" — causes AssertionError: prefix in aiohttp

3. Database `kwargs` must use `password` NOT `passwd`

aiomysql driver expects password, not PyMySQL's passwd:

"kwargs": {
  "host": "localhost",
  "port": 3306,
  "user": "hermes",
  "password": "<encrypted>",
  "db": "crm_db",
  "charset": "utf8mb4"
}

4. Password must be AES-encrypted when `password_key` is set

The password_key field triggers automatic AES decryption of the database password. Use:

from appPublic.aes import aes_encode_b64
key = config.password_key
encrypted = aes_encode_b64(key, 'plaintext_password')

5. `processors` must be list of lists (NOT a dict)

"processors": [
  [".ui", "bui"],
  [".dspy", "dspy"]
]

NOT: {".ui": "bui", ".dspy": "dspy"} — causes ValueError: too many values to unpack

The RBAC check_perm.py must allow login paths without authentication:

PUBLIC_PATHS = ['/main/login.ui', '/main/login.dspy']

async def objcheckperm(obj, request, userid, path):
    if path in PUBLIC_PATHS:
        return True
    # ... rest of permission check

7. sqlor-database-module time comparison rules

Never use NOW() or MySQL-specific functions in SQL. Compute timestamps in Python:

from datetime import datetime
now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
tasks = await self.db.sqlExe(
    "SELECT ... WHERE due_at < ${now}$",
    {'now': now},
    limit=10
)

NOT: where={'org_id': org_id, 'status': 'pending', 'due_at < NOW()'} — invalid Python dict syntax AND DB-specific

8. .dspy file return convention (CRITICAL)

.dspy files are wrapped as async def myfunc(request, **ns) by the processor. They MUST return a string, NOT print():

# CORRECT:
return json.dumps(result, ensure_ascii=False)

# WRONG:
print(json.dumps(result))  # Returns None → 500 error

POST/GET parameters are accessed via params_kw (a DictObject), NOT as function arguments:

username = params_kw.get('username', '')
password = params_kw.get('password', '')

After successful password verification, use user_login() to set the session:

from ahserver.auth_api import user_login
await user_login(request, user.id)

10. Bricks template escaping in .ui files

Jinja2 processes .ui files. Bricks runtime variables like {{params.row.id}} must be escaped:

"contract_id": "{% raw %}{{params.row.id}}{% endraw %}"

NOT: "contract_id": "{{params.row.id}}" — causes UndefinedError: 'params' is undefined

CRUD widgets in .ui files should use direct URL paths, NOT {{entire_url(...)}} with undefined variables:

"url": "/main/customer_management/api/customers_list.dspy"

NOT: "url": "{{entire_url(customers_list)}} — causes UndefinedError: 'customers_list' is undefined

12. Bricks framework files must be present

Copy bricks framework assets to wwwroot/bricks/:

cp ~/repos/bricks/bricks/*.tmpl wwwroot/bricks/
cp ~/repos/bricks/bricks/*.js wwwroot/bricks/
cp -r ~/repos/bricks/bricks/css wwwroot/bricks/
cp -r ~/repos/bricks/bricks/3parties wwwroot/bricks/

Also add .tmpl processor to config.json:

"processors": [
  [".ui", "bui"],
  [".dspy", "dspy"],
  [".tmpl", "tmpl"]
]

13. RBAC wildcard permission support

The RBAC permission check only does exact matching by default. Add wildcard (/*) support to rbac/userperm.py:

def check_roles_path(self, roles, path):
    for role in roles:
        paths = self.rp_caches.get(role)
        if not paths: continue
        if path in paths: return True
        for p in paths:
            if p.endswith('/*'):
                prefix = p[:-2]
                if path.startswith(prefix + '/') or path == prefix:
                    return True
    return False

This allows permissions like /main/* to match all sub-paths.

Step 4: Testing and Validation

Verify all eight modules load without conflicts
Test cross-module data integration including approval workflows
Validate dashboard data aggregation across all modules
Confirm mobile-responsive design on different devices

Usage Instructions

RBAC Permission Setup

The integrated CRM uses a 4-department role model (Sales, Marketing, Operations, Finance) with 17 roles total. Permission initialization requires two files:

perm_config.py — Defines three structures:

ROLES: Dict of role_id → (display_name, description)
PERMISSION_MATRIX: Dict of section_name → {URL_pattern: [role_ids]}
CRUD_TABLES: Dict of module_name → [table_names]

init_permissions.py — Executes 6 steps:

Load perm_config.py
Expand wildcards by scanning wwwroot directory
Connect to database via sqlor DBPools
Create/lookup roles (matches by name first, then ID)
Register permissions with dual-path (canonical + /main prefix)
Register CRUD API permissions + sync admin_superuser to customer-org roles

Deployment workflow:

cd ~/repos/integrated_crm_app
source py3/bin/activate
python app/init_permissions.py
# MUST restart app after init — RBAC caches permissions in memory
pkill -f integrated_crm_app.py
nohup python app/integrated_crm_app.py --port 8080 &

See references/rbac-permission-matrix.md for the complete 17-role / 4-department permission matrix.

Installation

Place all eight modules in ~/repos/ directory
Run ./build.sh to generate database schemas and links
Execute DDL scripts to create database tables
Run init_permissions.py to initialize RBAC roles and permissions (see RBAC Permission Setup above)
Start AhServer with the integrated application

Login: Access /main/login.ui for authentication
Main Interface: Redirects to /main/base.ui after login
Module Switching: Use TabPanel to navigate between modules
System Admin: Last tab contains RBAC and AppBase management

Customization Points

UI Layout: Modify base.ui to rearrange or add tabs
Authentication: Extend login.dspy for additional auth methods
Business Logic: Add cross-module validation in individual modules
Reporting: Create new reports using combined data sources

Verification Checklist

All eight modules load correctly in dependency order
Unified TabPanel interface displays all modules
Centralized authentication works with RBAC
RBAC roles and permissions initialized (run init_permissions.py)
Permission cache refreshed (app restarted after init)
Cross-module data relationships function properly
Organization-based data isolation enforced
Responsive design works on mobile/desktop
Build script processes all module types (JSON/XLSX)
Production-ready code with proper error handling
Complete documentation in README.md

Extension Opportunities

Advanced Features

Workflow Automation: Cross-module approval workflows
Advanced Analytics: Unified dashboards across all modules
Mobile App: Native mobile interface using same backend
API Gateway: RESTful API layer for external integration

Integration Scenarios

ERP Integration: Connect with external accounting systems
Marketing Automation: Link with email/campaign platforms
Document Management: Integrate with file storage services
Payment Gateways: Connect with online payment processors

This integrated CRM application provides a solid foundation for enterprise customer relationship management with full extensibility and customization capabilities.

25 KiB Raw Blame History