--- name: integrated-crm-app version: 1.1.0 description: Complete integrated CRM application combining customer management, contract management, opportunity management, financial management, appbase foundation, and RBAC security modules into a unified web application. trigger_conditions: - User requests to create an integrated CRM system - Task involves combining multiple business modules into a single application - Need unified interface for customer, contract, opportunity, and financial management - Require RBAC security and appbase foundation integration --- # Integrated CRM Application ## Overview This skill provides a complete, production-ready integrated CRM application that seamlessly combines eight core modules into a unified web interface: 1. **Customer Management** - Comprehensive client lifecycle management 2. **Opportunity Management** - Sales pipeline and revenue forecasting 3. **Contract Management** - Contract lifecycle with milestone tracking 4. **Financial Management** - Order-level receivables and payments management 5. **Workflow Approval** - Cross-module approval workflow management 6. **Unified Dashboard** - Real-time business intelligence and reporting 7. **AppBase** - Foundation module for code and parameter management 8. **RBAC** - Role-based access control and multi-tenant security The application follows all established module development specifications and provides a cohesive user experience through a tab-based navigation interface. ## Core Features ### Unified Interface - **TabPanel Navigation**: Single-page application with six main tabs - **Responsive Design**: Adapts to different screen sizes and devices - **Consistent UX**: Uniform styling and interaction patterns across all modules - **Integrated Login**: Centralized authentication using RBAC module ### Module Integration Points #### Customer ↔ Opportunity Integration - Customer 360° view includes associated opportunities - Opportunity creation can reference existing customers - Handover workflows include both customer and opportunity data #### Opportunity ↔ Contract Integration - One-click contract generation from opportunities - Contract status automatically updates opportunity stage - Revenue forecasting considers both open opportunities and active contracts #### Contract ↔ Financial Integration - Automatic receivable creation from contract payment milestones - Order-level financial tracking linked to contract fulfillment - Payment completion triggers contract milestone updates #### Financial ↔ Customer Integration - Customer financial summary shows total receivables/payments - Overdue notifications sent to both sales and finance teams - Customer credit limits enforced during opportunity/contract creation #### Cross-Module Approval Integration - **Customer**: Handover approvals, critical data changes - **Opportunity**: High-value creation, stage transitions - **Contract**: Creation/modification, special terms approval - **Financial**: Large expenses, exceptional payments - **Unified Interface**: Single approval center for all modules #### Unified Dashboard Integration - **Executive View**: Aggregated KPIs across all modules - **Sales View**: Opportunity pipeline and conversion metrics - **Finance View**: Receivables, revenue, and financial health - **Customer View**: Portfolio analysis and engagement metrics - **Real-time Data**: Live aggregation from all integrated modules ### Security and Multi-tenancy - **Organization Isolation**: All data separated by org_id - **RBAC Permissions**: Fine-grained access control per module and function — see `references/rbac-permission-matrix.md` for the 17-role / 4-department matrix - **Audit Trail**: Comprehensive logging of all critical operations - **API Key Management**: Programmatic access via userapp table ## Technical Architecture ### Frontend Architecture - **Bricks Framework**: JSON-driven component system - **Main Layout**: `base.ui` with TabPanel organizing all modules - **Authentication**: `login.ui` + `login.dspy` using RBAC functions - **Module Integration**: Frame components loading individual module UIs ### Backend Architecture - **Module Loader**: `init.py` loads all six modules in correct order - **Dependency Order**: AppBase → RBAC → Business Modules - **Function Exposure**: ServerEnv exposes all required functions - **Async Design**: Proper awaitify usage for synchronous functions ### Database Architecture - **Shared Schema**: All modules use same database with org_id isolation - **Referential Integrity**: Foreign keys maintain data consistency - **Performance Optimization**: Strategic indexing on frequently queried fields - **DDL Generation**: Automated schema creation from JSON/XLSX definitions ## Directory Structure ``` integrated_crm_app/ ├── integrated_crm_app/ # Python package │ ├── __init__.py # Package marker │ └── init.py # Main module loader ├── wwwroot/ # Main application frontend │ ├── base.ui # Unified layout with TabPanel │ ├── login.ui # Centralized login form │ └── login.dspy # RBAC-integrated auth handler ├── build.sh # Build script for all modules ├── pyproject.toml # Package configuration └── README.md # Comprehensive documentation ``` ## Implementation Workflow ### Step 1: Module Preparation - Ensure all eight modules exist in `~/repos/` directory - Verify each module follows development specifications - Confirm database table definitions are complete ### Step 2: Main Application Setup - Create main application directory structure - Implement unified `init.py` module loader (8 modules) - Design `base.ui` TabPanel layout with approval and dashboard tabs - Create centralized authentication flow ### Build Integration - Implement `build.sh` script to process all eight modules - **Critical**: The build script must check for `mysql.ddl.sql` files in each module directory and merge them into `integrated_crm_app_schema.sql` - Generate DDL scripts for database schema by concatenating all module DDL files - Create symbolic links for frontend resources - Test module loading and function exposure - **Verification**: Always run the build script after any module changes to ensure integration compatibility ### Deployment Process ### Prerequisites - MariaDB/MySQL installed and running - Python 3.10+ with venv support - All eight modules present in `~/repos/` directory ### Step-by-Step Deployment 1. **Database Setup**: ```bash mysql -u hermes -p'hermes123' -e "CREATE DATABASE crm_db CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;" ``` 2. **Build Script Configuration**: - The `build.sh` script in the integrated app directory references `~/repos/*` modules directly - Verify each module path is correct in build.sh before running - Run: `cd ~/repos/integrated_crm_app && ./build.sh` 3. **Database Schema Import**: ```bash mysql -u hermes -p'hermes123' crm_db < build/integrated_crm_app_schema.sql ``` - The build script merges all module `mysql.ddl.sql` files into one schema file 4. **Dependency Installation**: - The build script creates a venv and installs dependencies - Required packages: aiohttp, aiohttp-auth, PyMySQL, jinja2, cryptography, aiomysql, bcrypt, etc. - If installation fails, install missing packages manually: `source py3/bin/activate && pip install ` 5. **Configuration** (see **Critical config.json Format Requirements** below): - Edit `conf/config.json`: Set database host, port, username, password (AES-encrypted) - Ensure `paths`, `processors`, and URL prefix follow strict format requirements - Ensure `database.dbname` matches the created database (e.g., `crm_db`) 6. **RBAC Permission Initialization** (see **RBAC Permission Setup** below): - Create `perm_config.py` defining ROLES, PERMISSION_MATRIX, and CRUD_TABLES - Run `init_permissions.py` to register roles, expand wildcards, and grant permissions - See `references/rbac-permission-matrix.md` for the 4-department role design - **Critical**: Permission cache is in-memory — app MUST be restarted after init 7. **Start Application**: ```bash source py3/bin/activate python app/integrated_crm_app.py ``` ## Common Integration Issues and Solutions ### Dependency Resolution Issues **pyproject.toml dependency names must match actual package names from setup.cfg**: - `sqlor` — NOT `sqlor-database-module` - `bricks_for_python` — NOT `bricks-framework` (package name from setup.cfg, repo is `bricks-for-python`) - `apppublic` — installed from git, NOT declared as dependency - `ahserver` — installed from git, NOT declared as dependency - `rbac` — installed from git, NOT declared as dependency When running `pip install .` on a module, pip tries to resolve ALL dependencies from PyPI. If a dependency like `sqlor-database-module` doesn't exist on PyPI, installation fails with `No matching distribution found`. The solution is to remove local-only dependencies from `pyproject.toml` `dependencies` section, since `build.sh` installs them in order beforehand. **getConfig import path**: - CORRECT: `from appPublic.jsonConfig import getConfig` - WRONG: `from appPublic.Config import getConfig` — causes `ModuleNotFoundError` - **Missing DDL files**: If the integrated schema is empty, verify that each module has a non-empty `mysql.ddl.sql` file - **Module loading order**: Ensure dependency order is correct (AppBase → RBAC → Business Modules) to avoid import errors - **Missing `__init__.py`**: Some modules (like workflow_approval, unified_dashboard) may be missing their package `__init__.py` files — create them: `mkdir -p module/module && touch module/module/__init__.py` - **Missing UI files**: The main app requires `wwwroot/login.ui`, `wwwroot/base.ui`, and `wwwroot/login.dspy`. Individual modules may need their own UI files (e.g., `wwwroot/index.ui`, `wwwroot/api/*.dspy`) - **DDL generation issues**: DECIMAL types need proper syntax `DECIMAL(18,2)`, not `DECIMAL(18,)`. Index field names must exist in the table definition - **Permission table permtype column too short**: When rbac's `permission.xlsx` generates DDL via `xls2ddl`, the `permtype` column may be defined with insufficient length (e.g., `VARCHAR(4)`). Inserting values like `'module'` (6 chars) fails with `DataError: Data too long for column 'permtype'`. Fix: Add `ALTER TABLE permission MODIFY COLUMN permtype VARCHAR(255)` after schema import in `build.sh`. The `build.sh` already includes this fix. - **Git repository conflicts**: When multiple developers work on different modules, use `git pull --rebase` to handle remote updates - **Symbolic link problems**: Exclude `wwwroot/wwwroot` symlinks via .gitignore to prevent circular references in repositories - **Build script failures**: Check that all required modules exist in `~/repos/` before running the build script - **Wrong module name in init.py**: Ensure all module names in `init.py` match the actual directory names (e.g., `financial_management` not `accounting`) - **Missing Python packages**: `aiohttp-auth` is required for authentication — install via `pip install aiohttp-auth` ## API Endpoint Audit & Fix Workflow After deployment or when adding new modules, all API endpoint files (.dspy) must be audited against the DDL schema. This is the most common source of 500 errors. ### Step 1: Identify all .dspy API files ```bash find ~/repos -path "*/wwwroot/api/*.dspy" -not -path "*/py3/*" ``` ### Step 2: For each .dspy file, verify SQL columns match DDL 1. Read the .dspy file and extract all SQL SELECT statements 2. Read the corresponding `mysql.ddl.sql` for that module 3. Run `DESCRIBE table_name` in MySQL to confirm actual schema 4. Fix any column name mismatches ### Common column mismatch patterns found: - **customers**: DDL has `customer_name`, not `contact_person`; has `customer_type`, `industry`, `customer_level`, `region` — not generic `address`-only queries - **customer_pool**: DDL uses `recycle_reason` not `reason`; `pool_status` not `status`; has `original_owner_id`, `inactive_days`, `recycled_at` - **customer_handover**: DDL uses `from_owner_id` not `from_user_id`; `to_owner_id` not `to_user_id`; `current_stage` not `status`; `handover_reason` not `reason` - **receivables**: Actual DB columns are: `id, order_id, contract_id, customer_id, receivable_amount, received_amount, due_date, status, description, org_id, created_at, updated_at`. Does NOT have `credit_period`, `sales_owner_id`, or `receivable_date` despite what DDL/schema files may say ### Step 3: Create missing API endpoints If a module's UI references an API that doesn't exist, create it: ``` module_name/wwwroot/api/{table}_list.dspy ``` Standard template (uses global variables, NOT imports): ```python #!/usr/bin/env python3 import json result = {'success': False, 'rows': [], 'total': 0} try: dbname = get_module_dbname('module_name') ns = { 'page': int(params_kw.get('page', 1)), 'rows': int(params_kw.get('rows', 20)), 'sort': 'created_at desc' } sql = "SELECT col1, col2, ... FROM table_name" async with DBPools().sqlorContext(dbname) as sor: data = await sor.sqlExe(sql, ns) if isinstance(data, dict): result['total'] = data.get('total', 0) result['rows'] = [dict(r) for r in data.get('rows', [])] else: result['rows'] = [dict(r) for r in (data or [])] result['total'] = len(result['rows']) result['success'] = True except Exception as e: result['error'] = str(e) return json.dumps(result, ensure_ascii=False, default=str) ``` CRITICAL: .dspy file conventions: - DO NOT import DBPools, ServerEnv, etc. — they are pre-registered as globals - DO NOT do `env = ServerEnv(); env.get_module_dbname(...)` — use `get_module_dbname('mod')` directly - DO NOT use manual LIMIT/OFFSET — use sqlExe's built-in pagination via `ns={'page': N, 'rows': N}` - DO NOT use `print()` — always `return json.dumps(...)` — returning None causes 500 error - POST/GET params accessed via `params_kw` (DictObject) ### Step 4: Fix .ui file URL references Replace broken `{{entire_url('xxx.json')}}` patterns with direct paths: ```json "url": "/main/module_name/api/endpoint_list.dspy" ``` ### Step 5: Verify all endpoints ```bash # Login first curl -s -c /tmp/crm_cookies.txt "http://localhost:8080/main/login.dspy?username=admin&password=admin123" # Test each API curl -s -b /tmp/crm_cookies.txt "http://localhost:8080/main/module_name/api/endpoint_list.dspy?page=1&rows=20" ``` ### Step 6: Comprehensive Health Check Run all endpoints in a loop to verify: ```bash echo "=== UI Pages ===" && for path in \ "main/base.ui" \ "main/customer_management/base.ui" \ "main/opportunity_management/opportunity_management.ui" \ "main/contract_management/contract_list.ui" \ "main/financial_management/index.ui" \ "main/workflow_approval/approval_task_detail.ui" \ "main/unified_dashboard/mobile_dashboard.ui" \ "main/rbac/admin_menu.ui"; do code=$(curl -s -b /tmp/crm_cookies.txt -o /dev/null -w "%{http_code}" "http://localhost:8080/$path") echo "$code $path" done && echo "" && echo "=== API Endpoints ===" && for path in \ "main/customer_management/api/customers_list.dspy?page=1&rows=20" \ "main/opportunity_management/api/opportunities_list.dspy?page=1&rows=20" \ "main/contract_management/api/contract_list.dspy?page=1&rows=20" \ "main/financial_management/api/receivables.dspy?page=1&rows=20"; do code=$(curl -s -b /tmp/crm_cookies.txt -o /dev/null -w "%{http_code}" "http://localhost:8080/$path") echo "$code $path" done ``` ### Step 7: Insert Test Data Use Python with aiomysql to insert test data across modules: ```python import asyncio from aiomysql import create_pool from appPublic.uniqueID import getID async def insert_test_data(): pool = await create_pool(host='localhost', port=3306, user='hermes', password='hermes123', db='crm_db', charset='utf8mb4') async with pool.acquire() as conn: async with conn.cursor() as cur: # Insert sales stages, customers, opportunities, contracts, etc. cust_id = getID() await cur.execute("INSERT INTO customers (...) VALUES (...)", (...)) await conn.commit() pool.close() await pool.wait_closed() asyncio.run(insert_test_data()) ``` ### Step 8: Fix Missing API Endpoints When UI pages reference endpoints that don't exist: 1. Create `module_name/wwwroot/api/` directory if missing 2. Create `{table}_list.dspy` files following the standard template 3. Verify columns match DDL schema using `DESCRIBE table_name` 4. Test endpoint returns valid JSON with `success: true` ### Step 9: Fix Missing UI Sub-pages When index pages reference .ui files that don't exist (causing 500 errors): 1. Create stub pages in `module_name/wwwroot/` for missing files 2. Use standard "Feature under development" template 3. Verify all referenced .ui files return 200 ## Critical config.json Format Requirements The `conf/config.json` file has strict format requirements that cause silent startup failures if incorrect: ### 1. `paths` must be list of `[filepath, url_prefix]` tuples (NOT strings) ```json "paths": [ ["$[workdir]$/wwwroot", "/main"] ] ``` **NOT**: `["/main/login.ui"]` or `["$[workdir]$/wwwroot"]` ### 2. URL prefix cannot end with `/` - CORRECT: `"/main"` - WRONG: `"/main/"` — causes `AssertionError: prefix` in aiohttp ### 3. Database `kwargs` must use `password` NOT `passwd` aiomysql driver expects `password`, not PyMySQL's `passwd`: ```json "kwargs": { "host": "localhost", "port": 3306, "user": "hermes", "password": "", "db": "crm_db", "charset": "utf8mb4" } ``` ### 4. Password must be AES-encrypted when `password_key` is set The `password_key` field triggers automatic AES decryption of the database password. Use: ```python from appPublic.aes import aes_encode_b64 key = config.password_key encrypted = aes_encode_b64(key, 'plaintext_password') ``` ### 5. `processors` must be list of lists (NOT a dict) ```json "processors": [ [".ui", "bui"], [".dspy", "dspy"] ] ``` **NOT**: `{".ui": "bui", ".dspy": "dspy"}` — causes `ValueError: too many values to unpack` ### 6. RBAC PUBLIC_PATHS for login page The RBAC `check_perm.py` must allow login paths without authentication: ```python PUBLIC_PATHS = ['/main/login.ui', '/main/login.dspy'] async def objcheckperm(obj, request, userid, path): if path in PUBLIC_PATHS: return True # ... rest of permission check ``` ### 7. sqlor-database-module time comparison rules Never use `NOW()` or MySQL-specific functions in SQL. Compute timestamps in Python: ```python from datetime import datetime now = datetime.now().strftime('%Y-%m-%d %H:%M:%S') tasks = await self.db.sqlExe( "SELECT ... WHERE due_at < ${now}$", {'now': now}, limit=10 ) ``` **NOT**: `where={'org_id': org_id, 'status': 'pending', 'due_at < NOW()'}` — invalid Python dict syntax AND DB-specific ### 8. .dspy file return convention (CRITICAL) .dspy files are wrapped as `async def myfunc(request, **ns)` by the processor. They MUST `return` a string, NOT `print()`: ```python # CORRECT: return json.dumps(result, ensure_ascii=False) # WRONG: print(json.dumps(result)) # Returns None → 500 error ``` POST/GET parameters are accessed via `params_kw` (a DictObject), NOT as function arguments: ```python username = params_kw.get('username', '') password = params_kw.get('password', '') ``` ### 9. Login session management After successful password verification, use `user_login()` to set the session: ```python from ahserver.auth_api import user_login await user_login(request, user.id) ``` ### 10. Bricks template escaping in .ui files Jinja2 processes `.ui` files. Bricks runtime variables like `{{params.row.id}}` must be escaped: ```json "contract_id": "{% raw %}{{params.row.id}}{% endraw %}" ``` **NOT**: `"contract_id": "{{params.row.id}}"` — causes `UndefinedError: 'params' is undefined` ### 11. CRUD widget URL paths CRUD widgets in .ui files should use direct URL paths, NOT `{{entire_url(...)}}` with undefined variables: ```json "url": "/main/customer_management/api/customers_list.dspy" ``` **NOT**: `"url": "{{entire_url(customers_list)}}` — causes `UndefinedError: 'customers_list' is undefined` ### 12. Bricks framework files must be present Copy bricks framework assets to `wwwroot/bricks/`: ```bash cp ~/repos/bricks/bricks/*.tmpl wwwroot/bricks/ cp ~/repos/bricks/bricks/*.js wwwroot/bricks/ cp -r ~/repos/bricks/bricks/css wwwroot/bricks/ cp -r ~/repos/bricks/bricks/3parties wwwroot/bricks/ ``` Also add `.tmpl` processor to config.json: ```json "processors": [ [".ui", "bui"], [".dspy", "dspy"], [".tmpl", "tmpl"] ] ``` ### 13. RBAC wildcard permission support The RBAC permission check only does exact matching by default. Add wildcard (`/*`) support to `rbac/userperm.py`: ```python def check_roles_path(self, roles, path): for role in roles: paths = self.rp_caches.get(role) if not paths: continue if path in paths: return True for p in paths: if p.endswith('/*'): prefix = p[:-2] if path.startswith(prefix + '/') or path == prefix: return True return False ``` This allows permissions like `/main/*` to match all sub-paths. ### Step 4: Testing and Validation - Verify all eight modules load without conflicts - Test cross-module data integration including approval workflows - Validate dashboard data aggregation across all modules - Confirm mobile-responsive design on different devices ### Usage Instructions ### RBAC Permission Setup The integrated CRM uses a 4-department role model (Sales, Marketing, Operations, Finance) with 17 roles total. Permission initialization requires two files: **perm_config.py** — Defines three structures: - `ROLES`: Dict of role_id → (display_name, description) - `PERMISSION_MATRIX`: Dict of section_name → {URL_pattern: [role_ids]} - `CRUD_TABLES`: Dict of module_name → [table_names] **init_permissions.py** — Executes 6 steps: 1. Load perm_config.py 2. Expand wildcards by scanning wwwroot directory 3. Connect to database via sqlor DBPools 4. Create/lookup roles (matches by name first, then ID) 5. Register permissions with dual-path (canonical + /main prefix) 6. Register CRUD API permissions + sync admin_superuser to customer-org roles **Deployment workflow:** ```bash cd ~/repos/integrated_crm_app source py3/bin/activate python app/init_permissions.py # MUST restart app after init — RBAC caches permissions in memory pkill -f integrated_crm_app.py nohup python app/integrated_crm_app.py --port 8080 & ``` **See** `references/rbac-permission-matrix.md` for the complete 17-role / 4-department permission matrix. ### Installation 1. Place all eight modules in `~/repos/` directory 2. Run `./build.sh` to generate database schemas and links 3. Execute DDL scripts to create database tables 4. **Run `init_permissions.py` to initialize RBAC roles and permissions** (see RBAC Permission Setup above) 5. Start AhServer with the integrated application ### Navigation - **Login**: Access `/main/login.ui` for authentication - **Main Interface**: Redirects to `/main/base.ui` after login - **Module Switching**: Use TabPanel to navigate between modules - **System Admin**: Last tab contains RBAC and AppBase management ### Customization Points - **UI Layout**: Modify `base.ui` to rearrange or add tabs - **Authentication**: Extend `login.dspy` for additional auth methods - **Business Logic**: Add cross-module validation in individual modules - **Reporting**: Create new reports using combined data sources ## Verification Checklist - [x] All eight modules load correctly in dependency order - [x] Unified TabPanel interface displays all modules - [x] Centralized authentication works with RBAC - [x] **RBAC roles and permissions initialized** (run init_permissions.py) - [x] **Permission cache refreshed** (app restarted after init) - [x] Cross-module data relationships function properly - [x] Organization-based data isolation enforced - [x] Responsive design works on mobile/desktop - [x] Build script processes all module types (JSON/XLSX) - [x] Production-ready code with proper error handling - [x] Complete documentation in README.md ## Extension Opportunities ### Advanced Features - **Workflow Automation**: Cross-module approval workflows - **Advanced Analytics**: Unified dashboards across all modules - **Mobile App**: Native mobile interface using same backend - **API Gateway**: RESTful API layer for external integration ### Integration Scenarios - **ERP Integration**: Connect with external accounting systems - **Marketing Automation**: Link with email/campaign platforms - **Document Management**: Integrate with file storage services - **Payment Gateways**: Connect with online payment processors This integrated CRM application provides a solid foundation for enterprise customer relationship management with full extensibility and customization capabilities.