pipeline_core/skills_library/all/hermes-app-deploy/SKILL.md

name	title	description
hermes-app-deploy	Hermes Application Automated Deployment	Single-script interactive deployment pattern for multi-module Hermes applications

Hermes Application Automated Deployment

Overview

Deploy multi-module Hermes applications with a single interactive build.sh script that handles module cloning, database setup, configuration generation, and permission initialization in one step.

Architecture

All modules (reference + business) share a single database. rbac tables and business tables coexist — no separate rbac database.

Module Classification

Type	Modules	Rules
Reference	apppublic, sqlor, ahserver, appbase, rbac	Never modify
Business	Per-application (e.g., customer_management, etc.)	Follow module-development-spec

build.sh Pattern

The build script follows 10 sequential steps:

1. Create directories (pkgs/, logs/, files/, wwwroot/)
2. Setup Python venv
3. Install core deps (apppublic, sqlor, ahserver, bricks_for_python)
4. Clone ALL modules to pkgs/ (reference + business)
5. Generate DDL (xls2ddl/json2ddl) and CRUD UI (xls2ui)
6. Create wwwroot symlinks
7. Interactive DB configuration (prompts user)
8. Create DB, import schema, generate config.json (encrypted password)
9. Run permission initialization
10. Generate start.sh / stop.sh

Key Design Decisions

1. Clone modules to pkgs/ (not ~/repos) build.sh clones all modules to pkgs/ — self-contained deployment, no dependency on external ~/repos structure.

2. Interactive DB prompts Instead of hardcoded or external config files, prompt the user:

read -p "  MySQL host [localhost]: " DB_HOST
DB_HOST=${DB_HOST:-localhost}
read -sp "  MySQL admin password: " DB_ADMIN_PASS

3. AES encrypt password in config.json Use apppublic's aes_encode_b64() to encrypt the DB password before writing to config.json. The password_key field in config is used for decryption at runtime.

4. Single database for all modules rbac, appbase, and all business modules share ONE database. Do NOT create separate databases.

5. DDL generation — filter exception output json2ddl/xls2ddl prints Exception: lines to stdout when model files are malformed (e.g., missing summary field, wrong summary format). These MUST be filtered before writing to the combined schema SQL file, otherwise they corrupt the SQL and cause mysql import errors:

TEMP_DDL=$(mktemp)
json2ddl mysql . > "$TEMP_DDL" 2>/dev/null || true
grep -v "^Exception:" "$TEMP_DDL" > "$MOD_DIR/mysql.ddl.sql"

6. rbac xlsx-generated DDL may have undersized column lengths rbac models use .xlsx (not JSON) for table definitions. The xls2ddl tool may generate columns with insufficient VARCHAR lengths (e.g., permtype VARCHAR(4) instead of VARCHAR(255)). After importing the schema, always run:

mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "ALTER TABLE permission MODIFY COLUMN permtype VARCHAR(255);"

This is a known issue — the xlsx source has short field lengths for some columns. The ALTER TABLE must come AFTER the schema import, not before.

Model JSON Format (Critical Prerequisite)

All model JSON files in models/ must have summary as a list (NOT dict), with primary field:

{
    "summary": [{"name": "table_name", "title": "说明", "primary": "id", "catelog": "entity"}],
    "fields": [{"name": "id", "type": "str", "length": 32, ...}]
}

Common issues that cause DDL generation failure:

• summary is a dict instead of list → fix: convert to [{"name": ..., "title": ..., "primary": "id"}]
• summary missing primary key → add "primary": "id"
• Fields missing id field → every table must have an id field as primary key

Permission Configuration

perm_config.py Structure

# Role definitions (use underscores, NOT dots)
ROLES = [
    {'id': 'sales_manager', 'name': '销售经理', 'desc': '...'},
    {'id': 'admin_superuser', 'name': '超级用户', 'desc': '...'},
]

# Permission matrix
PERMISSION_MATRIX = {
    'customer_management': {
        '/customer_management/**': ['sales_manager', 'admin_superuser'],
    },
}

# CRUD table paths
CRUD_TABLES = {
    'customer_management': ['customers', 'customer_pool'],
}

Critical Rules

1. Role IDs use underscores, not dots The frontend displays roles as orgtype.role_name (e.g., sales.manager). If the role ID also uses dots, they collide. Always use sales_manager as ID.

2. Convention role IDs are FIXED strings rbac's userperm.py hardcodes checks:

if r.id == 'anonymous': k = 'anonymous'
elif r.id == 'any': k = 'any'
elif r.id == 'logined': k = 'logined'

These MUST use exact string IDs — never generate with getID().

3. Single-owner vs multi-org Most CRM applications are single-owner (one company's internal system). Don't create multi-org type structures (sales/customer/finance orgtypes) unless explicitly required.

Permission Initialization Flow

async def init_permissions_from_config(dbname):
    # 1. Create convention roles with FIXED IDs
    for fixed_id in ['any', 'logined']:
        await ensure_role(sor, fixed_id, fixed_id)
    
    # 2. Create defined roles from perm_config.py
    for role in ROLES:
        await ensure_role(sor, role['id'], role['name'])
    
    # 3. Register paths from PERMISSION_MATRIX
    for module, paths in PERMISSION_MATRIX.items():
        for path_pattern, role_list in paths.items():
            permid = await ensure_permission(sor, path_pattern)
            for role_name in role_list:
                await grant_permission(sor, role_ids[role_name], permid)
    
    # 4. Register CRUD paths
    for module, tables in CRUD_TABLES.items():
        for table in tables:
            crud_path = f'/{module}/{table}/'
            permid = await ensure_permission(sor, crud_path)

ensure_role: Name-based matching for existing roles

When roles are pre-created (e.g., during org setup), they get UUID IDs but with recognizable names. The ensure_role function must match by name first to reuse existing roles instead of creating duplicates:

async def ensure_role(sor, roleid, name, desc=''):
    # Try matching by name first (handles both Chinese and English names)
    for match_name in [name, roleid]:
        recs = await sor.R('role', {'name': match_name})
        if recs:
            return recs[0].id
    
    # Try matching by ID
    recs = await sor.R('role', {'id': roleid})
    if recs:
        return recs[0].id
    
    # Create new role
    await sor.C('role', {'id': roleid, 'orgtypeid': '*', 'name': name})
    return roleid

Customer-org role permission sync

Users created within an organization context get UUID-based role IDs (e.g., Eicxrx2f1jElr5OUTAB03 for admin, icKx69-9UXf60zDIll0rg for superuser) with orgtypeid=customer. These are separate from the convention string role IDs (admin, superuser).

After initializing permissions for string role IDs, you must also sync to all orgtypeid=customer roles:

# After regular permission init...
admin_super_perms = await sor.R('rolepermission', {'roleid': role_ids['admin_superuser']})
all_roles = await sor.R('role', {'orgtypeid': 'customer'})
for r in all_roles:
    if r.name in ('admin', 'superuser'):
        for g in admin_super_perms:
            await grant_permission(sor, r.id, g.permid)

Wildcard expansion: ** patterns at init time

RBAC only does exact string matching. ** patterns in perm_config.py must be expanded to actual file paths during init:

import os

def scan_wwwroot(base_dir='wwwroot'):
    """Scan all .ui/.dspy files and return normalized paths."""
    paths = set()
    for root, dirs, files in os.walk(base_dir):
        for f in files:
            if f.endswith(('.ui', '.dspy')):
                rel = os.path.relpath(os.path.join(root, f), base_dir)
                paths.add('/' + rel)
    return paths

def expand_wildcard(pattern, all_paths):
    if '**' not in pattern:
        return {pattern}
    prefix = pattern.replace('**', '').rstrip('/')
    return {p for p in all_paths if p.startswith(prefix)}

# Usage:
all_paths = scan_wwwroot()
for path_pattern, roles in role_paths.items():
    expanded = expand_wildcard(path_pattern, all_paths)
    for exact_path in expanded:
        # Register both /xxx and /main/xxx variants
        register_permission(sor, exact_path, roles)
        register_permission(sor, '/main' + exact_path, roles)

init_permissions.py: Wildcard expansion and /main prefix handling

The application's init_permissions.py must handle two critical path transformations at initialization time (since RBAC only does exact string matching):

1. ** wildcard expansion: Scan wwwroot/ for all .ui/.dspy files, then match ** patterns against actual file paths. Register the expanded exact paths to the database.

2. /main prefix duplication: URLs come in with /main prefix (e.g., /main/customer_management/...) but perm_config.py defines paths without it (e.g., /customer_management/...). Register both variants to the DB.

Example init_permissions.py workflow:

import os

async def init_permissions(dbname):
    # Step 1: Scan wwwroot for actual file paths
    all_paths = set()
    for root, dirs, files in os.walk('wwwroot'):
        for f in files:
            if f.endswith(('.ui', '.dspy')):
                rel = os.path.relpath(os.path.join(root, f), 'wwwroot')
                all_paths.add('/' + rel)
    
    # Step 2: Expand ** patterns and register with /main prefix
    for path_pattern, roles in role_paths.items():
        expanded = expand_wildcard(path_pattern, all_paths)
        for exact_path in expanded:
            permid = await ensure_permission(sor, exact_path, permtype='page')
            for role_name in roles:
                if role_name in role_ids:
                    await grant_permission(sor, role_ids[role_name], permid)
            # Also register /main variant
            main_path = '/main' + exact_path
            main_permid = await ensure_permission(sor, main_path, permtype='page')
            for role_name in roles:
                if role_name in role_ids:
                    await grant_permission(sor, role_ids[role_name], main_permid)
    
    # Step 3: Ensure convention roles exist (any, logined, anonymous)
    # Step 4: Sync customer-org roles (orgtypeid=customer) if applicable
    # Step 5: Restart app to reload permission cache

Nginx Production Deployment

Nginx configuration for HTTPS with Let's Encrypt

After deployment, set up nginx as a reverse proxy:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name crm.opencomputing.cn;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/crm.opencomputing.cn/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/crm.opencomputing.cn/privkey.pem;
    
    # Redirect root to /main/
    location = / {
        return 302 /main/;
    }
    
    # Redirect /main/ to /main/base.ui (no index.html in wwwroot)
    location = /main/ {
        return 302 /main/base.ui;
    }
    
    # Proxy all other requests to the app
    location / {
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Scheme $scheme;
        proxy_set_header X-real-ip $remote_addr;
        proxy_read_timeout 600s;
        proxy_pass http://localhost:8080/;
    }
    
    # WebSocket support
    location /wss/ {
        proxy_pass http://localhost:8080/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 86400;
    }
}

server {
    server_name crm.opencomputing.cn;
    listen 80;
    return 301 https://$host$request_uri;
}

Critical: wwwroot has no index.html

The wwwroot/ directory does NOT contain an index.html file. Accessing /main/ directly causes a 500 error ('NoneType' object is not iterable). The nginx redirect from /main/ to /main/base.ui is required.

Testing with curl through HTTPS

# Login
curl -sk -c /tmp/cookies.txt -X POST https://crm.opencomputing.cn/main/rbac/user/up_login.dspy \
  -d 'username=superadmin&password=Kyy@123456'

# Access with cookie
curl -sk -b /tmp/cookies.txt https://crm.opencomputing.cn/main/base.ui

Permission init + restart cycle

After modifying init_permissions.py or perm_config.py:

1. Stop the app: pkill -f integrated_crm_app.py
2. Run init: python app/init_permissions.py
3. Start the app: nohup python app/integrated_crm_app.py --port 8080 > /tmp/crm_app.log 2>&1 &
4. Wait 5 seconds for startup before testing

The app loads permissions into memory at startup. DB changes alone won't take effect without a restart.

rbac Login Integration

rbac provides complete login/registration in rbac/wwwroot/user/:

• login.ui / up_login.dspy — login page and handler
• register.ui / register.dspy — registration
• logout.dspy — logout
• userinfo.ui — user profile

Never create duplicate login files — just symlink rbac's wwwroot.

First User Setup

After deployment, create the initial admin:

1. Register via http://host:port/user/register.ui
2. Assign admin_superuser role:

INSERT INTO userroles (userid, roleid) VALUES ('<userid>', 'admin_superuser');

Sage App Start/Stop Scripts

Sage applications use start.sh and stop.sh in the app root directory for lifecycle management:

start.sh Pattern — Multi-Process Support

Modern Sage start.sh supports multi-process deployment:

• Reads CPU core count via nproc and launches one worker per core
• Each worker gets a unique port: base_port + worker_index (base from conf/config.json → website.port)
• All worker PIDs are written to sage.pid (one per line)
• Per-worker logs: logs/sage_worker_N.log
• Uses virtualenv Python: ./py3/bin/python app/sage.py --workdir "$WORKDIR" --port $PORT
• Checks for Redis (session storage dependency) — starts if not running: redis-server --daemonize yes

Legacy single-process start.sh simply launches one process on the configured port. Both patterns work; multi-process is for production scaling.

stop.sh Pattern — Multi-Process Support

• Reads ALL PIDs from sage.pid (one per line)
• Sends SIGTERM to each worker, waits up to 10 seconds, then SIGKILL
• Falls back to process name search (ps aux | grep "app/sage.py") if PID file missing
• Cleans up sage.pid

Password Encoding

For updating user passwords in the database:

from ahserver.globalEnv import password_encode
encoded = password_encode('plain_text_password')
# Then: sor.U('users', {'id': user_id, 'password': encoded})

Module pyproject.toml Dependencies

Business modules must declare correct dependencies in pyproject.toml:

• Use "sqlor" (NOT "sqlor-database-module")
• Use "bricks_for_python" (NOT "bricks-framework")
• Do NOT include foundation packages (ahserver, appbase, rbac, apppublic) — these are installed by build.sh
• build.sh also installs bricks_for_python via: pip install git+https://git.opencomputing.cn/yumoqing/bricks-for-python

Example:

[project]
name = "customer_management"
dependencies = [
    "sqlor",
    "bricks_for_python",
]

Pipeline App Deployment (pipeline.opencomputing.cn)

Directory Structure

~/pipeline-app/          # Source (git clone)
├── build.sh             # Creates independent venv, clones deps to pkgs/
├── start.sh / stop.sh   # Lifecycle
├── app/                 # Application code (pipeline_app.py)
├── conf/config.json     # Runtime config
├── wwwroot/             # Web files (bricks UI, dspy)
├── pkgs/                # Cloned dependencies
│   ├── appbase/         # Must be cloned (provides get_code.dspy + appcodes_kv)
│   ├── bricks/          # Frontend framework
│   └── pipeline-sdlc/   # SDLC module with wwwroot
├── pipeline_core/       # Business module (pip install .)
├── pipeline_ops/
└── py3/                 # Independent venv (NOT Sage's py3)

config.json Requirements

Must be copied from Sage and adjusted:

• password_key — MUST match Sage's key (used for AES password encrypt/decrypt)
• databases.pipeline — driver: aiosqlor, encrypted password
• databases.sage — driver: mysql, plaintext password (shared rbac/permissions)
• session — Redis config: {"storage":"redis","redis_url":"redis://localhost:6379/1"}
• website.processors — must include dspy/ui processors from Sage config

build.sh Key Steps

1. Create independent venv (python3 -m venv py3)
2. Clone ALL shared packages to pkgs/ (apppublic, sqlor, ahserver, rbac, appbase, etc.)
3. Build bricks frontend (pkgs/bricks/bricks/build.sh)
4. Symlink bricks -> pkgs/bricks/dist
5. Clone business modules to pkgs/ (pipeline-sdlc, showcase)
6. Install all modules with pip install .
7. Generate CRUD from models using xls2ui

start.sh / stop.sh

Same pattern as Sage: source py3/bin/activate && python app/pipeline_app.py -p 9090 -w .

Cold Start 500

First request after restart may return 500 due to reuse_port and event loop warmup. Retry 2-3 times.

See also: references/pipeline-permissions.md for RBAC wildcard/trailing-slash/underscore path rules.

Common Pitfalls

• pyproject.toml declares non-existent PyPI packages (sqlor-database-module, bricks-framework, ahserver) → pip install . fails
• Creating separate rbac database (wrong — use single DB)
• Using dots in role IDs (conflicts with display format)
• Using getID() for convention roles (rbac hardcodes the IDs)
• Creating duplicate login/register files (rbac already provides these)
• Hardcoding DB credentials in build.sh (use interactive prompts)
• Missing symlink for wwwroot/bricks (needed for frontend)
• .dspy files using JavaScript true/false instead of Python True/False → causes NameError at runtime. Fix: sed -i 's/: true$/: True/; s/: true,/: True,/; s/: false$/: False/; s/: false,/: False,/' *.dspy
• Permission paths in DB don't match actual URL paths → check init_permissions.py path generation vs config.json website.paths mapping
• rbac anonymous role has no permissions → login page returns 401 even for unauthenticated access
• Missing wwwroot symlinks for modules → all requests return 404 or "invalid path" errors

Multi-Process Deployment

See references/sage-multi-process.md for the complete Sage multi-process deployment pattern, including nproc-based worker scaling, per-worker logging, and nginx integration.