yumoqing/pipeline_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pipeline_core/skills_library/all/pccs-deploy/SKILL.md

12 KiB
Raw Blame History

name description version category
pccs-deploy Use when deploying pccs. Config pitfalls, build.sh patterns. 1.0.0 devops

PCCS Deployment Knowledge

Config

1. Database password MUST be AES-encrypted

sqlor auto-decrypts kwargs.password via aes_decode_b64(password_key, password). Plaintext causes binascii.Error: Incorrect padding.

Fix: aes_encode_b64(password_key, plaintext).

2. minsize/maxsize NOT in kwargs

These are pool-level params. In kwargs they cause connect() got an unexpected keyword argument 'minsize'.

Right:

{ "pccs": { "driver": "mysql", "minsize": 2, "maxsize": 10,
    "kwargs": { "host": "...", "password": "<encrypted>", "db": "pccs" } } }

3. processors MUST include [".tmpl","tmpl"]

Without it bricks UI returns 500 on /.

4. indexes MUST include "index.ui"

Without it / returns 500.

Server Access

  • SSH: ssh pccs@pccs.opencomputing.cn (NOT root)
  • App root: /d/pccs/ — build.sh lives here, copies files from git repos in pkgs/
  • Git repos: under /d/pccs/pkgs/ — pkgs/pccs/ is the main app repo (NOT /d/pccs/ itself)
  • Deploy: cd /d/pccs/pkgs/pccs && git pull origin main then sudo systemctl restart pccs OR run full bash /d/pccs/build.sh which handles all modules
  • Quick deploy (single-file): cp /d/pccs/pkgs/pccs/wwwroot/index.html /d/pccs/wwwroot/index.html && sudo systemctl restart pccs
  • Nginx: proxies all traffic (including /rbac/) to 127.0.0.1:9180 (ahserver). No pipeline proxy.

5. Static HTML pages MUST include viewport meta

The static index.html fallback page needs <meta name="viewport" content="width=device-width,initial-scale=1.0"> in <head>. Without it, mobile browsers render at desktop width. The Bricks index.ui pages include it automatically via bricks HTML template.

Pip mirror (China)

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
pip install xls2ddl -q --timeout=30 || echo "skip"

Don't install mysql-client on MariaDB

apt install mysql-client uninstalls mariadb-server. MariaDB provides mysql command.

Correct directory layout

<appdir>/
├── app/           ← entry point
├── conf/          ← config.json
├── pkgs/          ← all module repos
├── wwwroot/       ← symlinks only
├── py3/           ← venv
├── logs/ files/ scripts/
├── build.sh
└── start.sh

build.sh WORKDIR

Place build.sh at app root, compute WORKDIR via $(dirname "$0").

RBAC Permission Setup (build.sh step 7b)

After deployment, RBAC permissions MUST be initialized. The project follows the same pattern as Sage: setup_rbac_perms.sh + set_role_perm.py.

setup_rbac_perms.sh sets three tiers:

  1. Public (any): login/register/auth paths including /rbac/user/user_panel.ui and /appbase/menu.ui
  2. Logined: profile, usermenu, /index.ui
  3. Static (any + / wildcards):** /bricks/**, /imgs/**, /pcpool/**, etc.

set_role_perm.py is called per-path: ./py3/bin/python3 set_role_perm.py any /bricks/**

IMPORTANT: After running permissions setup, clear Redis RBAC cache:

redis-cli KEYS 'rbac*' | xargs redis-cli DEL
sudo systemctl restart pccs

Without this, stale cached permissions cause 401 on paths that now have any role.

Verify bricks.js is accessible (critical — the entire UI breaks without it):

curl -sk -o /dev/null -w "%{http_code}" https://pccs.opencomputing.cn/bricks/bricks.js
# must return 200

pip install: pyproject.toml REQUIRED (NOT setup.json)

Business modules (pcpool, pcc, storage_mgr, image_mgr) use setup.json (Sage format). pip install does NOT recognize setup.json — only setup.py, setup.cfg, or pyproject.toml. build.sh step 4 checks [ -f setup.py ] || [ -f setup.cfg ] || [ -f pyproject.toml ] || continue, silently skipping all modules with only setup.json.

Fix: generate pyproject.toml from setup.json for every module:

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "pcpool"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["apppublic", "sqlor", "ahserver", "appbase"]

[tool.setuptools.package-dir]
"pcpool" = "pcpool"

[tool.setuptools.packages.find]
where = ["."]

Multi-package modules (e.g., pcc with sub-plugins) need explicit package-dir entries:

[tool.setuptools.package-dir]
"pcc" = "pcc"
"pcc.k8s_plugin" = "pcc/k8s_plugin"
"pcc.slurm_plugin" = "pcc/slurm_plugin"

Nginx: proxy_set_header Host REQUIRED in location /

The location / block MUST include proxy_set_header Host $host;. Without it, nginx passes the upstream name (e.g., sage_backend) as Host, ahserver rejects DSPY paths with str(request.url)='http://sage_backend/...' invalid path.

i18n Setup (REQUIRED)

Every module needs i18n/{lang}/msg.txt (key: value per line). Merge into wwwroot/i18n/:

  1. merge_i18n.py (at app root, invoked by build.sh step 7b):

    • Reads all modules' i18n/{zh,en}/msg.txt from /d/pccs/pkgs/{module}/i18n/
    • Merges into wwwroot/i18n/{lang}/i18n.json (frontend bricks.js reads this)
    • Module order: pccs first (lowest priority), bricks last (highest)

  2. i18n_getmsgs.dspy at wwwroot/i18n_getmsgs.dspy:

    • bricks.js calls /i18n_getmsgs?lang=en (extensionless)
    • Returns wwwroot/i18n/{lang}/i18n.json content
    • Must NOT use import or os module (not pre-loaded in DSPY context)
    • Hardcode the path: open('/d/pccs/wwwroot/i18n/' + lang + '/i18n.json')

  3. RBAC for i18n: register BOTH /i18n_getmsgs AND /i18n_getmsgs.dspy with any role, plus /i18n/language.ui with any. Clear Redis cache after: redis-cli FLUSHDB.

Admin User: orgid REQUIRED

DSPY CRUD endpoints call await get_userorgid() which returns None for admin (orgid is null in the users table). This causes all CRUD list/get endpoints to return "Authorization Error / Please login".

Fix: Assign admin a valid orgid from the organization table:

SELECT id FROM organization LIMIT 1;  -- get an org id
UPDATE users SET orgid = '<org_id>' WHERE username = 'admin';

Then re-login to get a new session cookie with the orgid.

Tree Widget: textField MUST Match Data Key

Bricks Tree reads node text from user_data[opts.textField] (default "text"). If index.ui sidebar menu data uses "label" for display text, tree nodes render blank (icons only, no text). Add "textField": "label" to Tree widget options:

{"id": "pccs_menu_tree", "widgettype": "Tree", "options": {
    "textField": "label",
    "data": [{"id": "pool", "label": "算力池管理", ...}]
}}

Dashboard Stats API Endpoints

The index.ui dashboard calls per-module statistics APIs. Each consists of:

  1. A Python async function in {module}/__init__.py
  2. A thin DSPY wrapper in wwwroot/api/{name}.dspy
Endpoint Module Function
/pcpool/api/pool_stats.dspy pcpool pool_stats()
/pcc/api/cluster_status_all.dspy pcc cluster_status_all()
/storage_mgr/api/storage_stats.dspy storage_mgr storage_stats()
/image_mgr/api/image_stats.dspy image_mgr image_stats()

Function pattern (in module's __init__.py):

async def cluster_status_all(request, params_kw):
    env = request._run_ns
    dbname = env.get_module_dbname('pcc')
    async with DBPools().sqlorContext(dbname) as sor:
        clusters = await sor.R('cluster', {})
        statuses = {}
        for c in clusters:
            s = getattr(c, 'status', 'unknown')
            statuses[s] = statuses.get(s, 0) + 1
        return {'status': 'ok', 'data': {'total': len(clusters), ...}}

DSPY wrapper (thin, delegates to Python function):

result = await cluster_status_all(request, params_kw)
return result

After adding functions, must pip install the module and restart. Register paths in RBAC with logined role.

DSPY: os Module NOT Pre-loaded

os is NOT reliably available in the ahserver DSPY execution context. Use hardcoded paths instead:

# WRONG - os may not be defined
fp = os.path.join('/d/pccs/wwwroot/i18n', lang, 'i18n.json')

# CORRECT
with open('/d/pccs/wwwroot/i18n/' + lang + '/i18n.json') as f:
    return json.load(f)

Service Restart Pattern

redis-cli FLUSHDB                         # Clear stale RBAC cache
pkill -f 'pccs.py'                        # Stop
sleep 2
nohup bash /d/pccs/start.sh > /d/pccs/logs/nohup.log 2>&1 &
sleep 4
# Verify
curl -sk -o /dev/null -w '%{http_code}' https://pccs.opencomputing.cn/

Menu Widget Pitfalls (Tree → Menu Migration)

When switching from Tree to Menu for sidebar navigation:

  1. target MUST use app.xxx prefix — bricks.getWidgetById uses DOM closest()/querySelector(). From Menu widget (inside sidebar), sibling containers are unreachable. "target": "app.pccs_main_content" starts from body-level search. Plain "target": "pccs_main_content" fails.

  2. emoji icon field causes 401 — Menu items[].icon with emoji (e.g., "📊") gets treated as URL path /📊, triggering RBAC permission check. Use empty string "icon": "" — the label field already shows the emoji.

  3. Menu replaces entire target — the target widget's children get cleared and replaced. Put id on the content container (VScrollPanel or VBox inside HBox), NOT on the HBox that wraps sidebar+content.

  4. Bricks Submit button can't be CDP-clicked — use fetch('/rbac/user/up_login.dspy', {method:'POST', body:fd}) for login in browser tests.

CRUD Toolbar: Empty "tools": [] Removes Row Buttons

In CRUD JSON (*_list.json), params.toolbar.tools: [] causes row action buttons (edit/delete) to disappear. Delete the tools key entirely — never set it to empty array. The framework auto-generates default buttons when absent.

Auto-generated add/update DSPY: Datetime Field Cleanup

xls2ui-generated add_*.dspy and update_*.dspy pass all form fields to sor.C()/sor.U(). Empty string '' for datetime columns (last_heartbeat, *_at, *_time) causes MySQL error 1292. Add cleanup before the DB call:

for k in list(ns.keys()):
    if k.endswith('_at') or k.endswith('_time') or k == 'last_heartbeat':
        v = ns.get(k, '')
        if v == '' or v is None or v == 'None':
            ns[k] = None

RBAC any Role: Read-Only ONLY

any role MUST only include:

  • Stats/status APIs (pool_stats, cluster_status_all, storage_stats, image_stats)
  • UI pages (index.ui, menu.ui)
  • Info/health APIs (heartbeat, list_available)
  • Bricks static assets (/bricks/**, /favicon.ico)

CRUD operations (create/update/delete/deploy/allocate/release) MUST have logined only. get_*.dspy list queries require logined (they have internal get_userorgid() check).

DSPY: f-strings Cause unterminated string literal

DSPY exec context does NOT support f-strings with \n or complex expressions. Use string concatenation:

# WRONG
txt = f'CPU: {total_cpu}核\nGPU: {total_gpu}个'

# CORRECT
txt = 'CPU: ' + str(total_cpu) + '核 | GPU: ' + str(total_gpu) + '个'

Login Architecture (CRITICAL)

Login MUST use RBAC /rbac/user/login.ui PopupWindow — NEVER a custom HTML login form. A custom login that POSTs directly to a DSPY bypasses RBAC session establishment: AIOHTTP_SESSION cookie is never set, and all Bricks pages return 401.

The correct pattern: the login button on the static homepage redirects to /rbac/user/login.ui via a plain link or location.href. Do NOT use an iframe — RBAC login is a full Bricks page whose PopupWindow auto-opens.

After login, the DSPY dismissed bind navigates to /index.ui (Bricks dashboard). Also ensure all /rbac/user/* paths have roleid='any' in rolepermission.

Full detail: see references/auth-architecture.md. Bricks homepage pattern (index.ui + Tree sidebar + urlwidget user_panel): see references/bricks-homepage-pattern.md. Dashboard overview cards pattern (HBox colored cards from stats DSPY): see references/dashboard-stats-pattern.md. i18n setup (module translations, merge, RBAC, DSPY endpoint): see references/i18n-setup.md. i18n DSPY endpoint pattern (no-os, no-import, hardcoded path): see references/i18n-dspy-pattern.md. Tree Widget textField pitfall (menu items blank without textField=label): see references/tree-textfield-fix.md.