--- name: pccs-deploy description: "Use when deploying pccs. Config pitfalls, build.sh patterns." version: 1.0.0 category: devops --- # PCCS Deployment Knowledge ## Config ### 1. Database password MUST be AES-encrypted sqlor auto-decrypts `kwargs.password` via `aes_decode_b64(password_key, password)`. Plaintext causes `binascii.Error: Incorrect padding`. Fix: `aes_encode_b64(password_key, plaintext)`. ### 2. minsize/maxsize NOT in kwargs These are pool-level params. In kwargs they cause `connect() got an unexpected keyword argument 'minsize'`. Right: ```json { "pccs": { "driver": "mysql", "minsize": 2, "maxsize": 10, "kwargs": { "host": "...", "password": "", "db": "pccs" } } } ``` ### 3. processors MUST include `[".tmpl","tmpl"]` Without it bricks UI returns 500 on `/`. ### 4. indexes MUST include `"index.ui"` Without it `/` returns 500. ## Server Access - **SSH:** `ssh pccs@pccs.opencomputing.cn` (NOT root) - **App root:** `/d/pccs/` — build.sh lives here, copies files from git repos in `pkgs/` - **Git repos:** under `/d/pccs/pkgs/` — `pkgs/pccs/` is the main app repo (NOT `/d/pccs/` itself) - **Deploy:** `cd /d/pccs/pkgs/pccs && git pull origin main` then `sudo systemctl restart pccs` OR run full `bash /d/pccs/build.sh` which handles all modules - **Quick deploy (single-file):** `cp /d/pccs/pkgs/pccs/wwwroot/index.html /d/pccs/wwwroot/index.html && sudo systemctl restart pccs` - **Nginx:** proxies all traffic (including `/rbac/`) to `127.0.0.1:9180` (ahserver). No pipeline proxy. ### 5. Static HTML pages MUST include viewport meta The static `index.html` fallback page needs `` in ``. Without it, mobile browsers render at desktop width. The Bricks `index.ui` pages include it automatically via bricks HTML template. ### Pip mirror (China) ```bash pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/ pip install xls2ddl -q --timeout=30 || echo "skip" ``` ### Don't install mysql-client on MariaDB `apt install mysql-client` uninstalls mariadb-server. MariaDB provides `mysql` command. ### Correct directory layout ``` / ├── app/ ← entry point ├── conf/ ← config.json ├── pkgs/ ← all module repos ├── wwwroot/ ← symlinks only ├── py3/ ← venv ├── logs/ files/ scripts/ ├── build.sh └── start.sh ``` ### build.sh WORKDIR Place build.sh at app root, compute WORKDIR via `$(dirname "$0")`. ## RBAC Permission Setup (build.sh step 7b) After deployment, RBAC permissions MUST be initialized. The project follows the same pattern as Sage: `setup_rbac_perms.sh` + `set_role_perm.py`. **setup_rbac_perms.sh** sets three tiers: 1. **Public (any):** login/register/auth paths including `/rbac/user/user_panel.ui` and `/appbase/menu.ui` 2. **Logined:** profile, usermenu, `/index.ui` 3. **Static (any + /** wildcards):** `/bricks/**`, `/imgs/**`, `/pcpool/**`, etc. **set_role_perm.py** is called per-path: `./py3/bin/python3 set_role_perm.py any /bricks/**` **IMPORTANT:** After running permissions setup, clear Redis RBAC cache: ```bash redis-cli KEYS 'rbac*' | xargs redis-cli DEL sudo systemctl restart pccs ``` Without this, stale cached permissions cause 401 on paths that now have `any` role. **Verify** bricks.js is accessible (critical — the entire UI breaks without it): ```bash curl -sk -o /dev/null -w "%{http_code}" https://pccs.opencomputing.cn/bricks/bricks.js # must return 200 ``` ## pip install: pyproject.toml REQUIRED (NOT setup.json) Business modules (pcpool, pcc, storage_mgr, image_mgr) use `setup.json` (Sage format). pip install does NOT recognize `setup.json` — only `setup.py`, `setup.cfg`, or `pyproject.toml`. build.sh step 4 checks `[ -f setup.py ] || [ -f setup.cfg ] || [ -f pyproject.toml ] || continue`, silently skipping all modules with only `setup.json`. **Fix**: generate `pyproject.toml` from `setup.json` for every module: ```toml [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] name = "pcpool" version = "0.1.0" requires-python = ">=3.10" dependencies = ["apppublic", "sqlor", "ahserver", "appbase"] [tool.setuptools.package-dir] "pcpool" = "pcpool" [tool.setuptools.packages.find] where = ["."] ``` Multi-package modules (e.g., pcc with sub-plugins) need explicit package-dir entries: ```toml [tool.setuptools.package-dir] "pcc" = "pcc" "pcc.k8s_plugin" = "pcc/k8s_plugin" "pcc.slurm_plugin" = "pcc/slurm_plugin" ``` ## Nginx: proxy_set_header Host REQUIRED in location / The `location /` block MUST include `proxy_set_header Host $host;`. Without it, nginx passes the upstream name (e.g., `sage_backend`) as Host, ahserver rejects DSPY paths with `str(request.url)='http://sage_backend/...' invalid path`. ## i18n Setup (REQUIRED) Every module needs `i18n/{lang}/msg.txt` (key: value per line). Merge into `wwwroot/i18n/`: 1. **merge_i18n.py** (at app root, invoked by build.sh step 7b): - Reads all modules' `i18n/{zh,en}/msg.txt` from `/d/pccs/pkgs/{module}/i18n/` - Merges into `wwwroot/i18n/{lang}/i18n.json` (frontend bricks.js reads this) - Module order: pccs first (lowest priority), bricks last (highest) 2. **i18n_getmsgs.dspy** at `wwwroot/i18n_getmsgs.dspy`: - bricks.js calls `/i18n_getmsgs?lang=en` (extensionless) - Returns `wwwroot/i18n/{lang}/i18n.json` content - Must NOT use `import` or `os` module (not pre-loaded in DSPY context) - Hardcode the path: `open('/d/pccs/wwwroot/i18n/' + lang + '/i18n.json')` 3. **RBAC** for i18n: register BOTH `/i18n_getmsgs` AND `/i18n_getmsgs.dspy` with `any` role, plus `/i18n/language.ui` with `any`. Clear Redis cache after: `redis-cli FLUSHDB`. ## Admin User: orgid REQUIRED DSPY CRUD endpoints call `await get_userorgid()` which returns `None` for admin (orgid is null in the `users` table). This causes all CRUD list/get endpoints to return "Authorization Error / Please login". **Fix**: Assign admin a valid orgid from the `organization` table: ```sql SELECT id FROM organization LIMIT 1; -- get an org id UPDATE users SET orgid = '' WHERE username = 'admin'; ``` Then re-login to get a new session cookie with the orgid. ## Tree Widget: textField MUST Match Data Key Bricks Tree reads node text from `user_data[opts.textField]` (default `"text"`). If index.ui sidebar menu data uses `"label"` for display text, tree nodes render blank (icons only, no text). Add `"textField": "label"` to Tree widget options: ```json {"id": "pccs_menu_tree", "widgettype": "Tree", "options": { "textField": "label", "data": [{"id": "pool", "label": "算力池管理", ...}] }} ``` ## Dashboard Stats API Endpoints The index.ui dashboard calls per-module statistics APIs. Each consists of: 1. A Python async function in `{module}/__init__.py` 2. A thin DSPY wrapper in `wwwroot/api/{name}.dspy` | Endpoint | Module | Function | |----------|--------|----------| | `/pcpool/api/pool_stats.dspy` | pcpool | `pool_stats()` | | `/pcc/api/cluster_status_all.dspy` | pcc | `cluster_status_all()` | | `/storage_mgr/api/storage_stats.dspy` | storage_mgr | `storage_stats()` | | `/image_mgr/api/image_stats.dspy` | image_mgr | `image_stats()` | **Function pattern** (in module's `__init__.py`): ```python async def cluster_status_all(request, params_kw): env = request._run_ns dbname = env.get_module_dbname('pcc') async with DBPools().sqlorContext(dbname) as sor: clusters = await sor.R('cluster', {}) statuses = {} for c in clusters: s = getattr(c, 'status', 'unknown') statuses[s] = statuses.get(s, 0) + 1 return {'status': 'ok', 'data': {'total': len(clusters), ...}} ``` **DSPY wrapper** (thin, delegates to Python function): ```python result = await cluster_status_all(request, params_kw) return result ``` After adding functions, must `pip install` the module and restart. Register paths in RBAC with `logined` role. ## DSPY: `os` Module NOT Pre-loaded `os` is NOT reliably available in the ahserver DSPY execution context. Use hardcoded paths instead: ```python # WRONG - os may not be defined fp = os.path.join('/d/pccs/wwwroot/i18n', lang, 'i18n.json') # CORRECT with open('/d/pccs/wwwroot/i18n/' + lang + '/i18n.json') as f: return json.load(f) ``` ## Service Restart Pattern ```bash redis-cli FLUSHDB # Clear stale RBAC cache pkill -f 'pccs.py' # Stop sleep 2 nohup bash /d/pccs/start.sh > /d/pccs/logs/nohup.log 2>&1 & sleep 4 # Verify curl -sk -o /dev/null -w '%{http_code}' https://pccs.opencomputing.cn/ ``` ## Menu Widget Pitfalls (Tree → Menu Migration) When switching from Tree to Menu for sidebar navigation: 1. **`target` MUST use `app.xxx` prefix** — `bricks.getWidgetById` uses DOM `closest()`/`querySelector()`. From Menu widget (inside sidebar), sibling containers are unreachable. `"target": "app.pccs_main_content"` starts from body-level search. Plain `"target": "pccs_main_content"` fails. 2. **emoji `icon` field causes 401** — Menu `items[].icon` with emoji (e.g., `"📊"`) gets treated as URL path `/📊`, triggering RBAC permission check. Use empty string `"icon": ""` — the `label` field already shows the emoji. 3. **Menu replaces entire target** — the target widget's children get cleared and replaced. Put `id` on the content container (VScrollPanel or VBox inside HBox), NOT on the HBox that wraps sidebar+content. 4. **Bricks Submit button can't be CDP-clicked** — use `fetch('/rbac/user/up_login.dspy', {method:'POST', body:fd})` for login in browser tests. ## CRUD Toolbar: Empty `"tools": []` Removes Row Buttons In CRUD JSON (`*_list.json`), `params.toolbar.tools: []` causes row action buttons (edit/delete) to disappear. **Delete the `tools` key entirely** — never set it to empty array. The framework auto-generates default buttons when absent. ## Auto-generated add/update DSPY: Datetime Field Cleanup xls2ui-generated `add_*.dspy` and `update_*.dspy` pass all form fields to `sor.C()`/`sor.U()`. Empty string `''` for datetime columns (`last_heartbeat`, `*_at`, `*_time`) causes MySQL error 1292. Add cleanup before the DB call: ```python for k in list(ns.keys()): if k.endswith('_at') or k.endswith('_time') or k == 'last_heartbeat': v = ns.get(k, '') if v == '' or v is None or v == 'None': ns[k] = None ``` ## RBAC `any` Role: Read-Only ONLY `any` role MUST only include: - Stats/status APIs (pool_stats, cluster_status_all, storage_stats, image_stats) - UI pages (index.ui, menu.ui) - Info/health APIs (heartbeat, list_available) - Bricks static assets (`/bricks/**`, `/favicon.ico`) CRUD operations (create/update/delete/deploy/allocate/release) MUST have `logined` only. `get_*.dspy` list queries require `logined` (they have internal `get_userorgid()` check). ## DSPY: f-strings Cause `unterminated string literal` DSPY exec context does NOT support f-strings with `\n` or complex expressions. Use string concatenation: ```python # WRONG txt = f'CPU: {total_cpu}核\nGPU: {total_gpu}个' # CORRECT txt = 'CPU: ' + str(total_cpu) + '核 | GPU: ' + str(total_gpu) + '个' ``` ## Login Architecture (CRITICAL) Login MUST use RBAC `/rbac/user/login.ui` PopupWindow — NEVER a custom HTML login form. A custom login that POSTs directly to a DSPY bypasses RBAC session establishment: `AIOHTTP_SESSION` cookie is never set, and all Bricks pages return 401. The correct pattern: the login button on the static homepage redirects to `/rbac/user/login.ui` via a plain link or `location.href`. Do NOT use an iframe — RBAC login is a full Bricks page whose PopupWindow auto-opens. After login, the DSPY `dismissed` bind navigates to `/index.ui` (Bricks dashboard). Also ensure all `/rbac/user/*` paths have `roleid='any'` in rolepermission. Full detail: see `references/auth-architecture.md`. Bricks homepage pattern (index.ui + Tree sidebar + urlwidget user_panel): see `references/bricks-homepage-pattern.md`. Dashboard overview cards pattern (HBox colored cards from stats DSPY): see `references/dashboard-stats-pattern.md`. i18n setup (module translations, merge, RBAC, DSPY endpoint): see `references/i18n-setup.md`. i18n DSPY endpoint pattern (no-os, no-import, hardcoded path): see `references/i18n-dspy-pattern.md`. Tree Widget textField pitfall (menu items blank without textField=label): see references/tree-textfield-fix.md.