---
name: sage-module-deployment
description: "Deploy Sage module changes from dev to test to production and avoid common pitfalls."
tags: [sage, deployment, git, unipay, wechat]
---

# Sage Module Deployment

## Deployment Flow

```
repos/ → commit → push → pull on target → pip install --upgrade → restart Sage
```

**IMPORTANT: `git pull` alone does NOT update the running Python module.** The `sage_datamart`, `supplychain`, `llmage`, etc. Python packages are installed in `py3/lib/python3.10/site-packages/`. After `git pull` you MUST `pip install --upgrade` for Sage to see the new code.

**Deployment-gap detection** — pushed commits are NOT the same as deployed code. Three-way md5 check tells you exactly where the chain broke:
```bash
MODULE=llmage; FILE=llmage/llmclient.py
git -C /d/ymq/repos/$MODULE show HEAD:$FILE | md5sum                      # (a) local HEAD
ssh apitest@120.48.168.15 "cd /d/apitest/sage/pkgs/$MODULE && md5sum $FILE"  # (b) server repo
ssh apitest@120.48.168.15 "md5sum /d/apitest/sage/py3/lib/python3.10/site-packages/$FILE"  # (c) running code
```
(a)≠(b): commits never pulled on server. (b)≠(c): pulled but `pip install` skipped. Any mismatch = the fix believed live is NOT live; complete pull → pip install → restart → curl verify before reporting. Also `git status --short` the server repo — `.bak` files and untracked test pages are evidence of direct server edits.

1. Commit: `cd /d/ymq/repos/<module> && git status` — review before `git add`
2. Push: `git push`
3. Deploy to target:

```bash
# On target server (test or prod):
cd ~/repos/<module> && git pull
~/sage/py3/bin/pip install --upgrade ~/repos/<module>
# --upgrade forces overwrite; -e . is not enough for non-editable installs
```

4. Restart: see "Server restart" below

## Server restart

**Both test and production** use multiple workers via SO_REUSEPORT. `kill $(cat sage.pid)` alone leaves orphaned processes.

```bash
# Kill ALL sage processes
pkill -9 -f 'sage.py|backend_accounting'
sleep 2

# Restart from Sage root
cd ~/sage && bash start.sh
```

`start.sh` checks Redis, starts backend_accounting, then spawns N workers. PID files: `sage.pid`, `sage_backend.pid`.

**Verify**: `ps aux | grep -E 'sage|backend' | grep -v grep` — should show N workers + 1 backend.

**Manual fallback when start.sh is unusable**（observed 2026-08-04 on test server: `bin/` contained only `backend_accounting.py` and `backup_api.sh`, no start.sh/stop.sh）:

```bash
pkill -9 -f 'sage.py|backend_accounting'; sleep 2
cd /d/apitest/sage
for i in 1 2 3 4; do
  nohup ./py3/bin/python app/sage.py --workdir /d/apitest/sage --port 9180 >> logs/sage.log 2>&1 &
done
nohup ./py3/bin/python bin/backend_accounting.py >> logs/backend_accounting.log 2>&1 &
# verify: ss -tlnp | grep 9180 ; ps aux | grep -E 'sage.py|backend_accounting' | grep -v grep
```

All 4 workers share the identical command line (SO_REUSEPORT). `bin/backend_accounting.py` is the separate process that consumes the llmusage accounting queue — **forgetting to restart it leaves stale accounting code running** even though all sage workers are fresh. Check both process groups' start times after any restart.

**Pitfall — `stop.sh` may kill nothing**: test-server `stop.sh` can fail with `killname: command not found` (broken helper) and silently leave the old workers AND old `backend_accounting.py` running. Then `start.sh` spawns a second full set; SO_REUSEPORT lets the new workers bind 9180 fine, but the orphaned old backend keeps processing accounting events with stale code. After every restart, compare process start times and kill orphans:
```bash
ps -ef | grep -E 'sage.py|backend_accounting' | grep -v grep   # column 5 = start time
# all start times must be from the fresh start; kill older PIDs explicitly
```
Prefer the pkill-all approach above over trusting stop.sh.

**Pitfall — `kill -9` on Sage processes kills the SSH session**: When running `kill -9` on sage worker processes from an SSH session where the worker is handling the SSH connection's traffic (e.g., the SSH goes through nginx→sage on the same host), the command kills the SSH session itself, causing exit code 255 and losing any commands after the kill. Workaround: use `nohup bash -c "kill ..." &` to run the kill in a detached process, or kill PIDs individually with `kill` (SIGTERM) first, only using `-9` as a last resort. After a kill-induced disconnect, wait 3s then reconnect.

## Pitfalls

### Module .dspy/.ui 改动只需 git pull 即生效（无需 pip install/重启）

模块的 dspy/ui 文件经 `wwwroot/<module>` symlink 指向 `pkgs/<module>/wwwroot/`，每次请求从磁盘读取。对**纯 dspy 改动**，`cd /d/apitest/sage/pkgs/<module> && git pull` 之后下一次请求就跑新代码——不需要 pip install，不需要重启。

**实证 2026-08-04**：llmage 上架联动 hook 的 dspy 修复，在 `pkgs/llmage` git pull 后下一次 publish 请求立即执行新代码（无重启、无 pip install）。与下方 RAG 节的 live-serve 模式相同。

**混合改动的拆分策略**（如 dspy hook 在 llmage、Python 函数在 product_management）：dspy 模块 pull 后可立即测试链路；Python 包侧仍需 pip install + 重启。部署前先分清改动落在 .dspy/.ui 还是 .py。

### wwwroot symlinks are deployment artifacts, not source

`~/repos/sage/wwwroot/` is the Sage source repository. Module wwwroot symlinks (e.g. `supplychain -> /d/ymq/repos/supplychain/wwwroot`) are created by `build.sh` during deployment and should NEVER exist in the Sage source repo. They pollute the source directory and confuse `git status`.

```bash
# In ~/repos/sage/wwwroot/ — these are WRONG:
./supplychain -> /home/hermesai/repos/supplychain/wwwroot
./pricing -> /home/hermesai/repos/pricing/wwwroot
./bugfix -> /home/hermesai/test/bugfix/wwwroot
# and ALL module wwwroot symlinks (msp, rag, cpcc, dapi, rbac, ...)
```

Fix: `rm -f <symlink_name>` for every module wwwroot symlink. These are deployment concerns — they belong on the running server (created by build.sh), not in the git-tracked source tree.

Exceptions: `bricks -> ../../bricks/dist` (Bricks UI framework) and `shell_theme.*` files are Sage core, not module symlinks.

### Untracked files on remote blocking git pull

If a directory was created manually on the server (e.g. `wwwroot/<new_dir>/`) before a commit that adds the same path, `git pull` aborts with:
```
error: The following untracked working tree files would be overwritten by merge:
    wwwroot/<new_dir>/...
```
Fix: `rm -rf wwwroot/<new_dir>/` then `git pull`.

### build.sh: second `for m in` line can be accidentally deleted

When removing a module name from the business-module `for m in` list in `build.sh`, the entire `for m in ...` line can be deleted instead of just the module name. This leaves a bare `do` block without a loop variable. `merge_i18n.py` (which expects exactly 2 `for m in` lines) then fails with `"错误: 在 build.sh 中找到 1 个 for m in 行，期望至少2个"`.

```bash
# Verify both for m in lines are present:
grep -c '^for m in ' build.sh  # must return 2
```

**Fix**: restore the missing `for m in <modules...>` line before the bare `do`. Use `git show <prior-commit>:build.sh` to recover the original list, then remove only the unwanted module name.

### `merge_i18n.py`: modules under `pkgs/` need fallback lookup

`merge_i18n.py` looks for module i18n at `REPOS_DIR/<module>/i18n/` (e.g. `/d/apitest/bricks/i18n`). On test servers, modules are cloned under `SAGE_DIR/pkgs/<module>/` not `REPOS_DIR/<module>/`. The fix: add `SAGE_DIR/pkgs/<module>/i18n` as a secondary lookup path in `find_module_i18n_dir()`.

### Local changes on test server blocking git pull

测试服务器上可能有运行时产生的本地修改（如 `wwwroot/index.ui` 被部署脚本修改）。`git pull` 被 `local changes would be overwritten` 阻止。

```bash
# 标准三击：
ssh apitest@tokentest.opencomputing.cn "cd ~/sage && git stash && git pull && git stash pop"

# 若有冲突，接受远端版本并丢弃 stash：
ssh apitest@tokentest.opencomputing.cn "cd ~/sage && git checkout --theirs <file> && git add <file> && git stash drop"
```

**注意**: 测试服务器 `python3` 非 `python`；DNS 失效时用 IP `120.48.168.15`。

### `git pull` succeeds but new code still doesn't run

**Root cause**: `pip install` was skipped. Git updates the source files on disk, but Sage imports from `py3/lib/python3.10/site-packages/`. Without `pip install --upgrade`, the old bytecode in site-packages is still used.

**Symptom**: Jinja2 errors like `'j2_realtime_calls' is undefined` or `NameError: name 'api_xxx' is not defined` — functions that clearly exist in the source file.

**Fix**: Always run `pip install --upgrade` after git pull. Not `pip install -e .` (which only links), but `--upgrade` which forces overwrite of the installed package.

```bash
~/sage/py3/bin/pip install --upgrade ~/repos/<module>
```

### Bare `pip` silently installs to ~/.local — the venv never sees it

On the test server, bare `pip` resolves to the SYSTEM pip (`/usr/bin/pip`) and installs into the user site dir `/d/apitest/.local/lib/python3.10/site-packages/` — NOT the venv the service runs from. pip still prints `Successfully installed llmage-0.0.1`, so the deploy *looks* fine while the running code stays old. (Observed 2026-08-04: three `--force-reinstall` runs "succeeded" before anyone checked WHERE they landed.)

```bash
which pip; pip --version                        # /usr/bin/pip → system pip, WRONG target
/d/apitest/sage/py3/bin/pip --version           # must show the venv path

# Correct install, then verify by CONTENT, not by pip's success message:
/d/apitest/sage/py3/bin/pip install --force-reinstall --no-deps /d/apitest/sage/pkgs/<module>
grep -c '<new_function_marker>' /d/apitest/sage/py3/lib/python3.10/site-packages/<pkg>/<file>.py
```

Rule: a deployment is only real after grepping the venv site-packages for markers of the new code. pip output alone proves nothing.

### Never modify CRUD intermediate files

`wwwroot/<listname>/*.dspy` and `wwwroot/<listname>/*.ui` are generated from `json/*.json` by `xls2ui`. Never edit them directly — change the JSON source and regenerate. Never commit them.

If remote already tracks these files, clean up:
```bash
git rm -r --cached wwwroot/<listname>/
echo "wwwroot/<listname>/" >> .gitignore
git add .gitignore && git commit -m "chore: remove CRUD intermediate files from tracking"
```

`wwwroot/api/<name>.dspy` is exempt — those are independent APIs, not generated.

### Never commit egg-info / __pycache__

`git add -A` picks up `.pyc` and `*.egg-info/` junk.

Fix: add `.gitignore` with `__pycache__/`, `*.pyc`, `*.egg-info/`. Always `git status` before commit. If already tracked on remote, `git rm -r --cached`, commit, push.

### load_path.py crashes on `#` comment lines

`paths.split('\n')` iterates every line. A comment like `# sage_datamart` is not empty, so `l.split()` produces `['#', 'sage_datamart']` → `roles=['sage_datamart']`. This string is neither `any`/`logined`/`anonymous` nor has a `.` separator, so `role.split('.')` in `add_roles_perm()` raises `ValueError: not enough values to unpack (expected 2, got 1)`.

**Fix**: skip comment lines:
```python
for l in paths.split('\n'):
    if not l or l.startswith('#'):   # ← add l.startswith('#')
        continue
```

### load_path.py: must EXECUTE after editing — adding paths is not enough

Adding paths to `load_path.py` and committing is only the first step. The script must be **run** on the target server to register permissions in the database:

```bash
cd ~/sage && python load_path.py
```

**Symptom of missing this step**: new endpoints silently return **401/403** (`permission check failed` in the log) even though the path appears in `load_path.py`. All requests fail with `need login to access`.

**Also**: when replacing old UI paths (e.g., cockpit page replaces list views), update `global_menu.ui` to point to the new path.

### i18n not working — `merge_i18n.py` pipeline (NOT wheel JSON)

Sage i18n does NOT use per-package wheel JSON. Each module has `i18n/{lang}/msg.txt` at the **repo root** (e.g. `~/repos/sage_datamart/i18n/zh/msg.txt`), NOT under the Python package dir. Format: `key: value` per line (NOT JSON, NOT `key: value` with JSON blob on one line). `merge_i18n.py` in Sage root consolidates all modules into `wwwroot/i18n/{lang}/i18n.json` which the frontend loads via `/i18n_getmsgs`.

**Symptoms**: dashboard shows mixed languages — some labels correct (key=value coincidence), others show English or raw key.

**Checklist — three required conditions for i18n to work**:
1. Module has `i18n/{zh,en,jp,ko}/msg.txt` at **repo root** in `key: value` line format
2. Module listed in `build.sh`'s second `for m in` loop (line 29) — `merge_i18n.py` parses this list
3. `python merge_i18n.py` run after any msg.txt change to regenerate `wwwroot/i18n/`

**msg.txt format** (NOT JSON!):
```
今日调用笔数: 今日调用笔数
今日交易金额: 今日交易金额
```

**Pitfall — JSON blob in msg.txt**: A single-line JSON `{"key":"val",...}` is silently skipped by `parse_msg_txt()` which expects colon-delimited lines per entry.

**Pitfall — merge order matters**: `merge_i18n.py` processes modules in order; later overwrites earlier. All business modules define common keys (Submit, Reset, Cancel) as English→English in their en/msg.txt, overwriting bricks' zh translations. **bricks must be LAST** in merge order:
```python
# merge_i18n.py — bricks last so framework strings win
result = ['sage'] + modules + ['bricks']
```

**Pitfall — bricks Tabular strings**: The bricks dist has `bricks.app.i18n._()` calls for "Add record", "Update record", "Clone record", "Delete conform", "Submit", "Reset", "Cancel", "Conform", "Discard". These keys MUST be in `bricks/i18n/{lang}/msg.txt`. Also fix `dynamicaccordion.js` hardcoded `'Delete conform'` → `bricks.app.i18n._('Delete conform')`.

**After fixing msg.txt**:
```bash
cd ~/sage && python merge_i18n.py   # regenerate wwwroot/i18n/
# No pip install needed — i18n is wwwroot-served, not in Python package
```

### Never hardcode dbname in sqlorContext

```python
# WRONG — hardcoded 'sage' or 'llmage'
DBPools().sqlorContext('sage')
db.sqlorContext('llmage')

# RIGHT — use dynamic dbname
ServerEnv().get_module_dbname('sage')
ServerEnv().get_module_dbname('llmage')
env.get_module_dbname('discount')  # if env = request._run_ns
```

`_get_sor()` / `_get_dbname()` helpers in each module already use `get_module_dbname()`. Always follow the existing pattern in the module.

### DBPools() is NOT a singleton

`DBPools()` creates a fresh EMPTY instance. Must explicitly assign databases:
```python
db = DBPools()
config = getConfig()
db.databases = config.databases
async with db.sqlorContext(dbname) as sor:
    ...
```

### Never sync local site-packages

Local `/d/ymq/repos/sage/py3/lib/python3.10/site-packages/` is irrelevant. Test server runs the real code. Push to git, pull on target. User: "本地的同步 site-packages 没有任何意义，你的测试环境在tokentest而不是本地"

### Never scp/sed to deploy

Deploy via git pull + pip install only. No scp individual files, no sed on server.

### WeChat amount: yuan → fen (×100)

WeChat `amount.total` uses fen (分). User enters yuan (元). Must convert:
```python
"total": int(payload["amount"] * 100),  # CORRECT
```
`int(0.01)` = 0 which fails WeChat minimum validation.

### SQLor: str(limit) gets quoted → LIMIT '5' syntax error

SQLor's `${param}$` substitution quotes string params as SQL strings. `LIMIT ${limit}$` with `limit = str(5)` produces `LIMIT '5'` — invalid SQL. Pass integers directly:

```python
# WRONG
recs = await sor.sqlExe(sql, {'limit': str(limit)})   # → LIMIT '5'

# RIGHT
recs = await sor.sqlExe(sql, {'limit': limit})          # → LIMIT 5
```

### SQLor: verify column names against model JSON

Never assume column names. `users` table has `orgid`, NOT `userorgid`. Always grep the model JSON before writing SQL:

```python
# WRONG — users table does not have userorgid
"SELECT COUNT(*) FROM users WHERE userorgid = ${org}$"

# RIGHT
"SELECT COUNT(*) FROM users WHERE orgid = ${org}$"
```

Tables and their columns:
| Table | org_id column | notes |
|-------|--------------|-------|
| `users` | `orgid` | |
| `llmusage` | `userorgid` | |
| `dm_model_call_fact` | `userorgid` | |
| `organization` | `id` | org_type, resellerid |

### Never use get_sor_context(None, module) — sor.env is None

`get_sor_context(env, module)` requires a valid `env` (ServerEnv). Passing `sor.env` fails because `sor` objects from sqlExe contexts don't carry `env`. When all tables share one Sage database, use the passed `sor` directly:

```python
# WRONG — sor has no .env, get_sor_context(None) → NoneType.get_module_dbname
async with get_sor_context(sor.env if hasattr(sor, 'env') else None, 'accounting') as other_sor:
    rows = await other_sor.sqlExe(...)

# RIGHT — same database, use the existing sor
rows = await sor.sqlExe(sql, ns)
```

### Pipeline-app umbrella repo pattern

pipeline-app is a **pure umbrella** — it contains only config, `app/` entry point, and `wwwroot/` symlinks. All modules live as independent git repos under `pkgs/`. The umbrella repo itself must never contain embedded module source.

```bash
# Local structure:
/d/ymq/pipeline/pipeline-app/          # umbrella (pipeline-app.git)
/d/ymq/pipeline/pipeline-app/pkgs/     # independent module clones
/d/ymq/pipeline/pipeline-sdlc/         # also exists as standalone clone for solo dev
/d/ymq/pipeline/pipeline-service/      # same
```

When a module needs its own repo, extract it via `git subtree split` (see `references/git-audit-reconcile.md`). After extraction: add module dir to `.gitignore`, `git rm -r --cached` the old directory, clone the new repo into `pkgs/`, and set relative symlink `wwwroot/<module> -> ../pkgs/<module>/wwwroot`.

### 独立模块迁移三件套：r:p 权限 + 建表 + 初始化数据（替代每模块 load_path.py）

独立模块（产品/折扣/定价/支付/短信）迁移/复用到新平台（pipeline-app）时，除 clone + pip install + wwwroot symlink 外，还须补三件套——否则换环境部署又漏，用户会重复纠正：

1. **权限**：`conf/rp.json`（应用仓库定义角色→路径模式）+ `scripts/import_rp.py`（幂等导入 permission+rolepermission）。**替代每模块 `scripts/load_path.py`**（旧机制硬编码 PATHS_ANY/PATHS_LOGINED 角色 + `find_sage_root` 找 Sage 根，模块耦合 Sage + 角色体系）。
2. **建表**：`scripts/create_tables.py`（models/*.json → `json2ddl mysql <models_dir>` → DDL 按 `;` 分割、跳过 `--` 注释行，逐条 `await sor.execute(stmt, {})`）。
3. **初始化数据**：`scripts/import_init.py`（init/data.json 的 `appcodes`/`appcodes_kv`/`organization` 三段幂等导入，R 检查 + C 插入）。

三个脚本都固化进 build.sh 自动执行。

**rp.json 三层分离**（用户明确方向：模块声明路径 → 应用仓库定义 r:p → 脚本导入）：
```json
{"roles": {
  "any": ["/rbac/user/login.ui", "/i18n_getmsgs", "/index_tab.js"],
  "logined": ["/pipeline-sdlc/**", "/product_management/**", "/discount/**"],
  "owner.superuser": ["/rbac/**"]
}}
```
- `**` 前缀直接存 permission.path（**不展开**成具体路径），RBAC `check_roles_path` 原生前缀匹配。
- permission 表 path 唯一索引（permission_idx1），import_rp 按 path 去重复用 permid；rolepermission 按 roleid+permid 去重。
- 根目录 .js（如 `/index_tab.js`）会被 RBAC 拦 401，需在 rp.json `any` 加路径；模块子目录 .js 因 `/模块/**` 已有权限所以 200。

**关键陷阱**：
- **config.json 密码是 AES 加密的**，建表不能用 mysql 命令行（需明文密码），必须 DBPools + sor.execute（DBPools 内部解密）。
- json2ddl 生成的 DDL 含 `DROP TABLE IF EXISTS` + `CREATE TABLE`，幂等（表不存在时 DROP 无害）。
- **`load_xxx()` 不是导入数据**——它注册模块函数到 ServerEnv（让 .dspy 调模块函数）；`init/data.json` 才是字典数据导入。漏 load_xxx → DSPY NameError 500；漏 init/data.json → appcodes 字典空。
- CRUD 脚本（json/*.json + models → xls2ui → wwwroot）通常 repo 已提交，symlink 即部署，不用重新生成。

### git.opencomputing.cn SSH rate limit on test server

The test server (120.48.168.15) sometimes hits SSH rate limiting (`kex_exchange_identification`). Fall back to HTTPS for `git pull` on the server:

```bash
git pull https://git.opencomputing.cn/yumoqing/<repo>.git main
```

SSH works fine from the dev machine — only change the server-side pull command.

**JSON-format DSPY** (`{"python": {"import": "...", "call": "..."}}`) throws `return data type error, <class 'NoneType'>` when the called function returns `None` (e.g., `get_sor_context` fails silently). **Python script format** (`import json; data = await ...; return json.dumps(data)`) never has this problem because `json.dumps([])` always returns a valid JSON string.

```python
# WRONG — JSON format DSPY, NoneType on any exception
{
    "python": {
        "method": "get",
        "import": "sage_datamart.init",
        "call": "api_currency_stats"
    }
}

# RIGHT — Python script format, always returns valid JSON
import json
env = request._run_ns
data = await env.api_currency_stats(request)
return json.dumps(data, ensure_ascii=False)
```

**Symptom**: `return data type error, <class 'NoneType'>` in the traceback. Fix: convert ALL DSPY files to Python script format with `json.dumps()`.

## RAG Server Deployment

The RAG server (`ragserver`) deploys differently from Sage modules because .dspy/.ui files are served live from the wwwroot filesystem — no Python module import or pip install is involved.

### Deployment flow

```bash
# 1. Git pull the wwwroot files
cd /d/rag/ragserver/pkgs/rag && git pull

# 2. (If new endpoint) Add RBAC permission + rolepermission
# Check if permission already exists first:
mysql -h db -u test -ptest123 rag -e \
  "SELECT p.*, rp.roleid FROM permission p LEFT JOIN rolepermission rp ON rp.permid=p.id WHERE p.path LIKE '%storage_stats%'"

# If it doesn't exist:
# - permission.id is typically a short code like "perm_rag_storage_stats"
# - permission.path is the URL path, e.g. "/rag/knowledge_bases_list/storage_stats.dspy"
mysql -h db -u test -ptest123 rag -e \
  "INSERT INTO permission (id, name, path) VALUES ('perm_rag_<name>', 'RAG', '/rag/knowledge_bases_list/<name>.dspy')"
# Link to 'any' role:
mysql -h db -u test -ptest123 rag -e \
  "INSERT INTO rolepermission (id, roleid, permid) VALUES ('rp_<name>', 'any', 'perm_rag_<name>')"

# 3. Flush Redis cache so RBAC is re-read
redis-cli FLUSHDB

# 4. Restart is OPTIONAL for .dspy/.ui changes only
# .dspy files are read from disk on each HTTP request — git-pulled changes go live instantly.
# Only restart if Python backend code (app/*.py) changed.

# 5. Verify
curl -s -w '\nHTTP_CODE:%{http_code}' \
  'http://127.0.0.1:9181/rag/knowledge_bases_list/<name>.dspy?_webbricks_=1'
```

### Key differences from Sage

| Aspect | Sage module | RAG server |
|--------|-------------|------------|
| Code location | Installed in site-packages via pip | Read from wwwroot filesystem |
| After git pull | Must `pip install --upgrade` | No pip install needed |
| Restart needed | Always (Python modules cached in memory) | Only for app/*.py changes |
| URL prefix | Configurable per module | `/rag/` (fixed) |
| wwwroot symlink | Multiple `wwwroot/<module> -> pkgs/<module>/wwwroot` | Single `wwwroot/rag -> ../pkgs/rag/wwwroot` |

### Process management

- RAG server runs as user `rag`, not `apitest`
- SSHing as `apitest@120.48.168.15` (password: `sword/Aa@123456`) can read/write files, run mysql, redis, and curl — but CANNOT kill/restart the process:
  ```bash
  pkill -f ragserver        # fails: Operation not permitted (wrong user)
  ```
- To restart, either:
  - SSH directly as `rag` (key-based auth)
  - Use `su - rag` on the server (requires rag user's password)
- The process is started with:
  ```bash
  cd /d/rag/ragserver && source py3/bin/activate && setsid python -B app/ragserver.py
  ```
- Port: 9181 (loopback only; Sage frontend on port 9180 proxies `/rag/` to it)

### RBAC permission patterns

RAG permissions follow the Sage RBAC model (same database). Common patterns:

| permission.path | purpose |
|-----------------|---------|
| `/rag/knowledge_bases_list/storage_stats.dspy` | API endpoint (.dspy) |
| `/rag/knowledge_bases_list/index.ui` | UI page |
| `/knowledge_bases_list/upload.dspy` | Without `/rag` prefix (older style) |

Check existing rolepermission for 'any' role:
```bash
mysql -h db -u test -ptest123 rag -e "SELECT * FROM rolepermission WHERE roleid='any'"
```

### Verification

```bash
curl -s -w '\nHTTP_CODE:%{http_code}' \
  'http://127.0.0.1:9181/rag/knowledge_bases_list/storage_stats.dspy?_webbricks_=1'
```

Expected: HTTP 200 with a Bricks widget JSON response (VBox/HBox/Text etc.).

To check if a .dspy endpoint is referenced in the UI:
```bash
grep 'storage_stats' pkgs/rag/wwwroot/knowledge_bases_list/index.ui
```
Expected: a urlwidget or actiontype entry with `"url": "{{entire_url('./storage_stats.dspy')}}"`.

### Redis usage

- FLUSHDB clears cached RBAC permissions and session data
- Always run after adding/changing permissions
- Does NOT affect active HTTP connections (sessions re-authenticate on next request)

## Pitfalls (continued)

### Bricks: getWidgetById searches DOWNWARD (children), not UPWARD (ancestors)

**Critical pitfall**: `bricks.getWidgetById(id, widget)` searches the widget's subtree (children), NOT its parent chain:

```javascript
// getWidgetById starts from `widget` and searches its DESCENDANTS
bricks.getWidgetById = function(id, widget) {
    if (!widget) return null;
    if (widget.id == id) return widget;
    for (var child of widget.children) {  // ← DOWNWARD search
        var w = bricks.getWidgetById(id, child);
        if (w) return w;
    }
    return null;
}
```

To find an ANCESTOR widget (like `sage_main_content`), search from `bricks.app` (the global root):

```javascript
var t = bricks.getWidgetById('sage_main_content', bricks.app);  // ← CORRECT
```

When a button inside `model_plaza.ui` needs to replace the main content area, `getWidgetById('sage_main_content', w)` fails because `w` is the button widget and `sage_main_content` is its ancestor, not descendant.  

Consequence: `urlwidget` actiontype with `target: "sage_main_content"` silently fails because `buildEventHandler` resolves the target via `getWidgetById(target, w)` — same downward search. Workaround: use script actiontype with explicit `bricks.app` root search.

### Bricks: JSON escaping in Jinja2 templates

When embedding JavaScript in `"script"` binds, double quotes inside the script string break JSON:

```json
// WRONG — url:"...?" closes the JSON string prematurely
"script": "... url:\"{{base}}?catelogid={{c.id}}&search='+kw }..."

// RIGHT — use single quotes for inner strings
"script": "... url:'{{base}}?catelogid={{c.id}}&search='+kw }..."
```

### Bricks: RefreshWidget period_seconds=0 = infinite loop

`RefreshWidget.show_widget()` calls `schedule_once(this.show_widget.bind(this), this.opts.period_seconds)`. When `period_seconds` is 0, it re-schedules immediately → infinite reload loop. To disable auto-refresh, omit `period_seconds` entirely or use plain `urlwidget`.

### Bricks: DataViewer/Tabular expects {total, rows} format

DataViewer/PageDataLoader parse the response: `d.total` for pagination math and `d.rows` for the record array. Returning `{total, data}` (wrong key name) causes `d.total = undefined → NaN → infinite paging loop`.

### Bricks: Tabular AJAX may not include session cookies

Tabular/DataViewer make client-side HTTP calls to `data_url`. These AJAX requests may not include the session cookie, causing 401 on endpoints requiring authentication. Prefer server-side Jinja2 rendering for authenticated content.

### ahserver: static fast path overrides virtual paths

`processorResource._handle()` has a fast path for static file extensions (.mp4, .pdf, etc.) that bypasses auth and processor chains. Virtual file paths (managed by FileStorage, not disk files) under prefixes defined in `config.website.startswiths` crash because `StaticResource._handle()` can't find them on disk. Fix: skip fast path when path starts with any `startswiths.leading` prefix. Also, `static_exts` list should be configurable via `config.website.static_exts`.

### ALWAYS test on test server before committing UI changes

**User rule**: never commit UI/web changes without testing them first on the test server. `git push` without testing leads to repeated iteration cycles (committed 5+ times for the same card layout issue). On the test server: `git pull && pip install --upgrade && restart sage`, then verify via browser or curl.

### Stale .pyc / ghost processes blocks code updates

`pip install --upgrade` regenerates `.pyc` files but old worker processes might have the module loaded in memory. When changes don't take effect after install+restart:

```bash
# Kill ALL sage processes (pkill, not just PID files)
pkill -9 -f sage
sleep 2
# Clear all cached bytecode
find ~/sage -path "*/sage_datamart/__pycache__" -exec rm -rf {} \;
# Force reinstall (skip cache)
cd ~/sage/pkgs/sage_datamart && pip install --force-reinstall --no-cache-dir .
# Start fresh
cd ~/sage && ./start.sh
```

### Debug stack-top-first — error path IS the fix location

**User rule**: when an error traceback shows a specific file path in the first frame, start debugging from THAT file. Do NOT chase fixes in other files first.

- `File "/home/token/sage/wwwroot/sage_datamart/sect_top_users.ui", line 1: j2_top_users_data undefined` → check `init.py` registration, NOT `dashboards.py` SQL
- `Exception: /sage_datamart/api/currency_stats.dspy return data type error, NoneType` → check `currency_stats.dspy` format, NOT the Python function

**Anti-pattern**: 3 rounds of try/except and SQL fixes when the DSPY file itself was the wrong format.

### init_data.py: NOT NULL columns need defaults

When batch-inserting from source tables with optional columns, NULL values violate NOT NULL constraints. Add `or ''` or `or 0` defaults:

```python
# WRONG — userorgid can be None in old llmusage records
'userorgid': r.userorgid,

# RIGHT
userorgid = getattr(r, 'userorgid', None)
'userorgid': userorgid or '',
```

Always cross-reference the model JSON for `"nullable": "no"` fields before writing inserts.

### WeChat payment: use H5 endpoint for browser-openable URL

WeChat Native payment (`/v3/pay/transactions/native`) returns `weixin://wxpay/bizpayurl?pr=xxx` — a deep link that browsers CANNOT open. The H5 payment endpoint (`/v3/pay/transactions/h5`) returns `h5_url` which is a normal web URL and works directly in `NewWindow`.

```python
# wechat.py create_payment — use H5, not Native
url_path = "/v3/pay/transactions/h5"   # NOT /v3/pay/transactions/native
body["scene_info"] = {"payer_client_ip": ..., "h5_info": {"type": "Wap"}}
h5_url = ret.get("h5_url")             # NOT code_url
return h5_url                          # directly browser-openable
```

The `recharge.dspy` simply passes the URL to `NewWindow` — no conversion needed:
```python
url = await create_payment(request, params_kw)
return {"widgettype": "NewWindow", "options": {"url": url, ...}}
```

**DO NOT** try to convert `weixin://` to `wx.tenpay.com/f2f` — that URL returns 404.
**DO NOT** generate QR codes for Native payment — that was a workaround for browsers not supporting `weixin://`.

## Servers

| Role | Host | App path | Port | SSH |
|------|------|----------|------|-----|
| Test (Sage) | apitest@tokentest.opencomputing.cn | /d/apitest/sage/ | 9180 | apitest (免密) |
| Test (pipeline) | apitest@120.48.168.15 | /d/apitest/pipeline-app/ | 9090 | sword/Aa@123456 |
| Test (RAG) | rag@rag.opencomputing.cn | /d/rag/ragserver/ | 9181 | rag (免密), venv: py3/bin/python3 |
| Test (RAG, alt user) | apitest@120.48.168.15 | /d/rag/ragserver/ | 9181 | sword/Aa@123456 (same host as pipeline) |
| Prod | token.opencomputing.cn | /home/token/sage/ | — | — |

**Sage test server startup**: `cd /d/apitest/sage && bash start.sh` — starts 4 workers + 1 backend on port 9180.

**Sage production startup**: `cd /home/token/sage && bash start.sh`.

**RAG test server**: process runs as user `rag`. SSHing as `apitest` (same host, password auth) works for git pull, mysql, redis, and curl — but CANNOT `pkill` or restart the process (owned by `rag`). To restart, either SSH directly as `rag` (key auth) or use `su - rag` on the server.

## Sage DSPY Endpoint Verification

After deploying new DSPY endpoints, verify they return correct JSON format with an authenticated session:

```bash
# 1. Login to get session cookie
COOKIE_JAR=/tmp/sage_cookies.txt
rm -f "$COOKIE_JAR"
curl -s -c "$COOKIE_JAR" -b "$COOKIE_JAR" \
  -d "username=sword&password=Aa@123456" \
  "http://127.0.0.1:9180/rbac/user/up_login.dspy?_webbricks_=1"

# 2. Smoke-test all new endpoints
for ep in \
  "llmage/api/plaza_categories.dspy" \
  "llmage/api/plaza_providers.dspy" \
  "llmage/api/get_plaza_models.dspy"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" \
    "http://127.0.0.1:9180/$ep?_webbricks_=1")
  echo "$code $ep"
done
```

**Key points**:
- Login endpoint is `/rbac/user/up_login.dspy` (NOT `/sage/do_login` which always returns 401)
- Use plaintext `password` param — the DSPY calls `password_encode()` internally
- The `_webbricks_=1` query param returns Bricks JSON format, not HTML
- List APIs return `{"total": N, "rows": [...]}`; filter/enum APIs return `{"rows": [{"id","name"},...]}`
- 401 after adding paths to `load_path.py`? → run `redis-cli FLUSHDB`

See `references/dspy-endpoint-verification.md` for full details, including JSON shape expectations, common pitfalls, and multi-endpoint smoke test scripts.

### Audit & Reconcile — absorb direct server edits back into git

When developers bypass the local→commit→push→pull flow and edit files directly on the test server, you need a systematic recovery. See `references/git-audit-reconcile.md` for the full 4-step workflow: audit all repos → scp changes to local → commit+push → clean+sync server.

## Related references

- `references/git-audit-reconcile.md` — Systematic workflow for absorbing direct server edits back into git
- `references/git-umbrella-extraction.md` — Extracting embedded modules from umbrella repos with `git subtree split`
- `references/pipeline-app-deployment.md` — Pipeline-app specific deployment: architecture, restart, HTTPS git, module reconciliation
- `references/sage-datamart-dashboard.md` — Datamart dashboard architecture, role-based views, real-time cards, ETL flow, table schemas
- `references/bricks-tabular-dataviewer.md` — Bricks DataViewer/Tabular response format, cockpit navigation, widget availability
- `references/dspy-endpoint-verification.md` — Authenticated curl testing of DSPY endpoints, login cookie flow, JSON structure verification