yumoqing/pipeline_core

Fork 0

yumoqing 1abaa4ea2e feat: 技能库(skills_library 213个Hermes技能)+skill_pack技能集安装能力+ocai-h5-dev技能集(37个应用/模块开发技能)

2026-08-16 15:17:43 +08:00

35 KiB

Raw Blame History

name

description

Sage Module Deployment

Deployment Flow

repos/ → commit → push → pull on target → pip install --upgrade → restart Sage

IMPORTANT: git pull alone does NOT update the running Python module. The sage_datamart, supplychain, llmage, etc. Python packages are installed in py3/lib/python3.10/site-packages/. After git pull you MUST pip install --upgrade for Sage to see the new code.

Deployment-gap detection — pushed commits are NOT the same as deployed code. Three-way md5 check tells you exactly where the chain broke:

MODULE=llmage; FILE=llmage/llmclient.py
git -C /d/ymq/repos/$MODULE show HEAD:$FILE | md5sum                      # (a) local HEAD
ssh apitest@120.48.168.15 "cd /d/apitest/sage/pkgs/$MODULE && md5sum $FILE"  # (b) server repo
ssh apitest@120.48.168.15 "md5sum /d/apitest/sage/py3/lib/python3.10/site-packages/$FILE"  # (c) running code

(a)≠(b): commits never pulled on server. (b)≠(c): pulled but pip install skipped. Any mismatch = the fix believed live is NOT live; complete pull → pip install → restart → curl verify before reporting. Also git status --short the server repo — .bak files and untracked test pages are evidence of direct server edits.

Commit: cd /d/ymq/repos/<module> && git status — review before git add
Push: git push
Deploy to target:

# On target server (test or prod):
cd ~/repos/<module> && git pull
~/sage/py3/bin/pip install --upgrade ~/repos/<module>
# --upgrade forces overwrite; -e . is not enough for non-editable installs

Restart: see "Server restart" below

Server restart

Both test and production use multiple workers via SO_REUSEPORT. kill $(cat sage.pid) alone leaves orphaned processes.

# Kill ALL sage processes
pkill -9 -f 'sage.py|backend_accounting'
sleep 2

# Restart from Sage root
cd ~/sage && bash start.sh

start.sh checks Redis, starts backend_accounting, then spawns N workers. PID files: sage.pid, sage_backend.pid.

Verify: ps aux | grep -E 'sage|backend' | grep -v grep — should show N workers + 1 backend.

Manual fallback when start.sh is unusable（observed 2026-08-04 on test server: bin/ contained only backend_accounting.py and backup_api.sh, no start.sh/stop.sh）:

pkill -9 -f 'sage.py|backend_accounting'; sleep 2
cd /d/apitest/sage
for i in 1 2 3 4; do
  nohup ./py3/bin/python app/sage.py --workdir /d/apitest/sage --port 9180 >> logs/sage.log 2>&1 &
done
nohup ./py3/bin/python bin/backend_accounting.py >> logs/backend_accounting.log 2>&1 &
# verify: ss -tlnp | grep 9180 ; ps aux | grep -E 'sage.py|backend_accounting' | grep -v grep

All 4 workers share the identical command line (SO_REUSEPORT). bin/backend_accounting.py is the separate process that consumes the llmusage accounting queue — forgetting to restart it leaves stale accounting code running even though all sage workers are fresh. Check both process groups' start times after any restart.

Pitfall — stop.sh may kill nothing: test-server stop.sh can fail with killname: command not found (broken helper) and silently leave the old workers AND old backend_accounting.py running. Then start.sh spawns a second full set; SO_REUSEPORT lets the new workers bind 9180 fine, but the orphaned old backend keeps processing accounting events with stale code. After every restart, compare process start times and kill orphans:

ps -ef | grep -E 'sage.py|backend_accounting' | grep -v grep   # column 5 = start time
# all start times must be from the fresh start; kill older PIDs explicitly

Prefer the pkill-all approach above over trusting stop.sh.

Pitfall — kill -9 on Sage processes kills the SSH session: When running kill -9 on sage worker processes from an SSH session where the worker is handling the SSH connection's traffic (e.g., the SSH goes through nginx→sage on the same host), the command kills the SSH session itself, causing exit code 255 and losing any commands after the kill. Workaround: use nohup bash -c "kill ..." & to run the kill in a detached process, or kill PIDs individually with kill (SIGTERM) first, only using -9 as a last resort. After a kill-induced disconnect, wait 3s then reconnect.

Pitfalls

Module .dspy/.ui 改动只需 git pull 即生效（无需 pip install/重启）

模块的 dspy/ui 文件经 wwwroot/<module> symlink 指向 pkgs/<module>/wwwroot/，每次请求从磁盘读取。对纯 dspy 改动，cd /d/apitest/sage/pkgs/<module> && git pull 之后下一次请求就跑新代码——不需要 pip install，不需要重启。

实证 2026-08-04：llmage 上架联动 hook 的 dspy 修复，在 pkgs/llmage git pull 后下一次 publish 请求立即执行新代码（无重启、无 pip install）。与下方 RAG 节的 live-serve 模式相同。

混合改动的拆分策略（如 dspy hook 在 llmage、Python 函数在 product_management）：dspy 模块 pull 后可立即测试链路；Python 包侧仍需 pip install + 重启。部署前先分清改动落在 .dspy/.ui 还是 .py。

wwwroot symlinks are deployment artifacts, not source

~/repos/sage/wwwroot/ is the Sage source repository. Module wwwroot symlinks (e.g. supplychain -> /d/ymq/repos/supplychain/wwwroot) are created by build.sh during deployment and should NEVER exist in the Sage source repo. They pollute the source directory and confuse git status.

# In ~/repos/sage/wwwroot/ — these are WRONG:
./supplychain -> /home/hermesai/repos/supplychain/wwwroot
./pricing -> /home/hermesai/repos/pricing/wwwroot
./bugfix -> /home/hermesai/test/bugfix/wwwroot
# and ALL module wwwroot symlinks (msp, rag, cpcc, dapi, rbac, ...)

Fix: rm -f <symlink_name> for every module wwwroot symlink. These are deployment concerns — they belong on the running server (created by build.sh), not in the git-tracked source tree.

Exceptions: bricks -> ../../bricks/dist (Bricks UI framework) and shell_theme.* files are Sage core, not module symlinks.

Untracked files on remote blocking git pull

If a directory was created manually on the server (e.g. wwwroot/<new_dir>/) before a commit that adds the same path, git pull aborts with:

error: The following untracked working tree files would be overwritten by merge:
    wwwroot/<new_dir>/...

Fix: rm -rf wwwroot/<new_dir>/ then git pull.

build.sh: second `for m in` line can be accidentally deleted

When removing a module name from the business-module for m in list in build.sh, the entire for m in ... line can be deleted instead of just the module name. This leaves a bare do block without a loop variable. merge_i18n.py (which expects exactly 2 for m in lines) then fails with "错误: 在 build.sh 中找到 1 个 for m in 行，期望至少2个".

# Verify both for m in lines are present:
grep -c '^for m in ' build.sh  # must return 2

Fix: restore the missing for m in <modules...> line before the bare do. Use git show <prior-commit>:build.sh to recover the original list, then remove only the unwanted module name.

`merge_i18n.py`: modules under `pkgs/` need fallback lookup

merge_i18n.py looks for module i18n at REPOS_DIR/<module>/i18n/ (e.g. /d/apitest/bricks/i18n). On test servers, modules are cloned under SAGE_DIR/pkgs/<module>/ not REPOS_DIR/<module>/. The fix: add SAGE_DIR/pkgs/<module>/i18n as a secondary lookup path in find_module_i18n_dir().

Local changes on test server blocking git pull

测试服务器上可能有运行时产生的本地修改（如 wwwroot/index.ui 被部署脚本修改）。git pull 被 local changes would be overwritten 阻止。

# 标准三击：
ssh apitest@tokentest.opencomputing.cn "cd ~/sage && git stash && git pull && git stash pop"

# 若有冲突，接受远端版本并丢弃 stash：
ssh apitest@tokentest.opencomputing.cn "cd ~/sage && git checkout --theirs <file> && git add <file> && git stash drop"

注意: 测试服务器 python3 非 python；DNS 失效时用 IP 120.48.168.15。

`git pull` succeeds but new code still doesn't run

Root cause: pip install was skipped. Git updates the source files on disk, but Sage imports from py3/lib/python3.10/site-packages/. Without pip install --upgrade, the old bytecode in site-packages is still used.

Symptom: Jinja2 errors like 'j2_realtime_calls' is undefined or NameError: name 'api_xxx' is not defined — functions that clearly exist in the source file.

Fix: Always run pip install --upgrade after git pull. Not pip install -e . (which only links), but --upgrade which forces overwrite of the installed package.

~/sage/py3/bin/pip install --upgrade ~/repos/<module>

Bare `pip` silently installs to ~/.local — the venv never sees it

On the test server, bare pip resolves to the SYSTEM pip (/usr/bin/pip) and installs into the user site dir /d/apitest/.local/lib/python3.10/site-packages/ — NOT the venv the service runs from. pip still prints Successfully installed llmage-0.0.1, so the deploy looks fine while the running code stays old. (Observed 2026-08-04: three --force-reinstall runs "succeeded" before anyone checked WHERE they landed.)

which pip; pip --version                        # /usr/bin/pip → system pip, WRONG target
/d/apitest/sage/py3/bin/pip --version           # must show the venv path

# Correct install, then verify by CONTENT, not by pip's success message:
/d/apitest/sage/py3/bin/pip install --force-reinstall --no-deps /d/apitest/sage/pkgs/<module>
grep -c '<new_function_marker>' /d/apitest/sage/py3/lib/python3.10/site-packages/<pkg>/<file>.py

Rule: a deployment is only real after grepping the venv site-packages for markers of the new code. pip output alone proves nothing.

Never modify CRUD intermediate files

wwwroot/<listname>/*.dspy and wwwroot/<listname>/*.ui are generated from json/*.json by xls2ui. Never edit them directly — change the JSON source and regenerate. Never commit them.

If remote already tracks these files, clean up:

git rm -r --cached wwwroot/<listname>/
echo "wwwroot/<listname>/" >> .gitignore
git add .gitignore && git commit -m "chore: remove CRUD intermediate files from tracking"

wwwroot/api/<name>.dspy is exempt — those are independent APIs, not generated.

Never commit egg-info / pycache

git add -A picks up .pyc and *.egg-info/ junk.

Fix: add .gitignore with __pycache__/, *.pyc, *.egg-info/. Always git status before commit. If already tracked on remote, git rm -r --cached, commit, push.

load_path.py crashes on `#` comment lines

paths.split('\n') iterates every line. A comment like # sage_datamart is not empty, so l.split() produces ['#', 'sage_datamart'] → roles=['sage_datamart']. This string is neither any/logined/anonymous nor has a . separator, so role.split('.') in add_roles_perm() raises ValueError: not enough values to unpack (expected 2, got 1).

Fix: skip comment lines:

for l in paths.split('\n'):
    if not l or l.startswith('#'):   # ← add l.startswith('#')
        continue

load_path.py: must EXECUTE after editing — adding paths is not enough

Adding paths to load_path.py and committing is only the first step. The script must be run on the target server to register permissions in the database:

cd ~/sage && python load_path.py

Symptom of missing this step: new endpoints silently return 401/403 (permission check failed in the log) even though the path appears in load_path.py. All requests fail with need login to access.

Also: when replacing old UI paths (e.g., cockpit page replaces list views), update global_menu.ui to point to the new path.

i18n not working — `merge_i18n.py` pipeline (NOT wheel JSON)

Sage i18n does NOT use per-package wheel JSON. Each module has i18n/{lang}/msg.txt at the repo root (e.g. ~/repos/sage_datamart/i18n/zh/msg.txt), NOT under the Python package dir. Format: key: value per line (NOT JSON, NOT key: value with JSON blob on one line). merge_i18n.py in Sage root consolidates all modules into wwwroot/i18n/{lang}/i18n.json which the frontend loads via /i18n_getmsgs.

Symptoms: dashboard shows mixed languages — some labels correct (key=value coincidence), others show English or raw key.

Checklist — three required conditions for i18n to work:

Module has i18n/{zh,en,jp,ko}/msg.txt at repo root in key: value line format
Module listed in build.sh's second for m in loop (line 29) — merge_i18n.py parses this list
python merge_i18n.py run after any msg.txt change to regenerate wwwroot/i18n/

msg.txt format (NOT JSON!):

今日调用笔数: 今日调用笔数
今日交易金额: 今日交易金额

Pitfall — JSON blob in msg.txt: A single-line JSON {"key":"val",...} is silently skipped by parse_msg_txt() which expects colon-delimited lines per entry.

Pitfall — merge order matters: merge_i18n.py processes modules in order; later overwrites earlier. All business modules define common keys (Submit, Reset, Cancel) as English→English in their en/msg.txt, overwriting bricks' zh translations. bricks must be LAST in merge order:

# merge_i18n.py — bricks last so framework strings win
result = ['sage'] + modules + ['bricks']

Pitfall — bricks Tabular strings: The bricks dist has bricks.app.i18n._() calls for "Add record", "Update record", "Clone record", "Delete conform", "Submit", "Reset", "Cancel", "Conform", "Discard". These keys MUST be in bricks/i18n/{lang}/msg.txt. Also fix dynamicaccordion.js hardcoded 'Delete conform' → bricks.app.i18n._('Delete conform').

After fixing msg.txt:

cd ~/sage && python merge_i18n.py   # regenerate wwwroot/i18n/
# No pip install needed — i18n is wwwroot-served, not in Python package

Never hardcode dbname in sqlorContext

# WRONG — hardcoded 'sage' or 'llmage'
DBPools().sqlorContext('sage')
db.sqlorContext('llmage')

# RIGHT — use dynamic dbname
ServerEnv().get_module_dbname('sage')
ServerEnv().get_module_dbname('llmage')
env.get_module_dbname('discount')  # if env = request._run_ns

_get_sor() / _get_dbname() helpers in each module already use get_module_dbname(). Always follow the existing pattern in the module.

DBPools() is NOT a singleton

DBPools() creates a fresh EMPTY instance. Must explicitly assign databases:

db = DBPools()
config = getConfig()
db.databases = config.databases
async with db.sqlorContext(dbname) as sor:
    ...

Never sync local site-packages

Local /d/ymq/repos/sage/py3/lib/python3.10/site-packages/ is irrelevant. Test server runs the real code. Push to git, pull on target. User: "本地的同步 site-packages 没有任何意义，你的测试环境在tokentest而不是本地"

Never scp/sed to deploy

Deploy via git pull + pip install only. No scp individual files, no sed on server.

WeChat amount: yuan → fen (×100)

WeChat amount.total uses fen (分). User enters yuan (元). Must convert:

"total": int(payload["amount"] * 100),  # CORRECT

int(0.01) = 0 which fails WeChat minimum validation.

SQLor: str(limit) gets quoted → LIMIT '5' syntax error

SQLor's ${param}$ substitution quotes string params as SQL strings. LIMIT ${limit}$ with limit = str(5) produces LIMIT '5' — invalid SQL. Pass integers directly:

# WRONG
recs = await sor.sqlExe(sql, {'limit': str(limit)})   # → LIMIT '5'

# RIGHT
recs = await sor.sqlExe(sql, {'limit': limit})          # → LIMIT 5

SQLor: verify column names against model JSON

Never assume column names. users table has orgid, NOT userorgid. Always grep the model JSON before writing SQL:

# WRONG — users table does not have userorgid
"SELECT COUNT(*) FROM users WHERE userorgid = ${org}$"

# RIGHT
"SELECT COUNT(*) FROM users WHERE orgid = ${org}$"

Tables and their columns:

Table	org_id column	notes
`users`	`orgid`
`llmusage`	`userorgid`
`dm_model_call_fact`	`userorgid`
`organization`	`id`	org_type, resellerid

Never use get_sor_context(None, module) — sor.env is None

get_sor_context(env, module) requires a valid env (ServerEnv). Passing sor.env fails because sor objects from sqlExe contexts don't carry env. When all tables share one Sage database, use the passed sor directly:

# WRONG — sor has no .env, get_sor_context(None) → NoneType.get_module_dbname
async with get_sor_context(sor.env if hasattr(sor, 'env') else None, 'accounting') as other_sor:
    rows = await other_sor.sqlExe(...)

# RIGHT — same database, use the existing sor
rows = await sor.sqlExe(sql, ns)

Pipeline-app umbrella repo pattern

pipeline-app is a pure umbrella — it contains only config, app/ entry point, and wwwroot/ symlinks. All modules live as independent git repos under pkgs/. The umbrella repo itself must never contain embedded module source.

# Local structure:
/d/ymq/pipeline/pipeline-app/          # umbrella (pipeline-app.git)
/d/ymq/pipeline/pipeline-app/pkgs/     # independent module clones
/d/ymq/pipeline/pipeline-sdlc/         # also exists as standalone clone for solo dev
/d/ymq/pipeline/pipeline-service/      # same

When a module needs its own repo, extract it via git subtree split (see references/git-audit-reconcile.md). After extraction: add module dir to .gitignore, git rm -r --cached the old directory, clone the new repo into pkgs/, and set relative symlink wwwroot/<module> -> ../pkgs/<module>/wwwroot.

独立模块迁移三件套：r:p 权限 + 建表 + 初始化数据（替代每模块 load_path.py）

独立模块（产品/折扣/定价/支付/短信）迁移/复用到新平台（pipeline-app）时，除 clone + pip install + wwwroot symlink 外，还须补三件套——否则换环境部署又漏，用户会重复纠正：

权限：conf/rp.json（应用仓库定义角色→路径模式）+ scripts/import_rp.py（幂等导入 permission+rolepermission）。替代每模块 scripts/load_path.py（旧机制硬编码 PATHS_ANY/PATHS_LOGINED 角色 + find_sage_root 找 Sage 根，模块耦合 Sage + 角色体系）。
建表：scripts/create_tables.py（models/*.json → json2ddl mysql <models_dir> → DDL 按 ; 分割、跳过 -- 注释行，逐条 await sor.execute(stmt, {})）。
初始化数据：scripts/import_init.py（init/data.json 的 appcodes/appcodes_kv/organization 三段幂等导入，R 检查 + C 插入）。

三个脚本都固化进 build.sh 自动执行。

rp.json 三层分离（用户明确方向：模块声明路径 → 应用仓库定义 r:p → 脚本导入）：

{"roles": {
  "any": ["/rbac/user/login.ui", "/i18n_getmsgs", "/index_tab.js"],
  "logined": ["/pipeline-sdlc/**", "/product_management/**", "/discount/**"],
  "owner.superuser": ["/rbac/**"]
}}

** 前缀直接存 permission.path（不展开成具体路径），RBAC check_roles_path 原生前缀匹配。
permission 表 path 唯一索引（permission_idx1），import_rp 按 path 去重复用 permid；rolepermission 按 roleid+permid 去重。
根目录 .js（如 /index_tab.js）会被 RBAC 拦 401，需在 rp.json any 加路径；模块子目录 .js 因 /模块/** 已有权限所以 200。

关键陷阱：

config.json 密码是 AES 加密的，建表不能用 mysql 命令行（需明文密码），必须 DBPools + sor.execute（DBPools 内部解密）。
json2ddl 生成的 DDL 含 DROP TABLE IF EXISTS + CREATE TABLE，幂等（表不存在时 DROP 无害）。
load_xxx() 不是导入数据——它注册模块函数到 ServerEnv（让 .dspy 调模块函数）；init/data.json 才是字典数据导入。漏 load_xxx → DSPY NameError 500；漏 init/data.json → appcodes 字典空。
CRUD 脚本（json/*.json + models → xls2ui → wwwroot）通常 repo 已提交，symlink 即部署，不用重新生成。

git.opencomputing.cn SSH rate limit on test server

The test server (120.48.168.15) sometimes hits SSH rate limiting (kex_exchange_identification). Fall back to HTTPS for git pull on the server:

git pull https://git.opencomputing.cn/yumoqing/<repo>.git main

SSH works fine from the dev machine — only change the server-side pull command.

JSON-format DSPY ({"python": {"import": "...", "call": "..."}}) throws return data type error, <class 'NoneType'> when the called function returns None (e.g., get_sor_context fails silently). Python script format (import json; data = await ...; return json.dumps(data)) never has this problem because json.dumps([]) always returns a valid JSON string.

# WRONG — JSON format DSPY, NoneType on any exception
{
    "python": {
        "method": "get",
        "import": "sage_datamart.init",
        "call": "api_currency_stats"
    }
}

# RIGHT — Python script format, always returns valid JSON
import json
env = request._run_ns
data = await env.api_currency_stats(request)
return json.dumps(data, ensure_ascii=False)

Symptom: return data type error, <class 'NoneType'> in the traceback. Fix: convert ALL DSPY files to Python script format with json.dumps().

RAG Server Deployment

The RAG server (ragserver) deploys differently from Sage modules because .dspy/.ui files are served live from the wwwroot filesystem — no Python module import or pip install is involved.

Deployment flow

# 1. Git pull the wwwroot files
cd /d/rag/ragserver/pkgs/rag && git pull

# 2. (If new endpoint) Add RBAC permission + rolepermission
# Check if permission already exists first:
mysql -h db -u test -ptest123 rag -e \
  "SELECT p.*, rp.roleid FROM permission p LEFT JOIN rolepermission rp ON rp.permid=p.id WHERE p.path LIKE '%storage_stats%'"

# If it doesn't exist:
# - permission.id is typically a short code like "perm_rag_storage_stats"
# - permission.path is the URL path, e.g. "/rag/knowledge_bases_list/storage_stats.dspy"
mysql -h db -u test -ptest123 rag -e \
  "INSERT INTO permission (id, name, path) VALUES ('perm_rag_<name>', 'RAG', '/rag/knowledge_bases_list/<name>.dspy')"
# Link to 'any' role:
mysql -h db -u test -ptest123 rag -e \
  "INSERT INTO rolepermission (id, roleid, permid) VALUES ('rp_<name>', 'any', 'perm_rag_<name>')"

# 3. Flush Redis cache so RBAC is re-read
redis-cli FLUSHDB

# 4. Restart is OPTIONAL for .dspy/.ui changes only
# .dspy files are read from disk on each HTTP request — git-pulled changes go live instantly.
# Only restart if Python backend code (app/*.py) changed.

# 5. Verify
curl -s -w '\nHTTP_CODE:%{http_code}' \
  'http://127.0.0.1:9181/rag/knowledge_bases_list/<name>.dspy?_webbricks_=1'

Key differences from Sage

Aspect	Sage module	RAG server
Code location	Installed in site-packages via pip	Read from wwwroot filesystem
After git pull	Must `pip install --upgrade`	No pip install needed
Restart needed	Always (Python modules cached in memory)	Only for app/*.py changes
URL prefix	Configurable per module	`/rag/` (fixed)
wwwroot symlink	Multiple `wwwroot/<module> -> pkgs/<module>/wwwroot`	Single `wwwroot/rag -> ../pkgs/rag/wwwroot`

Process management

RAG server runs as user rag, not apitest
SSHing as apitest@120.48.168.15 (password: sword/Aa@123456) can read/write files, run mysql, redis, and curl — but CANNOT kill/restart the process:
```
pkill -f ragserver        # fails: Operation not permitted (wrong user)
```
To restart, either:
- SSH directly as rag (key-based auth)
- Use su - rag on the server (requires rag user's password)

The process is started with:

cd /d/rag/ragserver && source py3/bin/activate && setsid python -B app/ragserver.py

Port: 9181 (loopback only; Sage frontend on port 9180 proxies /rag/ to it)

RBAC permission patterns

RAG permissions follow the Sage RBAC model (same database). Common patterns:

permission.path	purpose
`/rag/knowledge_bases_list/storage_stats.dspy`	API endpoint (.dspy)
`/rag/knowledge_bases_list/index.ui`	UI page
`/knowledge_bases_list/upload.dspy`	Without `/rag` prefix (older style)

Check existing rolepermission for 'any' role:

mysql -h db -u test -ptest123 rag -e "SELECT * FROM rolepermission WHERE roleid='any'"

Verification

curl -s -w '\nHTTP_CODE:%{http_code}' \
  'http://127.0.0.1:9181/rag/knowledge_bases_list/storage_stats.dspy?_webbricks_=1'

Expected: HTTP 200 with a Bricks widget JSON response (VBox/HBox/Text etc.).

To check if a .dspy endpoint is referenced in the UI:

grep 'storage_stats' pkgs/rag/wwwroot/knowledge_bases_list/index.ui

Expected: a urlwidget or actiontype entry with "url": "{{entire_url('./storage_stats.dspy')}}".

Redis usage

FLUSHDB clears cached RBAC permissions and session data
Always run after adding/changing permissions
Does NOT affect active HTTP connections (sessions re-authenticate on next request)

Pitfalls (continued)

Bricks: getWidgetById searches DOWNWARD (children), not UPWARD (ancestors)

Critical pitfall: bricks.getWidgetById(id, widget) searches the widget's subtree (children), NOT its parent chain:

// getWidgetById starts from `widget` and searches its DESCENDANTS
bricks.getWidgetById = function(id, widget) {
    if (!widget) return null;
    if (widget.id == id) return widget;
    for (var child of widget.children) {  // ← DOWNWARD search
        var w = bricks.getWidgetById(id, child);
        if (w) return w;
    }
    return null;
}

To find an ANCESTOR widget (like sage_main_content), search from bricks.app (the global root):

var t = bricks.getWidgetById('sage_main_content', bricks.app);  // ← CORRECT

When a button inside model_plaza.ui needs to replace the main content area, getWidgetById('sage_main_content', w) fails because w is the button widget and sage_main_content is its ancestor, not descendant.

Consequence: urlwidget actiontype with target: "sage_main_content" silently fails because buildEventHandler resolves the target via getWidgetById(target, w) — same downward search. Workaround: use script actiontype with explicit bricks.app root search.

Bricks: JSON escaping in Jinja2 templates

When embedding JavaScript in "script" binds, double quotes inside the script string break JSON:

// WRONG — url:"...?" closes the JSON string prematurely
"script": "... url:\"{{base}}?catelogid={{c.id}}&search='+kw }..."

// RIGHT — use single quotes for inner strings
"script": "... url:'{{base}}?catelogid={{c.id}}&search='+kw }..."

Bricks: RefreshWidget period_seconds=0 = infinite loop

RefreshWidget.show_widget() calls schedule_once(this.show_widget.bind(this), this.opts.period_seconds). When period_seconds is 0, it re-schedules immediately → infinite reload loop. To disable auto-refresh, omit period_seconds entirely or use plain urlwidget.

Bricks: DataViewer/Tabular expects {total, rows} format

DataViewer/PageDataLoader parse the response: d.total for pagination math and d.rows for the record array. Returning {total, data} (wrong key name) causes d.total = undefined → NaN → infinite paging loop.

Bricks: Tabular AJAX may not include session cookies

Tabular/DataViewer make client-side HTTP calls to data_url. These AJAX requests may not include the session cookie, causing 401 on endpoints requiring authentication. Prefer server-side Jinja2 rendering for authenticated content.

ahserver: static fast path overrides virtual paths

processorResource._handle() has a fast path for static file extensions (.mp4, .pdf, etc.) that bypasses auth and processor chains. Virtual file paths (managed by FileStorage, not disk files) under prefixes defined in config.website.startswiths crash because StaticResource._handle() can't find them on disk. Fix: skip fast path when path starts with any startswiths.leading prefix. Also, static_exts list should be configurable via config.website.static_exts.

ALWAYS test on test server before committing UI changes

User rule: never commit UI/web changes without testing them first on the test server. git push without testing leads to repeated iteration cycles (committed 5+ times for the same card layout issue). On the test server: git pull && pip install --upgrade && restart sage, then verify via browser or curl.

Stale .pyc / ghost processes blocks code updates

pip install --upgrade regenerates .pyc files but old worker processes might have the module loaded in memory. When changes don't take effect after install+restart:

# Kill ALL sage processes (pkill, not just PID files)
pkill -9 -f sage
sleep 2
# Clear all cached bytecode
find ~/sage -path "*/sage_datamart/__pycache__" -exec rm -rf {} \;
# Force reinstall (skip cache)
cd ~/sage/pkgs/sage_datamart && pip install --force-reinstall --no-cache-dir .
# Start fresh
cd ~/sage && ./start.sh

Debug stack-top-first — error path IS the fix location

User rule: when an error traceback shows a specific file path in the first frame, start debugging from THAT file. Do NOT chase fixes in other files first.

File "/home/token/sage/wwwroot/sage_datamart/sect_top_users.ui", line 1: j2_top_users_data undefined → check init.py registration, NOT dashboards.py SQL
Exception: /sage_datamart/api/currency_stats.dspy return data type error, NoneType → check currency_stats.dspy format, NOT the Python function

Anti-pattern: 3 rounds of try/except and SQL fixes when the DSPY file itself was the wrong format.

init_data.py: NOT NULL columns need defaults

When batch-inserting from source tables with optional columns, NULL values violate NOT NULL constraints. Add or '' or or 0 defaults:

# WRONG — userorgid can be None in old llmusage records
'userorgid': r.userorgid,

# RIGHT
userorgid = getattr(r, 'userorgid', None)
'userorgid': userorgid or '',

Always cross-reference the model JSON for "nullable": "no" fields before writing inserts.

WeChat payment: use H5 endpoint for browser-openable URL

WeChat Native payment (/v3/pay/transactions/native) returns weixin://wxpay/bizpayurl?pr=xxx — a deep link that browsers CANNOT open. The H5 payment endpoint (/v3/pay/transactions/h5) returns h5_url which is a normal web URL and works directly in NewWindow.

# wechat.py create_payment — use H5, not Native
url_path = "/v3/pay/transactions/h5"   # NOT /v3/pay/transactions/native
body["scene_info"] = {"payer_client_ip": ..., "h5_info": {"type": "Wap"}}
h5_url = ret.get("h5_url")             # NOT code_url
return h5_url                          # directly browser-openable

The recharge.dspy simply passes the URL to NewWindow — no conversion needed:

url = await create_payment(request, params_kw)
return {"widgettype": "NewWindow", "options": {"url": url, ...}}

DO NOT try to convert weixin:// to wx.tenpay.com/f2f — that URL returns 404. DO NOT generate QR codes for Native payment — that was a workaround for browsers not supporting weixin://.

Servers

Role	Host	App path	Port	SSH
Test (Sage)	apitest@tokentest.opencomputing.cn	/d/apitest/sage/	9180	apitest (免密)
Test (pipeline)	apitest@120.48.168.15	/d/apitest/pipeline-app/	9090	sword/Aa@123456
Test (RAG)	rag@rag.opencomputing.cn	/d/rag/ragserver/	9181	rag (免密), venv: py3/bin/python3
Test (RAG, alt user)	apitest@120.48.168.15	/d/rag/ragserver/	9181	sword/Aa@123456 (same host as pipeline)
Prod	token.opencomputing.cn	/home/token/sage/	—	—

Sage test server startup: cd /d/apitest/sage && bash start.sh — starts 4 workers + 1 backend on port 9180.

Sage production startup: cd /home/token/sage && bash start.sh.

RAG test server: process runs as user rag. SSHing as apitest (same host, password auth) works for git pull, mysql, redis, and curl — but CANNOT pkill or restart the process (owned by rag). To restart, either SSH directly as rag (key auth) or use su - rag on the server.

Sage DSPY Endpoint Verification

After deploying new DSPY endpoints, verify they return correct JSON format with an authenticated session:

# 1. Login to get session cookie
COOKIE_JAR=/tmp/sage_cookies.txt
rm -f "$COOKIE_JAR"
curl -s -c "$COOKIE_JAR" -b "$COOKIE_JAR" \
  -d "username=sword&password=Aa@123456" \
  "http://127.0.0.1:9180/rbac/user/up_login.dspy?_webbricks_=1"

# 2. Smoke-test all new endpoints
for ep in \
  "llmage/api/plaza_categories.dspy" \
  "llmage/api/plaza_providers.dspy" \
  "llmage/api/get_plaza_models.dspy"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" \
    "http://127.0.0.1:9180/$ep?_webbricks_=1")
  echo "$code $ep"
done

Key points:

Login endpoint is /rbac/user/up_login.dspy (NOT /sage/do_login which always returns 401)
Use plaintext password param — the DSPY calls password_encode() internally
The _webbricks_=1 query param returns Bricks JSON format, not HTML
List APIs return {"total": N, "rows": [...]}; filter/enum APIs return {"rows": [{"id","name"},...]}
401 after adding paths to load_path.py? → run redis-cli FLUSHDB

See references/dspy-endpoint-verification.md for full details, including JSON shape expectations, common pitfalls, and multi-endpoint smoke test scripts.

Audit & Reconcile — absorb direct server edits back into git

When developers bypass the local→commit→push→pull flow and edit files directly on the test server, you need a systematic recovery. See references/git-audit-reconcile.md for the full 4-step workflow: audit all repos → scp changes to local → commit+push → clean+sync server.

references/git-audit-reconcile.md — Systematic workflow for absorbing direct server edits back into git
references/git-umbrella-extraction.md — Extracting embedded modules from umbrella repos with git subtree split
references/pipeline-app-deployment.md — Pipeline-app specific deployment: architecture, restart, HTTPS git, module reconciliation
references/sage-datamart-dashboard.md — Datamart dashboard architecture, role-based views, real-time cards, ETL flow, table schemas
references/bricks-tabular-dataviewer.md — Bricks DataViewer/Tabular response format, cockpit navigation, widget availability
references/dspy-endpoint-verification.md — Authenticated curl testing of DSPY endpoints, login cookie flow, JSON structure verification

35 KiB Raw Blame History Unescape Escape