pipeline_core/skills_library/all/ragserver-development/SKILL.md

name	description	tags
ragserver-development	Develop the ragserver RAG application — models, DSPY pages, API handlers, Tabular CRUD, RBAC, and tag management patterns.	rag ahserver dspy bricks rbac sage

RagServer Development

• references/media-tag-popup-pattern.md — face/voice gallery popup (media_cards.dspy), tag dialog (tag_form.dspy), add_tag.dspy contract, file_type='other' pitfall (filter by extension), RBAC rows needed for every new .dspy

Module scope: .ui pages must not reference /api/ routes (user rule)

A module's wwwroot pages must be self-contained. Calling server-level /api/... endpoints from module .ui files (e.g. entire_url('/api/kb/list') in index.ui) is "超出了模块的范围" — rejected in review. The /api/* routes live in ragserver conf/config.json (startswiths mapping) pointing at handlers in rag/init.py; the module must not depend on that server-level wiring.

Fix pattern: move the handler logic into a module-local .dspy and point the urlwidget at it:

// WRONG — depends on ragserver-level /api/ route
"url": "{{entire_url('/api/kb/list')}}"
// RIGHT — module-local dspy (port the init.py handler logic, no f-strings)
"url": "{{entire_url('./kb_list.dspy')}}"

Then grant RBAC for the new .dspy or it 403s: INSERT into permission (path = /rag/knowledge_bases_list/<name>.dspy) + rolepermission (roleid='any', matching sibling dspys like storage_stats), and restart the service — RBAC perms are cached, the new path stays 403 until restart. Idempotent SQL: INSERT ... SELECT ... FROM DUAL WHERE NOT EXISTS (...). Verify with login cookie: curl -b "$CK" $HOST/rag/knowledge_bases_list/<name>.dspy should return widget JSON, not 403: Forbidden.

Search quality debugging (results return but are irrelevant)

When search returns hits but none are relevant, do NOT debug the search code first — verify the data:

1. Log incoming params server-side: temporarily add info(f'[handler] params_kw={params_kw}') at the top of the result DSPY, deploy, and have the user reproduce. The log line is ground truth for what the frontend actually sends (catches [object Object], wrong param names, double-encoding). Log signature decoding:
   • kb_id='%5Bobject%20Object%5D' → frontend stringified a JS object into the URL. In bricks, UiType.getValue() returns an OBJECT — script binds must use resultValue() for URL params (see bricks-widget-development skill, "UiType getValue() returns an object" section).
   • kb_id='all' or other fallback value → getWidgetById failed to find the widget (sibling lookup without bricks.app second arg).
   • query looks double-encoded (ai%20o%20http) or UI text the user reports does NOT exist in the server .ui file → stale browser cache; user must Ctrl+F5 before further debugging.
2. Direct DB recall audit: if params are clean but recall is irrelevant, verify the KB actually contains the term by querying document_chunks directly (LIKE '%term%') and count matches. DB password is AES-encrypted — decrypt with appPublic.aes.aes_decode_b64(password_key, enc), NOT rc4 (unpassword is the wrong path). Full recipe + hybrid-search fix pattern (keyword LIKE recall merged with vector results, kw hits scored 0.99 + 📌 badge) in references/hybrid-search-and-db-diagnostics.md. Also note: a brand-new .dspy file returns 403 until RBAC permission is granted — for throwaway diagnostics use a standalone python script instead of a temp .dspy.
3. Check what's actually in the KB: list documents via the file_list endpoint (or SELECT file_name FROM documents WHERE kb_id=...), then grep -ril "<term>" the raw files in the storage dir (e.g. /d/rag/ragserver/files/). If no document contains the query term, no recall is possible — that's not a bug, tell the user to ingest real content.
4. Embedding model consistency: confirm the ingest DSPY (upload_file/batch_ingest) and the query DSPY hit the same embedding endpoint with the same model name. Mismatched models = meaningless scores.
5. Score-band clustering = degenerate model: if top-K scores all cluster in a narrow band (e.g. everything 62–64%), the embedding model can't discriminate this corpus and ranking is noise. CLIP-ViT-H-14 is a vision-language model and performs poorly for text-only retrieval — prefer a dedicated text embedding model.
6. Reverse probe: search for a term unique to a known ingested document. If that document doesn't rank top, recall is broken regardless of the user's query.

Stale browser cache trap: .ui files are static and served live (no restart needed after git pull). If the user describes UI text that doesn't exist in the current source (grep for it), the browser is showing a cached old page. Verify the served version with curl -s http://localhost:PORT/path/to/x.ui | grep <marker>; if served content is correct, have the user Ctrl+F5 / incognito. Don't re-fix code that is already right.

User preferences

• Test before asking: Curl-test every new DSPY/endpoint from the server itself before declaring it done. Do NOT deploy and assume — verify with actual HTTP requests. Use scripts/test_rag.sh as baseline.
• Testing is MANDATORY: Every change must be tested. Failing to test is a process failure. The smoke test must pass (0 errors) before considering work complete.
• Just do it: Do NOT ask the user to perform actions on servers. You have SSH access — do it yourself.
• Prefer DSPY over registerfunction: Use .dspy files for new features; minimize init.py/config.json changes. DSPY files benefit from hot_reload.
• Code always locally, then push: Never manually edit files on the server. Edit locally → git commit → git push → server git pull. This prevents lost work from git pull overwrites.
• Deploy to the checkout the running process actually uses: servers can hold multiple ragserver checkouts (e.g. /d/apitest/ragserver vs /d/rag/ragserver). Pulling/restarting the wrong one leaves the running instance untouched. Verify first: pid=$(pgrep -f ragserver.py); readlink /proc/$pid/cwd — then deploy under THAT cwd. Real incident: a fix was pulled + smoke-tested in the wrong checkout and declared working while the running instance (different user/path) still served old code.
• Deploy semantics — static vs dspy: .ui/static files are served fresh from disk right after git pull (verify with curl .../xxx.ui | grep <marker> — no restart needed). .dspy changes require a restart (bash stop.sh && bash start.sh).
• Browser cache hides frontend fixes: if the user still sees a bug after a verified frontend deploy, grep the user-visible literal text (button labels, placeholders) in the server's .ui files — if it's not there, the browser cached an old page; tell the user to Ctrl+F5 / use incognito.

Test Server

Host	User	Path	Port	Notes
rag.opencomputing.cn	rag (免密SSH)	/d/rag/ragserver	9181	venv: py3/bin/python, DB: rag(test/test123), admin/admin123

Architecture

rag (business module) → yumoqing/rag.git
  setup.py / pyproject.toml
  rag/                  → Python package (__init__.py + init.py with handlers)
  wwwroot/              → module pages (knowledge_bases_list/, engine_configs_list/, subscriptions_list/)
  json/                 → CRUD definitions
  models/               → DB model JSON files
  i18n/                 → i18n translations (optional)

ragserver (frontend shell) → yumoqing/ragserver.git
  app/ragserver.py      → Entry point — imports rag.init.init_rag_module()
  conf/config.json      → Routes, DB config, hot_reload
  wwwroot/              → Shell pages (index.ui, top.ui, menu.ui, shell_theme.css, user_menu.ui, i18n/)
  build.sh              → Clones rag module, symlinks wwwroot, builds bricks, generates DDL
  scripts/init_rbac.py  → Permission seeding
  start.sh / stop.sh    → Service management

The business module's wwwroot is symlinked into the shell's wwwroot by build.sh. Module and app are separate git repos.

Models

Model JSON files define DB tables used by xls2ddl and Sage ORM. Format:

{
    "summary": [{"name": "table_name", "title": "显示名", "primary": ["id"]}],
    "fields": [
        {"name": "id", "title": "ID", "type": "str", "length": 32},
        {"name": "org_id", "title": "所属机构", "type": "str", "length": 32},
        {"name": "created_at", "title": "创建时间", "type": "datetime"}
    ],
    "codes": [
        {"field": "kb_id", "table": "knowledge_bases", "valuefield": "id", "textfield": "name"}
    ],
    "indexes": [
        {"name": "idx_name", "idxtype": "index", "idxfields": ["col"]},
        {"name": "idx_unique", "idxtype": "unique", "idxfields": ["col1", "col2"]}
    ]
}

API Handlers

In init.py, handlers are async functions registered via RegisterFunction:

async def my_handler(request, params_kw, *args, **kwargs):
    env = request._run_ns
    try:
        userorgid = await env.get_userorgid()
        async with get_sor_context(env, 'rag') as sor:
            recs = await sor.R("table", {"org_id": userorgid})
            return json.dumps({"status": "SUCCEEDED", "data": [...]}, ensure_ascii=False, default=str)
    except Exception as e:
        exception(f"my_handler: {e}, {format_exc()}")
        return json.dumps({"error": str(e)})

def init_ragserver():
    env = ServerEnv()
    rf = RegisterFunction()
    rf.register("my_handler", my_handler)

Routes are mapped in conf/config.json under website.startswiths:

{"leading": "/api/my/endpoint", "registerfunction": "my_handler"}

DSPY Pages (Full-Page Widget JSON)

DSPY files that generate complete page widgets use DBPools() + sqlorContext + import:

ns = params_kw.copy()
kb_id = ns.get('kb_id', '')
import json, uuid

db = DBPools()
dbname = get_module_dbname('rag')

async with db.sqlorContext(dbname) as sor:
    recs = await sor.R('tags', {'kb_id': kb_id})
    tags = [{"id": r.id, "name": r.name, "color": r.color} for r in recs]

result = {
    "widgettype": "VBox",
    "options": {"cheight": 40, "width": "100%", "padding": "16px", "spacing": "12px"},
    "subwidgets": [
        {"widgettype": "Text", "options": {"text": "Title", "cfontsize": 20, "fontWeight": "bold"}},
        # ...widgets...
    ]
}
result  # Last expression is the return value

Key pattern: ahserver wraps .dspy code in async def myfunc(request, **ns): + await func(). You MUST use explicit return result (or return from inside async with block). A bare result as the last expression does NOT work. See dspy-file-implementation-spec for the ahserver execution model.

CRITICAL: The request._run_ns.func() pattern and ServerEnv registration (env.func = func) both return NoneType in DSPY context. ServerEnv merges inconsistently — the function may be set correctly on the ServerEnv singleton but unavailable in the DSPY execution namespace. The ONLY reliable zero-import pattern is inline all logic directly in the DSPY file: no external imports, no ServerEnv function calls. Use only ahserver pre-loaded globals (json, uuid, DBPools, get_sor_context, request.read(), params_kw).

Working DSPY upload pattern (confirmed smoke test):

ns = params_kw.copy()
kb_id = ns.get('kb_id', '')
folder_id = ns.get('folder', '')
file_name = ns.get('file_name', 'upload.bin')

if not kb_id:
    return json.dumps({"status": "error", "error": "kb_id required"}, ensure_ascii=False)

file_data = await request.read()
if not file_data:
    return json.dumps({"status": "error", "error": "no file data"}, ensure_ascii=False)

env = request._run_ns
userorgid = await env.get_userorgid()
doc_id = str(uuid()).replace('-', '')[:16]
ext = '.' + file_name.rsplit('.', 1)[1] if '.' in file_name else '.bin'
saved_name = doc_id + ext
file_path = '/d/rag/ragserver/pkgs/rag/rag/files/' + saved_name
with open(file_path, "wb") as f:
    f.write(file_data)
file_size = len(file_data)

async with get_sor_context(env, 'rag') as sor:
    await sor.sqlExe(
        "INSERT INTO documents (id, kb_id, folder_id, file_name, file_type, file_size, file_path, mime_type, status, org_id, created_at, updated_at) "
        "VALUES (${id}$, ${kb_id}$, ${folder_id}$, ${file_name}$, 'other', ${file_size}$, ${file_path}$, 'application/octet-stream', 'done', '', ${org_id}$, NOW(), NOW())",
        {"id": doc_id, "kb_id": kb_id, "folder_id": folder_id, "file_name": file_name,
         "file_size": file_size, "file_path": "/idfile/files/" + saved_name,
         "org_id": userorgid})
    await sor.sqlExe(
        "UPDATE knowledge_bases SET doc_count=doc_count+1, total_size=total_size+${size}$ WHERE id=${kb_id}$",
        {"size": file_size, "kb_id": kb_id})

return json.dumps({"status": "SUCCEEDED", "doc_id": doc_id, "file_name": file_name,
    "file_size": file_size, "folder_id": folder_id}, ensure_ascii=False, default=str)

0 imports, all pre-loaded functions. Files directory hardcoded — os/__file__ not available.

GPU service endpoints (port 10443):

• Face detect: POST https://media.opencomputing.net:10443/face/api/detect with {"images": ["base64..."]} (array, NOT "image")
• Voiceprint: POST https://media.opencomputing.net:10443/voiceprint/extract/submit (multipart file upload, synchronous)

Voiceprint service startup:

cd /share/ymq/run/voiceprint
pip install aiohttp_auth aiohttp_cors aiohttp_middlewares openpyxl rsa qrcode asyncssh
# Fix imports: sed -i 's|from longtasks.longtasks import|from longtasks.longtasks.longtasks import|' vp/engine.py
# Auth bypass: add 'return' at start of setupAuth() in ahserver/ahserver/auth_api.py
# Auth bypass: replace 'self.user = await auth.get_auth(request)' with 'self.user = None'
PYTHONPATH=.:sqlor:appPublic:ahserver:longtasks/longtasks nohup python3 ah.py -p 9087 > logs/vp.log 2>&1 &
curl http://localhost:9087/api/status  # → {"ready": true}

RBAC permission cache: After ANY permission change (SQL INSERT, rbac script), flush Redis and restart:

redis-cli FLUSHDB
kill $(lsof -ti:9181); sleep 2
cd /d/rag/ragserver && source py3/bin/activate && setsid py3/bin/python -B app/ragserver.py &

Widget navigation: Use urlwidget binds to reload the same DSPY with different params:

base_url = entire_url('/my_page/index.dspy')
tag_buttons.append({
    "widgettype": "Button",
    "options": {"label": t['name'], "bgcolor": bg, "color": clr, ...},
    "binds": [{
        "wid": "self", "event": "click",
        "actiontype": "urlwidget",
        "target": "app.rag_main_content",
        "mode": "replace",
        "options": {"url": base_url + '?kb_id=' + kb_id + '&tag_id=' + t['id']}
    }]
})

Tabular CRUD Pattern

For list management pages, use the Tabular widget with DSPY-backed CRUD:

wwwroot/<name>_list/index.ui — Tabular widget with editable config:

{
    "widgettype": "Tabular",
    "options": {
        "dataurl": "{{entire_url('/<name>_list/data.dspy')}}",
        "title": "列表",
        "browserfields": [
            {"field": "name", "title": "名称", "width": "25%"},
            {"field": "created_at", "title": "时间", "width": "30%"}
        ],
        "editable": {
            "fields": [
                {"field": "name", "title": "名称", "uitype": "str", "label": "名称", "required": true}
            ],
            "add_url": "{{entire_url('/<name>_list/add.dspy')}}",
            "update_url": "{{entire_url('/<name>_list/update.dspy')}}",
            "delete_url": "{{entire_url('/<name>_list/delete.dspy')}}"
        }
    }
}

data.dspy — Returns record list:

ns = params_kw.copy()
db = DBPools()
dbname = get_module_dbname('rag')
async with db.sqlorContext(dbname) as sor:
    recs = await sor.R('table_name', {})
    return [{"id": r.id, "name": r.name, "created_at": str(r.created_at)} for r in recs]
return []

add.dspy — Creates record, returns Message widget:

ns = params_kw.copy()
db = DBPools()
dbname = get_module_dbname('rag')
async with db.sqlorContext(dbname) as sor:
    new_id = uuid()
    await sor.sqlExe("INSERT INTO table_name (id, name, created_at) VALUES (${id}$, ${name}$, NOW())",
                     {"id": new_id, "name": ns.get('name', '')})
    return {"widgettype": "Message", "options": {"user_data": {"id": new_id, "name": ns.get('name', '')}}}
return {"error": "failed"}

delete.dspy — Cascade delete if needed:

ns = params_kw.copy()
db = DBPools()
dbname = get_module_dbname('rag')
async with db.sqlorContext(dbname) as sor:
    await sor.sqlExe("DELETE FROM child_table WHERE parent_id=${id}$", {"id": ns.get('id', '')})
    await sor.sqlExe("DELETE FROM table_name WHERE id=${id}$", {"id": ns.get('id', '')})
    return {"widgettype": "Message", "options": {"user_data": {"id": ns.get('id', '')}}}
return {"error": "failed"}

KB Rename / Delete Patterns

Full reference: references/kb-crud-patterns.md — rename via popup form (two-file pattern: form+handler), cascade delete order (chunks→media_tags→tags→docs→physical files→kb), NULL org_id handling (OR org_id IS NULL), JS confirm()+fetch+window.location.href reload, physical file deletion via env.realpath()+os.remove().

Polymorphic Tag Pattern

Full reference: references/tag-management.md — DB schema, API endpoints, UI pages, deployment steps.

For tagging different entity types (documents, faces, voices), use a media_tags association table:

CREATE TABLE media_tags (
    id VARCHAR(32) PRIMARY KEY,
    kb_id VARCHAR(32) NOT NULL,
    media_type VARCHAR(32) NOT NULL,  -- 'document' | 'face' | 'voice'
    media_id VARCHAR(32) NOT NULL,
    tag_id VARCHAR(32) NOT NULL,
    UNIQUE (media_type, media_id, tag_id)
);

Tag assignment uses URL-driven toggle buttons — click to assign, click again to unassign. The DSPY handles both actions in one file by reading ?action=assign|unassign&tag_id=X from query params.

Tag search supports match_mode=any|all (OR/AND) for combining multiple tags.

media_type values: Use document, face, voice — do NOT split document into text/image/audio/video. The documents table already has file_type for that distinction.

RBAC Initialization

Use scripts/init_rbac_v2.py (not the old init_rbac.py). Key improvements:

• Auto-scans pkgs/rag/wwwroot/ for ALL .dspy/.ui files — no manual path listing
• Uses md5-based short permission IDs (12 chars) — avoids varchar(32) truncation
• Idempotent: safe to run after every deployment
• Business pages → logined role; shell/infra → any role

Run after any file change: cd /d/rag/ragserver && py3/bin/python3 scripts/init_rbac_v2.py

Then verify with the smoke test: bash scripts/test_rag.sh

Testing Discipline (MANDATORY)

Every change must be tested before declaring done. The smoke test (scripts/test_rag.sh) covers:

• Public endpoints accessible without login
• KB pages return 401 without login (not 403, not 200)
• All core DSPY endpoints return 200 after login
• Storage card shows real data (>0MB, not hardcoded "0MB")

After deployment: git pull → init_rbac_v2.py → restart → test_rag.sh

Never skip this. "你的测试没做吗？不能这样，必须做测试"

Menu Configuration

wwwroot/menu.ui uses Jinja2 + bricks Menu widget:

{% set roles = get_user_roles(get_user()) %}
{% set role_str = roles|join(',') %}
{
    "widgettype": "Menu",
    "id": "rag_menu",
    "options": {
        "items": [
            {"name": "section", "label": "名称", "icon": "{{entire_url('/bricks/imgs/icon.svg')}}",
             "url": "{{entire_url('/section/index.ui')}}", "target": "app.rag_main_content"}
        ]
    }
}

Available icons are in wwwroot/bricks/imgs/ (symlinked to bricks dist). Common: app.svg, tag.png, edit.svg, search.svg, rocket.svg, coins.svg.

Bricks Widget Patterns (Learned)

id at widget level, NOT in options

For global app.<id> references, id goes on the widget object, not inside options:

// ✅ CORRECT — bricks registers this as app.rag_main_content
{"widgettype": "VBox", "id": "rag_main_content", "options": {"css": "filler"}}

// ❌ WRONG — get_by_id() returns null
{"widgettype": "VBox", "options": {"id": "rag_main_content", "css": "filler"}}

🔴 widgettype "code" — use "UiCode" instead (NOT registered in bricks.Factory)

The "code" widgettype is registered only in bricks.uitypesdef (ViewBuilder registry) and Input.uitypes (form input registry), but NOT in bricks.Factory (Widget class registry). widgetBuild() calls bricks.Factory.get(desc.widgettype) — which returns null for "code". The widget is silently skipped: no DOM element is created, no error is shown.

bricks.UiCode (a <select> dropdown) IS registered in bricks.Factory via Input.register('UiCode', 'code', bricks.UiCode). The correct widgettype for a top-level dropdown is "UiCode", not "code".

// ❌ WRONG — silently not rendered; bricks.Factory.get('code') → null
{"widgettype": "code", "id": "kb_selector", "options": {"name": "kb_id", ...}}

// ✅ CORRECT — renders as <select id="kb_selector">
{"widgettype": "UiCode", "id": "kb_selector", "options": {"name": "kb_id", ...}}

Symptom: get_by_id() return null <id> in console even though the JSON has "id". Browser snapshot shows the widget is missing entirely (no <select> element). The page looks fine otherwise — just the dropdown is absent.

🔴 getWidgetById scope: urlwidget sub-pages need this, not bricks.app.root

When a .ui page is loaded as a urlwidget sub-page (inside a target container), its widgets are NOT registered in bricks.app.root's global tree. getWidgetById('id', bricks.app.root) returns null.

Fix: Use this (the parent widget owning the binds array) as the search root:

// ❌ WRONG — returns null in urlwidget sub-pages
var target = bricks.getWidgetById('search_results', bricks.app.root);

// ✅ CORRECT — this = the VBox that owns the binds, parent of search_results
var target = bricks.getWidgetById('search_results', this);

this in a script bind (without target on the bind) = the widget that owns the binds array. Since search_results and search_bar are children of this same VBox, getWidgetById from this finds them. Also applies to kb_selector and any other sibling widgets referenced in the script.

Symptom: get_by_id() return null <widget_id> in browser console, even though the widget exists in the DOM.

Target-bound script — this is the target, use .parent for siblings:

When a bind has "target": "search_file", this in the script is search_file (the UiFile), NOT the parent VBox. To find sibling widgets in the same HBox row, use this.parent. To find widgets in the grandparent VBox, use this.parent.parent:

// ❌ WRONG — this = UiFile, not the VBox; bricks.app.root also wrong for urlwidget sub-pages
{"wid": "search_file", "event": "changed", "target": "search_file",
 "script": "var kb=bricks.getWidgetById('kb_selector',bricks.app.root);"}

// ✅ CORRECT — this.parent = HBox row, this.parent.parent = outer VBox
{"wid": "search_file", "event": "changed", "target": "search_file",
 "script": "var kb=bricks.getWidgetById('kb_selector',this.parent);var pv=this.parent.parent;fetch(...).then(function(d){var fp=bricks.getWidgetById('search_results',pv);})"}

this.parent chain: UiFile (target) → parent HBox → parent VBox (owns binds + search_results). Capture pv before the fetch callback because this may not survive the async boundary.

🔴 buildUrlwidgetHandler returns a function — MUST call it with () to fire HTTP request

bricks.buildUrlwidgetHandler(w, target, rtdata, desc) returns _buildWidget.bind(null, ...) — a bound function, not the result. The caller must invoke it. In actiontype: "script" binds, the script's return value is passed to universal_handler which does return await f(event) — if the script returns the bound function without calling it, the function is discarded and the HTTP request never fires.

// ❌ WRONG — returns bound function but never calls it; no HTTP request made
bricks.buildUrlwidgetHandler(this, target, {}, {options:{url:u}, mode:'replace'})

// ✅ CORRECT — () invokes the bound function, triggering widgetBuild → httpcall
bricks.buildUrlwidgetHandler(this, target, {}, {options:{url:u}, mode:'replace'})()

Symptom: console shows idset=search_results (widget found) but Network tab shows zero requests. No errors in console. The search/action appears to "do nothing."

Button events: use click, NOT tap

Bricks Button binds use "event": "click". Using "event": "tap" silently fails — no error, no action.

Form widget spec

// Parent has the bind, listening for Form's "submit" event
{
    "widgettype": "VBox",
    "subwidgets": [
        {"widgettype": "Form", "id": "my_form", "options": {"cols": 1, "fields": [...]}}
    ],
    "binds": [
        {"wid": "my_form", "event": "submit", "actiontype": "urlwidget",
         "target": "self", "options": {"url": "..."}}
    ]
}

Popup from button: target: "Popup" + dismiss config

{"event": "click", "actiontype": "urlwidget", "target": "Popup",
 "popup_options": {"auto_dismiss": true},
 "options": {"url": "..."}}

Use auto_dismiss: true (NOT auto_close: false). Without it, clicking anywhere closes the popup.

Tree widget editable CRUD

{
    "widgettype": "Tree",
    "options": {
        "parentField": "parentid", "idField": "id", "textField": "label",
        "dataurl": "...?kb_id=...",
        "newdata_params": {"kb_id": "..."},
        "editable": {
            "fields": [{"name": "name", "title": "名称", "type": "str", "length": 255, "uitype": "str"}],
            "add_url": "...", "update_url": "...", "delete_url": "..."
        }
    }
}

• newdata_params is appended to the form data on create (for passing kb_id etc.)
• CRUD dspy files must return {"widgettype": "Message", "options": {"user_data": {"id": ..., "label": ...}}}
• Data dspy returns [{id, parentid, label}] — parentid empty for root nodes
• Tree auto-detects root vs child: select a node → creates child; no selection → creates root

Post-submit refresh: dspy returns urlwidget redirect

After form submit (e.g., create KB), close popup and refresh the list:

return {"widgettype": "urlwidget", "options": {"url": entire_url('/list_page/index.ui')}}

UiFile widget — drag-and-drop + click file upload (REPLACES Droppable + manual script)

Bricks has a built-in UiFile widget that handles BOTH drag-and-drop AND click-to-select file upload. It extends VBox, has droparea CSS class, manages dragover/dragenter/dragleave/drop events with preventDefault(), and creates an invisible <input type="file"> for click selection.

{
    "widgettype": "UiFile",
    "id": "file_drop_zone",
    "options": {
        "accept": "",
        "multiple": true,
        "preview": false,
        "otext": "📂 拖拽文件到此处上传，或点击选择文件",
        "width": "100%",
        "cheight": 10,
        "css": "card",
        "border": "2px dashed #3b82f6"
    },
    "binds": [
        {"wid": "self", "event": "dragover", "actiontype": "script", "script": "event.preventDefault();event.stopPropagation()"},
        {"wid": "self", "event": "drop", "actiontype": "script", "script": "event.preventDefault();event.stopPropagation()"}
    ]
}

Key properties:

• accept: file type filter (e.g., "image/*", ".pdf"), empty = all files
• multiple: allow multiple file selection
• preview: show image/video/audio previews after selection
• otext: prompt text shown in the drop zone (supports i18n)

How it works:

• UiFile extends VBox, creates a hidden <input type="file">
• Drop events → dropHandle() extracts event.dataTransfer.files, stores in this.value
• Click events → the hidden input's change event → handleFileSelect()
• Both paths dispatch changed event with this.value (single File or File[] array)

Upload via changed event (parent-level bind):

{
    "wid": "file_drop_zone",
    "event": "changed",
    "actiontype": "script",
    "target": "file_drop_zone",
    "script": "var kb_id=this.opts.kb_id||'';var files=this.value;if(!files)return;if(!Array.isArray(files))files=[files];...x.open('POST','/api/doc/upload?kb_id='+encodeURIComponent(kb_id)+'&file_name='+encodeURIComponent(f.name)+'&folder='+encodeURIComponent(folder));x.send(f)..."
}

CRITICAL: Use this.value (NOT event.params) to access File objects. UiFile's getValue() returns an object like {undefined: resultValue()} — not the raw File array. this.value is the raw File/File[] set directly by dropHandle/handleFileSelect.

Script this context: When target is set on the bind, this in the script is the target widget (here file_drop_zone = UiFile). Without target, this falls back to the widget owning the binds array (the parent VBox). Always set target explicitly when you need the event source widget's properties.

Passing server values to script: Widget options (NOT script strings) DO process Jinja2 {{...}}. To pass server-rendered values to a script, add them as custom options on the target widget (e.g., "kb_id": "{{params_kw.kb_id}}" on UiFile) and read via this.opts.kb_id in the script. Do NOT try to embed {{...}} directly in script strings — they're sent literally to the browser.

⚠️ UiFile's built-in drag handlers call preventDefault() but NOT stopPropagation(). If you need stopPropagation (to prevent parent widgets from also receiving the event), add explicit binds on the UiFile widget itself for dragover and drop events — UiFile's own handlers and your binds both fire on the same DOM element via addEventListener, so both run.

⚠️ Previous note about Droppable was wrong — bricks does NOT have a Droppable widget. The correct widget for OS file drag-and-drop is UiFile.

Debugging binds that don't fire

Step-by-step methodology validated in production:

1. Verify event fires: Change actiontype to "script" with console.log to confirm the event is dispatched and caught:

{"wid": "dir_tree", "event": "node_selected", "actiontype": "script",
 "script": "console.log('FIRED:', event.params)"}

2. Check Network tab: Once event fires, look for the HTTP request to your DSPY.
3. Verify DSPY endpoint directly: curl from the server to confirm the DSPY returns valid widget JSON with the expected params.
4. Switch to urlwidget: Only after confirming steps 1-3, change back to actiontype: "urlwidget".

Tree node_selected event with event_params (no datawidget needed)

Tree dispatches node_selected (NOT "selected"). The event carries event.params = {id, parentid, label, selected: true} — usable directly as URL query params without datawidget/datascript overhead:

// bind on parent VBox — listens for Tree selection, loads DSPY into target panel
{
    "wid": "dir_tree",
    "event": "node_selected",
    "actiontype": "urlwidget",
    "target": "file_list_panel",
    "mode": "replace",
    "options": {
        "url": "{{entire_url('./file_list.dspy')}}?kb_id={{params_kw.kb_id}}"
    }
}

The {id, parentid, label} from the selected node are auto-merged into the URL query string. The DSPY reads them via ns.get('id'), ns.get('label'). The selected: true param is harmless (ignored by DSPY).

Verification pattern: start with actiontype: "script" + console.log('FIRED:', event.params) to confirm the event fires before switching to urlwidget.

Tree virtual root node

Add a non-expandable "根目录" node at the top of the tree for easy navigation back to root-level files:

# In get_tree_data.dspy — prepend to root query results
if not id:
    recs = await sor.sqlExe(...)
    result = [{"id": "__root__", "parentid": "", "label": "📁 根目录", "is_leaf": True}]
    result += [dict(r) for r in recs]
    return result

is_leaf: True prevents the root node from being expandable. When clicked, it fires node_selected with id: "__root__". Handle this in file_list.dspy: if folder_id == '__root__': folder_id = '' (treated same as no folder — shows root-level files).

Folder-based document filtering

Documents are associated with tree folders via a folder_id column on the documents table. The tree's node_selected event passes the folder id, and the DSPY filters by it.

Schema:

ALTER TABLE documents ADD COLUMN folder_id VARCHAR(32) DEFAULT '' AFTER kb_id;

DSPY query pattern — root vs folder:

if folder_id:
    recs = await sor.sqlExe(
        "SELECT ... FROM documents WHERE kb_id=${kb_id}$ AND folder_id=${folder_id}$ ORDER BY created_at DESC",
        {"kb_id": kb_id, "folder_id": folder_id})
else:
    recs = await sor.sqlExe(
        "SELECT ... FROM documents WHERE kb_id=${kb_id}$ AND (folder_id IS NULL OR folder_id='') ORDER BY created_at DESC",
        {"kb_id": kb_id})

• Root (no folder selected): shows files with empty/null folder_id
• Folder selected: shows files with matching folder_id
• Upload handler should also accept and store folder param to populate this column

VScrollPanel target for urlwidget replace mode

Prefer plain VBox over VScrollPanel as a target for urlwidget with mode: "replace". VScrollPanel's extra CSS/overflow handling can cause visual update issues after clear_widgets() + add_widget(w). VBox is simpler and works reliably for content swapping.

Initial load pattern: child urlwidget

To load content on page init AND support later refresh via binds, use a urlwidget as a child of the target container:

{
    "widgettype": "VBox",
    "id": "file_list_panel",
    "options": {"css": "filler"},
    "subwidgets": [{
        "widgettype": "urlwidget",
        "options": {
            "url": "{{entire_url('./file_list.dspy')}}?kb_id={{params_kw.kb_id}}",
            "css": "filler"
        }
    }]
}

The urlwidget auto-loads on widget build. Binds target file_list_panel with mode: "replace" to swap content.

actiontype: "script" in .ui files does NOT process {{...}} Jinja2 expressions. Variables like {{params_kw.kb_id}} are sent literally to the browser. Instead, read values from JavaScript context:

• URL params: new URL(window.location.href).searchParams.get('kb_id')
• Global variables set by other widgets: window._rag_current_folder
• DOM elements: document.getElementById('element_id').value

File upload via script action

For file upload in bricks without page navigation:

{
    "widgettype": "Button",
    "options": {"label": "选择文件", "bgcolor": "#3b82f6", "color": "#fff"},
    "binds": [{
        "wid": "self",
        "event": "click",
        "actiontype": "script",
        "script": "var i=document.createElement('input');i.type='file';i.onchange=function(){var f=i.files[0];var x=new XMLHttpRequest();x.open('POST','/api/doc/upload?kb_id='+K+'&file_name='+encodeURIComponent(f.name));x.send(f)};i.click()"
    }]
}

Keep scripts SHORT — avoid nested quotes. For status feedback, add a Text widget with id and update via document.getElementById().

Theme shell layout (top/center/bottom)

Sage-style layout structure:

VBox (app, filler, 100%)
├── urlwidget → top.ui (cheight: 2.5)
├── HBox (filler, width: 100%)
│   ├── VBox (sidebar, cwidth: 13-16, bgcolor: #2c2c2c)
│   │   └── Tree or Menu widget
│   └── VBox (id: rag_main_content, filler)
│       └── Tabular / DataViewer / content
└── HBox (rag-bottom) — copyright + ICP

The content area must have id at widget level (not inside options) for app.<id> references to work.

urlwidget with datawidget — reading another widget's state

When a button click needs context from another widget (e.g., which tree node is selected), use datawidget on the bind. The bricks framework reads the source widget's current data and appends it to the URL query params automatically.

The bind goes on a PARENT widget, not on the button itself. The parent's binds array listens for the child button's event:

{
    "widgettype": "VBox",
    "subwidgets": [
        {"widgettype": "Tree", "id": "dir_tree", "options": {...}},
        {"widgettype": "Button", "id": "select_file_btn", "options": {"label": "选择文件"}},
        {"widgettype": "VScrollPanel", "id": "file_list_panel", "options": {"css": "filler"}}
    ],
    "binds": [{
        "wid": "select_file_btn",
        "event": "click",
        "actiontype": "urlwidget",
        "datawidget": "dir_tree",
        "target": "file_list_panel",
        "mode": "replace",
        "options": {
            "url": "{{entire_url('./file_list.dspy')}}?kb_id={{params_kw.kb_id}}"
        }
    }]
}

How it works:

• wid: the button whose click triggers the action
• datawidget: the Tree whose selected node data is read and merged into the URL params
• target: the VScrollPanel where the loaded DSPY content is inserted
• mode: "replace": replace the target's content with the loaded widget

The Tree's selected node fields (id, parentid, label) are auto-appended to the DSPY URL query string. The DSPY reads them from params_kw.

Pitfall: Tree.getValue() returns whole tree, not just selected node

When datawidget points to a Tree widget without datascript or datamethod, bricks calls the default getValue() method. Tree's getValue() returns the ENTIRE tree structure including all children arrays recursively — these serialize as children=[object Object] in the URL, and the SELECTED node's id/parentid/label is NOT passed.

Fix: Add datascript to extract just the selected node's data:

{
    "datawidget": "dir_tree",
    "datascript": "if(this.selected_node) return this.selected_node.user_data; return {}"
}

This returns {id: 'xxx', parentid: 'yyy', label: '目录名'} from the selected node (if any), which becomes clean URL query params. With no selection, returns {} so only the static options.params are sent.

Pitfall: The button must NOT have its own binds array — the bind lives on the parent. If the button has its own binds, the parent's bind won't fire for it.

urlwidget relative URL pitfall — use absolute paths in scripts

When a .ui page is loaded as a sub-widget via urlwidget within the shell, the browser's base URL is the shell page, NOT the sub-widget's URL. Relative URLs like ./upload_file.dspy in XHR/script resolve to the ROOT, not to the sub-widget directory. Always use absolute paths in script XHR calls: /rag/knowledge_bases_list/upload_file.dspy.

Upload completion refresh — put refreshList AFTER if/else

When uploading multiple files, the refresh call must be placed OUTSIDE the if/else block so it fires regardless of success/failure, after all uploads complete:

x.onload=function(){
    done++;
    if(x.status===200){ st.innerText='上传成功 '+done+'/'+total }
    else{ st.innerText='上传失败' }
    if(done>=total)refreshList()  // outside if/else — always runs after last upload
};

Voiceprint ahserver startup — auth bypass needed for dependency compatibility

When starting voiceprint with ahserver (python3 ah.py -p 9087), the imported ahserver may use a different aiohttp_auth version than what's installed. The installed aiohttp_auth.auth module may lack the setup() method, and auth.get_auth() may throw RuntimeError: auth_middleware not installed.

Fix — two changes needed:

1. In ahserver/ahserver/auth_api.py, add return at the start of setupAuth() (skip auth middleware setup)
2. In ahserver/ahserver/processorResource.py, change self.user = await auth.get_auth(request) to self.user = None # auth disabled

Dependencies needed (install once):

pip install aiohttp_auth aiohttp_session aiohttp_cors aiohttp_middlewares openpyxl rsa qrcode asyncssh

Start command:

cd /share/ymq/run/voiceprint
PYTHONPATH=.:sqlor:appPublic:ahserver:longtasks/longtasks \
  nohup python3 ah.py -p 9087 > logs/voiceprint.log 2>&1 &

Verify: curl http://localhost:9087/api/status → should return JSON with "ready": true

Face API parameter format — images (array), NOT image

The face-service accepts {"images": ["base64_str", ...]} — an array of base64 strings. Sending {"image": "..."} (singular, non-array) returns 500.

### UAPI GPU services — port 10443 (NOT 443)

GPU services listen on port **10443**, not the default HTTPS port 443. The `upapp` table baseurls must include the port. Without this, `_call_uapi` and inline aiohttp calls hang with connection timeout.

```sql
UPDATE upapp SET baseurl='https://embedding.opencomputing.net:10443' WHERE id='rag-embedding';
UPDATE upapp SET baseurl='https://vectordb.opencomputing.net:10443' WHERE id='rag-vdb';
UPDATE upapp SET baseurl='https://reranker.opencomputing.net:10443' WHERE id='rag-reranker';
UPDATE upapp SET baseurl='https://graphdb.opencomputing.net:10443' WHERE id='rag-graph';
UPDATE upapp SET baseurl='https://entities.opencomputing.net:10443' WHERE id='rag-ner';
UPDATE upapp SET baseurl='https://face.opencomputing.net:10443' WHERE id='rag-face';

After updating upapp: redis-cli FLUSHDB + restart server. RAG ingest → embedding/VDB → 200.

User mandate: "尽量用ui/dspy方式实现功能，少用registerfunction" — prefer .dspy files over init.py registerfunction handlers for new features. DSPY files benefit from hot_reload (no restart needed) and keep module code minimal.

Key finding: In DSPY context, params_kw correctly receives ALL query string parameters for POST requests — unlike registerfunction handlers where getPostData() can consume the body and params may be unreliable.

Shell escaping pitfall — use file-based Python, not inline -c

When running Python on remote servers, python3 -c "..." with triple-quoted strings or chr(34) workarounds causes endless escaping issues. Always write the Python script to a file and run it instead:

# ❌ DON'T — shell escaping nightmare
ssh host "python3 -c 'import json\nd=json.load(...)\n...'"

# ✅ DO — write file, then execute
cat <<'PYEOF' > /tmp/script.py
import json
d = json.load(open('/path'))
...
PYEOF
scp /tmp/script.py host:/tmp/
ssh host "python3 /tmp/script.py"

DSPY upload file pattern (upload_file.dspy):

ns = params_kw.copy()
kb_id = ns.get('kb_id', '')
folder_id = ns.get('folder', '')  # ✅ DSPY context: query params always available
file_name = ns.get('file_name', 'upload.bin')

if not kb_id:
    return json.dumps({"status": "error", "error": "kb_id required"}, ensure_ascii=False)

file_data = await request.read()
if not file_data:
    return json.dumps({"status": "error", "error": "no file data"}, ensure_ascii=False)

env = request._run_ns
userorgid = await env.get_userorgid()
doc_id = str(uuid()).replace('-', '')[:16]
ext = '.' + file_name.rsplit('.', 1)[1] if '.' in file_name else '.bin'
saved_name = doc_id + ext
file_path = '/d/rag/ragserver/pkgs/rag/rag/files/' + saved_name
with open(file_path, "wb") as f:
    f.write(file_data)

file_size = len(file_data)

async with get_sor_context(env, 'rag') as sor:
    await sor.sqlExe(
        "INSERT INTO documents (id, kb_id, folder_id, file_name, file_type, file_size, file_path, mime_type, status, org_id, created_at, updated_at) "
        "VALUES (${id}$, ${kb_id}$, ${folder_id}$, ${file_name}$, 'other', ${file_size}$, ${file_path}$, 'application/octet-stream', 'pending', ${org_id}$, NOW(), NOW())",
        {"id": doc_id, "kb_id": kb_id, "folder_id": folder_id, "file_name": file_name,
         "file_size": file_size, "file_path": "/idfile/files/" + saved_name,
         "org_id": userorgid})

    await sor.sqlExe(
        "UPDATE knowledge_bases SET doc_count=doc_count+1, total_size=total_size+${size}$ WHERE id=${kb_id}$",
        {"size": file_size, "kb_id": kb_id})

return json.dumps({
    "status": "SUCCEEDED",
    "doc_id": doc_id,
    "file_name": file_name,
    "file_size": file_size,
    "folder_id": folder_id
}, ensure_ascii=False, default=str)

No imports needed: json, uuid, get_sor_context, DBPools are pre-loaded. Use str(uuid()).replace('-', '')[:16] for ID generation (not uuid.uuid4().hex — uuid may be the function, not the module).

Files directory: Use hardcoded absolute path (/d/rag/ragserver/pkgs/rag/rag/files/) — os and __file__ are NOT available in DSPY context.

Contrast with registerfunction: The same folder parameter passed via ?folder=xxx was consistently empty in params_kw when using the registerfunction handler (doc_upload_handler). Switching to DSPY resolved this — no getPostData body-consumption issue.

Upload completion refresh: put refreshList AFTER if/else, not inside else

// ❌ WRONG — refreshList only called on failure
x.onload=function(){
    done++;
    if(x.status===200){
        st.innerText='上传成功 '+done+'/'+total
    }else{
        st.innerText='上传失败';
        if(done>=total)refreshList()  // <-- inside else block!
    }
};

// ✅ CORRECT — refreshList called after all uploads regardless of status
x.onload=function(){
    done++;
    if(x.status===200){
        st.innerText='上传成功 '+done+'/'+total
    }else{
        st.innerText='上传失败'
    }
    if(done>=total)refreshList()  // <-- outside if/else
};

Linked References

• references/rag-pitfalls-learned.md — Recurring DSPY pitfalls: sqlor %% escaping in LIKE, document_chunks only has created_at, bricks UiCode multiple fix, tag-only search must query documents not just document_chunks, face-generated doc filtering, video voiceprint extraction, voiceprint API 405, permission+rolepermission dual requirement, re-processing pending videos

• references/bricks-patterns-learned.md — urlwidget relative URL pitfall, tree virtual root node, file action buttons, upload refresh pattern

• references/dspy-upload-pipeline.md — DSPY upload pipeline pattern: file classification, text extraction, RAG ingest, face detection via GPU service; pip install -e . deployment step; ServerEnv vs explicit import; available libraries

• references/voiceprint-server-setup.md — Voiceprint server startup, API endpoints, nginx routing, engine import fix

• references/gpu-services.md — GPU server service map (VDB, CLIP, reranker, face, graph, NER) with ports, endpoints, and uapi integration pattern

• references/gpu-services-routing.md — Exact GPU API endpoints: face /face/api/detect, voiceprint /voiceprint/extract/submit, nginx routing via media.opencomputing.net:10443, API parameter formats

• references/search-form-patterns.md — Bricks search forms: SearchBar keyword key, code dropdown [{value,text}], datawidget crash with code, UiFile file search, fetch must include _webbricks_=1

• references/file-download-url.md — File download URL: always /idfile/{doc_id}.{ext}, NOT /idfile/files/... or entire_url(file_path)

• references/vdb-api-reference.md — VDB API specs: exact params (colname not collection), response format (data.rows[]), embedding/reranker/face/voiceprint formats

• references/deployment-errors.md — Specific error patterns and fixes from deployment sessions

• references/bootstrap-sync-errors.md — Bootstrap/re-sync error patterns: untracked-file pull blocks, config.json corruption, RBAC 403 after sync

• references/tag-management.md — Full tag management reference (DB schema, API, UI)

• references/file-pipeline-types.md — File type classification table: which types go through RAG ingest vs face/voiceprint only; supported extensions per category

• references/detail-ui-template.md — Complete detail.ui template: tree + UiFile drop zone + file_list_panel + 3 binds pattern

• references/rag-ingest-pipeline.md — Full working DSPY pattern: text extraction → chunking → CLIP embedding → VDB → document_chunks. Zero external imports, all inline. File type support table and API endpoint reference.

• references/streamhttpclient-api.md — StreamHttpClient API reference: pre-loaded DSPY global, request/json/files patterns, face/voice/embedding/VDB examples

• references/search-dspy-pattern.md — search_result.dspy pattern: StreamHttpClient + try/except for embedding/VDB search, frontend integration, pitfall of bare aiohttp crashing on GPU flap

DSPY Background Processing (background_reco)

background_reco is pre-loaded in DSPY globals (g.background_reco = background_reco in globalEnv.py:345). It's an asyncio.create_task wrapper — fire-and-forget. Use it to run ingestion/embedding/VDB work after returning the upload response.

# background_reco(func, *args) — calls func(*args) in a background task
background_reco(func, arg1, arg2, arg3)

Critical rules:

• The background function must be self-contained: create its own DBPools() + sqlorContext() — do NOT pass env, request, or any proxy objects. They may be invalid after the response completes.
• Pass only serializable/primitive values: strings, ints, bytes, dicts of primitives. No lambdas, no proxy objects.
• The background function is async. Define it inside the DSPY file (not in an external module — ServerEnv functions don't work in DSPY context).
• Do NOT use add_startup or ensure_future — use background_reco only.

Pattern — upload → immediate response → background ingest:

file_data = await request.read()
env = request._run_ns
userorgid = await env.get_userorgid()
web_path = await env.save_file(file_data, file_name)
file_size = len(file_data)

# Sync: insert document as 'pending', update KB counts, return immediately
async with get_sor_context(env, 'rag') as sor:
    await sor.sqlExe("INSERT INTO documents (...) VALUES (..., 'pending', ...)", {...})
    await sor.sqlExe("UPDATE knowledge_bases SET doc_count=doc_count+1, total_size=total_size+${size}$ WHERE id=${kb_id}$", {"size": file_size, "kb_id": kb_id})

# Fire background ingest — pass kb_id, doc_id, file bytes, etc. as primitives
background_reco(ingest_doc, kb_id, doc_id, file_data, file_name, ext_l, userorgid, web_path)

return json.dumps({"status": "SUCCEEDED", "doc_id": doc_id, ...})

# Background function — completely self-contained
async def ingest_doc(kb_id, doc_id, file_data, file_name, ext_l, userorgid, web_path):
    # Own DB connection
    db = DBPools()
    async with db.sqlorContext('rag') as sor:
        # text extraction, chunking, embedding, VDB, document_chunks...
        await sor.sqlExe("UPDATE documents SET status='done', chunk_count=${n}$ WHERE id=${id}$", {"n": n, "id": doc_id})
        await sor.sqlExe("UPDATE knowledge_bases SET chunk_count=chunk_count+${n}$ WHERE id=${kb_id}$", {"n": n, "kb_id": kb_id})

Verification: after upload, query documents — status should be 'pending' immediately, then 'done' within seconds. Check document_chunks for rows.

Pitfalls

.ui files start with {% set ... %} and contain {{entire_url(...)}} expressions. They are NOT valid JSON until the Sage bui processor renders them. Don't try to json.loads() them directly.

Test server DB tables

Models define tables but the DDL must be run manually on the test server. Check table existence:

ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && py3/bin/python3 -c \"
from sqlor.dbpools import DBPools; from appPublic.jsonConfig import getConfig
...\""

hot_reload means no restart for UI changes

conf/config.json has "hot_reload": true — .dspy and .ui changes take effect without restart. Only init.py changes need a restart (handlers re-registered).

Tabular editable requires Message widget return

add.dspy, update.dspy, delete.dspy must return {"widgettype": "Message", "options": {"user_data": {...}}} format, not raw JSON data.

🔴 DSPY uuid is SHADOWED by getID function — MUST import uuid explicitly

In the DSPY exec context, uuid resolves to getID (a function from globalEnv.py:306: g.uuid = getID), NOT Python's uuid module. Calling uuid.uuid4() without import uuid raises AttributeError: 'function' object has no attribute 'uuid4'. If wrapped in try/except: pass, the error is silently swallowed — popup closes but no data is written.

Fix: Every DSPY file that calls uuid.uuid4() MUST have import uuid at the top:

# ✅ CORRECT — explicit import overrides the shadowed global
import uuid
mt_id = uuid.uuid4().hex[:16]

# ❌ WRONG — uuid resolves to getID function, not the module
mt_id = uuid.uuid4().hex[:16]  # AttributeError

Files that use explicit import: create_kb.dspy, new_tree_item.dspy. Fixed 2026-08-07: save_tags.dspy (was missing import → silent save failure). Always audit new DSPY files — this shadow is universal across all DSPY exec contexts.

DSPY last expression is the return — MUST be inside async with block

A .dspy file is wrapped by ahserver. The return value comes from the function's return statement or the last expression. The CRITICAL rule: the returning expression MUST be INSIDE the async with db.sqlorContext(...) block. Variables assigned inside the async block but referenced outside will be NoneType.

# ✅ CORRECT — result expression inside async block
async with db.sqlorContext(dbname) as sor:
    recs = await sor.R('tags', {'kb_id': kb_id})
    ...
    result = {"widgettype": "VBox", ...}
    result   # <-- last expression inside async block ✅

# ❌ WRONG — result assigned inside, referenced outside → NoneType 500
async with db.sqlorContext(dbname) as sor:
    result = {"widgettype": "VBox", ...}
result   # <-- outside async block → None ❌

# ✅ ALSO CORRECT — explicit return inside async block
async with db.sqlorContext(dbname) as sor:
    ...
    return result

🔴 tag_form submit: NEVER location.reload() — close popup + AJAX refresh media panel

After saving tags, calling location.reload() destroys the entire page state. The correct pattern:

1. fetch save_tags.dspy to persist tags server-side
2. Close the popup (pw.dismiss(); pw.destroy())
3. AJAX-reload media_cards.dspy and rebuild #media_card_panel in-place via bricks.widgetBuild():

// ✅ CORRECT — close popup, then refresh the media panel in-place
fetch(save_url).then(function(r){return r.json()}).then(function(d){
    var pw=null; var w=self;
    while(w){if(w instanceof bricks.PopupWindow||w instanceof bricks.Popup){pw=w;break};w=w.parent};
    if(pw){pw.dismiss();pw.destroy()};
    var mp=document.querySelector('#media_card_panel');
    if(mp&&mp.bricks_widget){
        fetch(media_cards_url).then(function(r2){return r2.json()}).then(function(d2){
            var mw=mp.bricks_widget; mw.clear_widgets();
            bricks.widgetBuild(d2,mw).then(function(nw){if(nw)mw.add_widget(nw)});
        });
    }
});

// ❌ WRONG — full page refresh, user loses context
location.reload();

Fixed 2026-08-07: tag_form.dspy submit_js was using location.reload().

SQL parameter syntax

Use ${param}$ in SQL strings: "SELECT * FROM t WHERE id=${id}$". Do NOT use %(param)s or f-strings for SQL parameters.

CRUD conflict: json/.json vs hand-written wwwroot/_list/

When json/<table>.json (CRUD DataViewer config) exists, the Sage framework auto-generates CRUD endpoints for /<table>_list/. Creating hand-written wwwroot/<table>_list/ with custom DSPY files (data.dspy, add.dspy, etc.) conflicts with the auto-generated routes and causes 403 Forbidden.

Fix: If json/<table>.json exists, delete the hand-written wwwroot/<table>_list/ directory. Let the framework auto-CRUD handle the list page. The json/<table>.json editfields/browserfields control the Tabular widget.

🔴 upload_file.dspy deployed WITHOUT PDF/DOCX/PPTX/XLSX extraction → silent zero-chunk

The deployed upload_file.dspy can be MISSING the elif branches for PDF/DOCX/PPTX/XLSX text extraction — only has the plain-text if ext_l in text_exts branch. When non-text files (PDF, DOCX, PPTX, MP4, etc.) are uploaded, text stays '', the if text and len(text.strip()) > 10: guard is never true, and the entire RAG ingest block (chunking → embedding → VDB) is skipped. Documents get inserted with status='done' and chunk_count=0 — looks successful, nothing ingested.

The correct full pattern (with elif ext_l == '.pdf': ... elif ext_l == '.docx': ... etc.) IS in references/rag-ingest-pipeline.md. Before deploying, diff against that reference.

Symptom: document_chunks is empty for all non-plain-text files.

Verification query (run after any upload deploy):

mysql -h db -u test -ptest123 rag -e "
SELECT d.file_name, d.status, COUNT(c.id) AS chunks
FROM documents d LEFT JOIN document_chunks c ON c.doc_id = d.id
WHERE d.kb_id = '<kb_id>'
GROUP BY d.id, d.file_name, d.status
HAVING chunks = 0"

Any row in the output = a file that uploaded but was never ingested.

🔴 base64 NOT in DSPY pre-loaded globals — crashes image/video upload

base64 module is NOT among ahserver's DSPY pre-loaded globals (only hex2base64 function is). Using base64.b64encode() without import base64 raises NameError: name 'base64' is not defined, crashing the entire DSPY before the DB insert.

Confirmed in server logs:

NameError: name 'base64' is not defined
  at upload_file.dspy line 41: img_b64 = base64.b64encode(file_data).decode()

Fix: Add import base64 at the top of any DSPY file that uses base64 (e.g., for face detection on image/video frames to build data URIs for GPU API calls).

🔴 DSPY HTTP calls: use StreamHttpClient, NOT aiohttp

StreamHttpClient IS a pre-loaded DSPY global (from globalEnv.py: g.StreamHttpClient = StreamHttpClient). aiohttp is NOT available in DSPY context. Using aiohttp.ClientSession(...) without import raises NameError. The critical danger: if wrapped in bare except: pass, the NameError is silently swallowed — the file upload succeeds (DB insert at end runs), but EVERY processing step fails silently: no embedding, no face detection, no voiceprint, no VDB upsert. Result: status='done', chunk_count=0.

Confirmed in server logs:

NameError: name 'aiohttp' is not defined
  at upload_file.dspy line 160: async with aiohttp.ClientSession(...)

StreamHttpClient API (pre-loaded, no import needed):

# Simple JSON POST — returns bytes, parse with json.loads()
client = StreamHttpClient()
resp = await client.request('POST', url, json={"key": "value"})
result = json.loads(resp)

# File upload (multipart)
resp = await client.request('POST', url, files={'field': (filename, file_bytes)})

# No timeout parameter — StreamHttpClient handles retry internally
# No context manager — use as a plain object, each call is independent

Full example — face detection in DSPY:

try:
    client = StreamHttpClient()
    resp = await client.request('POST', 'https://media.opencomputing.net/face/api/detect',
        json={"images": [img_b64]})
    fd = json.loads(resp)
    results = fd.get("results", [])
    if results and isinstance(results[0], dict):
        face_count = len(results[0].get("faces", results[0].get("detections", [])))
except: pass

Voiceprint with file upload:

try:
    client = StreamHttpClient()
    resp = await client.request('POST', 'https://media.opencomputing.net/voiceprint/extract/submit',
        files={'file': (file_name, file_data)})
    vd = json.loads(resp)
    voice_speakers = vd.get('speakers', 1) if vd.get('status') == 'SUCCEEDED' else 0
except: pass

Verification: After any upload DSPY change, upload a test .txt file and query document_chunks to confirm it gained a row. Never rely on status='done' alone — check chunk_count > 0.

Anti-pattern — bare except: pass buries NameErrors:

# ❌ DANGEROUS — NameError silently swallowed
try:
    client = aiohttp.ClientSession(...)  # NameError
except: pass  # all processing silently skipped

# ✅ CORRECT — StreamHttpClient is pre-loaded, no NameError possible
try:
    client = StreamHttpClient()
    resp = await client.request('POST', url, json=data)
except: pass

Bootstrap / Re-sync Local Repos

When the local environment has NO ragserver or rag repos (first setup, or after cleanup), bootstrap them from remote and sync any test-server-only files:

# 1. Create work dir and clone both repos from remote
mkdir -p /d/ymq/rag
cd /d/ymq/rag
git clone git@git.opencomputing.cn:yumoqing/ragserver.git ragserver
git clone git@git.opencomputing.cn:yumoqing/rag.git rag

# 2. Audit test server for uncommitted/untracked files
ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && git status --short"
ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver/pkgs/rag && git status --short"

# 3. scp any server-only files to local
scp rag@rag.opencomputing.cn:/d/rag/ragserver/scripts/init_rbac_v2.py /d/ymq/rag/ragserver/scripts/
scp rag@rag.opencomputing.cn:/d/rag/ragserver/pkgs/rag/wwwroot/.../new_file.dspy /d/ymq/rag/rag/wwwroot/.../

# 4. Stage, commit, push from local
cd /d/ymq/rag/ragserver  # or rag
git add ... ; git commit -m "sync: ..." ; git push origin main

# 5. Server pull (with pre-flight cleanup — see below)

Clean up old duplicate repos after bootstrap: remove any stale local clones under /d/ymq/rag*, /d/ymq/rag-review, /d/ymq/repos/rag so there is exactly ONE canonical local copy at /d/ymq/rag/.

Deployment Pitfalls

🔴 Untracked files on server block git pull

When a file exists on the server as untracked and the remote now tracks it (from a recent commit), git pull fails:

error: The following untracked working tree files would be overwritten by merge:
    scripts/init_rbac_v2.py
Please move or remove them before you merge.
Aborting

Fix: delete the untracked file on the server before pulling:

ssh rag@rag.opencomputing.cn "cd /d/rag/ragserver && rm -f scripts/init_rbac_v2.py && git pull"

This happens during the bootstrap sync workflow: the server had the file first (untracked), then you commit it locally and push. The server's copy must be removed so git can place the tracked version.

Config overwritten by git pull

conf/config.json has local changes on the test server (encrypted password, driver, pool params removed). After git pull, these are reset. Also, git pull silently fails when local config.json has uncommitted changes — the server runs old code.

Pattern: always use git checkout conf/config.json && git pull before deploy, then re-apply config fixes:

ssh rag@rag.pd4e.com "cd ~/ragserver && git checkout conf/config.json && git pull && source py3/bin/activate && python3 -c \"
import json;c=json.load(open('conf/config.json'))
c['databases']['rag']['driver']='mysql'
c['databases']['rag']['kwargs']['password']='cybEz86hASifn+iwFHirOQ=='
c['databases']['rag']['kwargs'].pop('minsize',None)
c['databases']['rag']['kwargs'].pop('maxsize',None)
json.dump(c,open('conf/config.json','w'),indent=4,ensure_ascii=False)
\" && ps aux|grep 'app/ragserver'|grep -v grep|awk '{print \$2}'|xargs kill -9 2>/dev/null;sleep 1&&export PYTHONPATH=\$PWD:\$PWD/pkgs/rag-pipeline&&source py3/bin/activate&&nohup py3/bin/python app/ragserver.py>logs/startup.log 2>&1&sleep 3&&curl -so /dev/null -w '%{http_code}' http://localhost:9181/&&echo' ok'"

RBAC cache clearing after manual SQL permission insert

When permissions are added directly via SQL (not init_rbac.py), the in-memory cache may NOT refresh even after process restart. The load_roleperms method has a guard: if the DB returns 0 records (can happen during restart race), it keeps the previous cache with a debug message 'got 0 records, keeping previous cache'. Full cache clearing:

cd /d/rag/ragserver
bash stop.sh
pkill -9 -f ragserver.py  # ensure no orphans
sleep 2
find pkgs -name __pycache__ -exec rm -rf {} + 2>/dev/null
redis-cli FLUSHDB        # clear session + RBAC caches
bash start.sh

The permission cache TTL is 10 minutes. Flushing Redis ensures both the RBAC rp_caches and user session state start fresh.

Table column naming: users.orgid vs knowledge_bases.org_id

users table uses orgid (no underscore). knowledge_bases uses org_id (with underscore). get_userorgid() returns the user's orgid. Admin user's orgid is NOT "0" — on ragserver it's 4772b9b7031b4676. When filtering by org in DSPY queries, use the correct column name for each table.

ln -sf ../pkgs/rbac/wwwroot wwwroot/rbac

Without this, /rbac/user/login.ui returns 500 (file not found).

Session persistence: add session_max_time + session_issue_time

Without these in conf/config.json, Redis sessions are never written and login is lost on restart:

"session_max_time": 3000,
"session_issue_time": 2500,

🔴 CRITICAL: site-packages copy stale after source edits — always reinstall

When editing ANY Python module in pkgs/<module>/ (rag, llmage, sqlor, etc.), the SERVER imports from py3/lib/python3.10/site-packages/<module>/, NEVER from pkgs/. Source edits have ZERO effect until reinstalled. After ANY source change: cd /d/apitest/sage && ./py3/bin/pip install --upgrade /d/apitest/sage/pkgs/<module> then restart Sage. This applies to ALL modules — pip install -e . symlinks can break silently; --upgrade is more reliable.

# Check if site-packages has the new function
grep 'extract_voiceprint' /d/rag/ragserver/py3/lib/python3.10/site-packages/rag/pipeline.py
# If NOT found, copy the source
cp /d/rag/ragserver/pkgs/rag/rag/pipeline.py /d/rag/ragserver/py3/lib/python3.10/site-packages/rag/pipeline.py

Also try pip install -e . for new modules:

cd /d/rag/ragserver/pkgs/rag
/d/rag/ragserver/py3/bin/pip install -e .

Then restart. Without this, from rag.pipeline import ... → ModuleNotFoundError for brand-new .py files.

Symptom: ImportError: cannot import name 'X' from 'rag.pipeline' — despite the function clearly existing in the source file. The site-packages copy is stale.

🔴 ServerEnv registration — DOES NOT WORK for DSPY context (proven in production)

Full reference: references/dspy-context-imports.md — definitive analysis of what works and what doesn't in DSPY exec context.

Registering functions on ServerEnv (env.func = func in init_rag_module()) does NOT make them available in DSPY files. The DSPY exec context inherits pre-loaded globals (json, uuid, DBPools, get_sor_context, params_kw, request) from globalEnv.py, but ServerEnv attributes set by init_rag_module() do NOT propagate to request._run_ns in DSPY.

Verified failure pattern (init_rag_module sets the function, server starts without error):

def init_rag_module():
    env = ServerEnv()
    from .pipeline import process_upload
    env.process_upload = process_upload  # set on ServerEnv singleton
    rf = RegisterFunction()
    ...

DSPY calls it — ALWAYS returns NoneType:

env = request._run_ns
result = await env.process_upload(env, ...)  # TypeError: 'NoneType' object is not callable

The ONLY reliable zero-import pattern: inline ALL logic directly in the DSPY file. No external function calls, no ServerEnv dependencies, no from rag.pipeline import. Use only ahserver pre-loaded globals (json, uuid, DBPools, get_sor_context, request.read(), params_kw).

For text extraction from documents (PDF/DOCX/PPTX/XLSX): PyPDF2, python-docx, python-pptx, openpyxl ARE importable from DSPY — they're installed packages in the venv, not custom module code. Import them inline at point of use (not via separate pipeline.py).

Confirmed working pattern (deployed to ragserver, commit 1cb82b9):

# Place BEFORE the plain-text branch — office docs get priority
if ext_l == '.pdf' and not text:
    import io; from PyPDF2 import PdfReader
    reader = PdfReader(io.BytesIO(file_data))
    text = '\n'.join(p.extract_text() or '' for p in reader.pages)
elif ext_l == '.docx' and not text:
    import io; from docx import Document
    doc = Document(io.BytesIO(file_data))
    text = '\n'.join(p.text for p in doc.paragraphs)
elif ext_l == '.pptx' and not text:
    import io; from pptx import Presentation
    prs = Presentation(io.BytesIO(file_data))
    parts = []
    for slide in prs.slides:
        for shape in slide.shapes:
            if hasattr(shape, 'text') and shape.text:
                parts.append(shape.text)
    text = '\n'.join(parts)
elif ext_l == '.xlsx' and not text:
    import io; from openpyxl import load_workbook
    wb = load_workbook(io.BytesIO(file_data), data_only=True)
    parts = []
    for sheet in wb.worksheets:
        for row in sheet.iter_rows(values_only=True):
            parts.append('\t'.join(str(c or '') for c in row))
    text = '\n'.join(parts)

!!! NO filetxt package needed — the inline pattern above is self-contained and uses only the four venv-installed packages. The filetxt module (git.opencomputing.cn:yumoqing/filetxt.git) exists as a reference but has heavy deps (spacy, langchain_community, mobi, ebooklib) and is NOT used in DSPY.

base64 in DSPY: import base64 works in DSPY files now that ahserver/globalEnv.py exposes g.base64 = base64 (commit 93152f2). DSPY files still need import base64 at the top — it is NOT a pre-loaded global. Without the import, NameError: name 'base64' is not defined will still occur.

For RAG ingest: The _rag_ingest_async function from init.py is NOT available in DSPY. Inline the UAPI calls directly in the DSPY using StreamHttpClient (pre-loaded global — no import needed). See references/streamhttpclient-api.md for patterns.

Why ServerEnv fails: The DSPY exec wraps code in async def myfunc(request, **ns): with exec(txt, lenv, lenv). lenv is populated from globalEnv pre-loaded globals, NOT from ServerEnv's dynamic attributes. The run_ns.update(ServerEnv._ns_) merge in baseProcessor happens at request time but the DSPY func's closure captures lenv at exec time — custom ServerEnv attributes aren't in scope.

Never delete pycache blindly — the .pyc may be the only working version

When debugging a Python service, do NOT run rm -rf __pycache__ before checking if the .py source is complete. The .pyc file may contain compiled code from a PREVIOUS version that had server startup logic, while the current .py file may be a stripped-down version (e.g., just model functions without __main__). Deleting __pycache__ can permanently break a working service.

When a new .py file is added to the module package (e.g., rag/pipeline.py), the server can't find it until the package is reinstalled:

cd /d/rag/ragserver/pkgs/rag
/d/rag/ragserver/py3/bin/pip install -e .

Then restart the server. Without this step, from rag.pipeline import ... → ModuleNotFoundError.

Missing .tmpl processor causes 'NoneType' object has no attribute 'be_call' on bricks header template load.

🔴 DSPY HTTP calls: use StreamHttpClient, NOT aiohttp

StreamHttpClient IS a pre-loaded DSPY global (g.StreamHttpClient = StreamHttpClient). aiohttp is NOT available in DSPY context. Using aiohttp.ClientSession(...) without import raises NameError. Critical danger: if wrapped in bare except: pass, the NameError is silently swallowed — the file upload succeeds but EVERY processing step fails silently: no embedding, no face detection, no voiceprint, no VDB upsert. Result: status='done', chunk_count=0.

StreamHttpClient API (no import needed — pre-loaded global):

# JSON POST — returns bytes, parse with json.loads()
client = StreamHttpClient()
resp = await client.request('POST', url, json={"key": "value"})
result = json.loads(resp)

# File upload (multipart)
resp = await client.request('POST', url, files={'field': (filename, file_bytes)})

Always wrap in try/except — if the remote service is unreachable, the DSPY crashes 500 otherwise:

try:
    client = StreamHttpClient()
    resp = await client.request('POST', url, json=payload)
    result = json.loads(resp)
except:
    result = {}

Pitfall: StreamHttpClient.request() returns raw bytes — no .status, no .json(). Must json.loads(resp).

🔴 buildUrlwidgetHandler: 4-parameter signature, not 3

When calling bricks.buildUrlwidgetHandler() from a script action, the function takes 4 parameters:

// bricks.js line 465
bricks.buildUrlwidgetHandler = function(w, target, rtdata, desc){
    var options = objcopy(desc.options||{});  // ❌ TypeError if desc is undefined

• w — source widget (this in script without target)
• target — from getWidgetById
• rtdata — data object ({} if no datawidget)
• desc — {options: {url: ...}, mode: 'replace'} (NOTE: mode goes INSIDE desc)

❌ Wrong (3 args, desc=undefined → TypeError):

buildUrlwidgetHandler({options:{url:u}}, target, 'replace')

✅ Correct (4 args):

var t = bricks.getWidgetById('results', bricks.app.root);
if (t) bricks.buildUrlwidgetHandler(this, t, {}, {options:{url:u}, mode:'replace'});

Full reference: references/bricks-patterns-learned.md

🔴 search_result.dspy: MUST have try/except around HTTP calls

The shell's wwwroot/rag MUST be a symlink targeting ../pkgs/rag/wwwroot. This is the SINGLE MOST COMMON cause of mysterious 500 errors after deployments, git operations, or server restarts. Check this FIRST before debugging anything else.

If wwwroot/rag is a regular directory (not a symlink), ALL /rag/knowledge_bases_list/ paths fail with 500 "invalid path" — every DSPY, every .ui, every page. Other routes (/api/status, /) work fine, making it look like a routing issue when it's not.

Symptom: every page under /rag/ returns 500, but other routes (/api/status, /) work fine.

Diagnose:

ls -la wwwroot/rag
# Should show: rag -> ../pkgs/rag/wwwroot
# If it's a directory (drwxr-xr-x), the symlink is broken
ls wwwroot/rag/knowledge_bases_list/
# Should list all .ui/.dspy files; if empty or missing, symlink is broken

Fix:

rm -rf wwwroot/rag
ln -sf ../pkgs/rag/wwwroot wwwroot/rag

No server restart needed — file resolution happens on each request.

POST request params_kw: body consumed by getPostData before handler

For POST requests, ahserver's getArgs() calls getPostData(request) instead of reading request.query directly. getPostData tries three body-reading strategies in order:

1. request.multipart() — fails for raw binary (not multipart form data)
2. request.post() — returns empty dict for raw binary (not form-urlencoded)
3. request.read() — consumes the request body

The query string IS merged into params_kw via multiDict2Dict(request.query) before the body read, so GET-style query params like ?folder=xxx DO appear in params_kw even for POST requests.

Consequence for upload handlers: when the handler later calls await request.read(), the body may already be consumed (depending on aiohttp's internal buffering). If the file data is still received (some aiohttp versions buffer), the handler works but the body was read twice.

Debugging: when params_kw.get("some_key") returns empty but the Network tab shows the key in the URL, add the suspect key to the handler's response JSON to verify what was received:

return json.dumps({
    "status": "SUCCEEDED",
    "folder_id_received": folder_id,  # <-- debug field
    ...
})

Then check the Network response panel to see the actual value. If empty but the request URL had it, the issue is upstream in getPostData or DictObject conversion — not in your handler code.

Location: pkgs/ahserver/ahserver/processorResource.py, method getPostData + getArgs. See lines ~265-295.

DSPY file placement vs URL path — MUST match wwwroot layout

The wwwroot/rag symlink (-> ../pkgs/rag/wwwroot) means module pages at pkgs/rag/wwwroot/knowledge_bases_list/ are accessed via /rag/knowledge_bases_list/ URL, NOT /knowledge_bases_list/. When adding new dspy files:

• Place file: pkgs/rag/wwwroot/knowledge_bases_list/new.dspy
• URL path: /rag/knowledge_bases_list/new.dspy
• RBAC permission: /rag/knowledge_bases_list/new.dspy
• In .ui / .dspy files: {{entire_url('/rag/knowledge_bases_list/new.dspy')}}

Pitfall: Using /knowledge_bases_list/new.dspy (without /rag/ prefix) → 500 "invalid path" because no such file in wwwroot/knowledge_bases_list/.

RBAC permission path must match exact URL — verify with test

After adding permissions, always verify with a real curl test (with session cookie if logined):

# 1. Login to get cookie
curl -s -c /tmp/cookies.txt -X POST "http://localhost:9181/rbac/user/up_login.dspy" \
  -d "username=admin&password=admin123"
# 2. Test the dspy with the cookie
curl -s -b /tmp/cookies.txt "http://localhost:9181/rag/knowledge_bases_list/storage_card.dspy?_webbricks_=1"

401 = auth missing, 403 = permission wrong (path mismatch), 500 = dspy code bug, 200 = working.

Always curl-test new DSPY endpoints after deployment

User mandate: Every new dspy MUST be tested via curl before declaring done. Do NOT just deploy and assume it works. Test from the server itself (localhost) with a session cookie. Check the server log (logs/ragserver.log) for exceptions. 500 errors are invisible in the browser (bricks swallows them) but logged server-side.

New DSPY files MUST be added to init_rbac.py — or get 403

Every new .dspy file under a module's wwwroot/ directory needs an explicit path entry in scripts/init_rbac.py (PUBLIC or LOGINED list). The directory-level entry (/rag/knowledge_bases_list/) does NOT auto-cover files within — each file needs its own permission row.

Checklist when adding a new .dspy:

1. Add the path to init_rbac.py PUBLIC or LOGINED list
2. Run cd /d/rag/ragserver && py3/bin/python3 scripts/init_rbac.py
3. Verify the rolepermission association was created — the script uses INSERT IGNORE, and permission ID truncation (see below) causes the rolepermission insert to silently fail
4. Restart the server

Verification query:

mysql -h db -u test -p'test123' rag -e "
SELECT p.id, p.path, rp.roleid 
FROM permission p 
LEFT JOIN rolepermission rp ON rp.permid = p.id 
WHERE p.path LIKE '%your_new_file%'
"

If rp.roleid is NULL, the rolepermission wasn't created — insert it manually.

Permission ID truncation (varchar 32) causes silent rolepermission failures

init_rbac.py generates permission IDs as perm_<path_with_underscores>. For long paths, this exceeds the permission.id column's varchar(32) limit. MySQL silently truncates the ID on insert, but the rolepermission insert uses the full (un-truncated) rp_perm_<path> as its ID and perm_<path> as permid. Since the permid references the truncated permission ID, the INSERT IGNORE fails silently (FK constraint or similar).

Example: Path /rag/knowledge_bases_list/file_list.dspy generates

• pid = perm__rag_knowledge_bases_list_file_list.dspy (45 chars)
• Stored as perm__rag_knowledge_bases_list_f (truncated to 32)
• rp_perm__rag_knowledge_bases_list_file_list.dspy tries to link to truncated perm__rag_knowledge_bases_list_f — mismatch causes silent skip

Fix: After running init_rbac.py, always verify with the query above. If missing, insert manually:

INSERT IGNORE INTO rolepermission (id, roleid, permid) 
VALUES ('rp_<truncated_32_id>', 'any', '<truncated_32_id>');

Where <truncated_32_id> is the actual stored permission ID (first 32 chars).

User passwords: RC4, not AES

RBAC login uses rc4.password(s, key=k). DB connection uses aes_encode_b64. They are different systems — don't mix them up.

Hardcoded storage display — "已用 0MB" always shows zero

The KB list page knowledge_bases_list/index.ui hardcoded {"text": "已用 0MB / 总计 100MB"} — never queries DB. Fix: replace with urlwidget loading a DSPY.

storage_card.dspy:

ns = params_kw.copy()
env = request._run_ns

async with get_sor_context(env, 'rag') as sor:
    rec = await sor.sqlExe("SELECT COALESCE(SUM(total_size),0) used FROM knowledge_bases", {})
    used_bytes = int(rec[0].used) if rec else 0

used_mb = round(used_bytes / 1048576, 1)
pct = min(round(used_bytes / 104857600 * 100), 100) if used_bytes > 0 else 0

return {
    "widgettype": "VBox",
    "options": {"width": "100%", "bgcolor": "#f8f9fa", "padding": "16px", "css": "card", "spacing": "4px"},
    "subwidgets": [
        {"widgettype": "HBox", "options": {"width": "100%", "justifyContent": "space-between"}, "subwidgets": [
            {"widgettype": "Text", "options": {"text": "💾 存储容量", "cfontsize": 14, "fontWeight": "bold", "color": "#333"}},
            {"widgettype": "Text", "options": {"text": "已用 " + str(used_mb) + "MB / 总计 100MB", "cfontsize": 12, "color": "#888"}}
        ]},
        {"widgettype": "VBox", "options": {"width": "100%", "cheight": 0.5, "bgcolor": "#e0e0e0"}, "subwidgets": [
            {"widgettype": "VBox", "options": {"cwidth": pct, "cheight": 0.5, "bgcolor": "#4a90d9"}}
        ]}
    ]
}

index.ui patch: Replace the hardcoded VBox card with:

{"widgettype": "urlwidget", "options": {"url": "{{entire_url('/rag/knowledge_bases_list/storage_card.dspy')}}"}

detail.ui architecture — tree + file_list.dspy with node_selected

The knowledge base detail page uses a split layout:

• Left: Tree widget (dir_tree) with editable CRUD for folder management
• Right: urlwidget (file_list_panel) loading file_list.dspy for the current folder
• Bind: tree's node_selected event reloads file_list.dspy with the selected node's {id, label} as query params

{
    "widgettype": "VBox",
    "id": "detail_content",
    "options": {"css": "filler", "padding": "16px", "spacing": "0"},
    "subwidgets": [
        {
            "widgettype": "urlwidget",
            "id": "file_list_panel",
            "options": {
                "url": "{{entire_url('./file_list.dspy')}}?kb_id={{params_kw.kb_id}}&id=__root__"
            }
        }
    ],
    "binds": [
        {
            "wid": "dir_tree",
            "event": "node_selected",
            "actiontype": "urlwidget",
            "target": "file_list_panel",
            "mode": "replace",
            "options": {
                "url": "{{entire_url('./file_list.dspy')}}?kb_id={{params_kw.kb_id}}"
            }
        }
    ]
}

file_list.dspy reads kb_id, id (folder_id), label from params_kw, queries documents for the given folder, and renders upload UI + file list.

Pitfall: detail.ui was modified on the server but overwritten by git pull because changes weren't committed. Always git add && git commit && git push after modifying .ui/.dspy files on the server.

RBAC permission path MUST include /rag/ prefix (post fbba95e)

The fbba95e commit added /rag/ prefix to all module URLs. Permissions created before this commit (without prefix) won't match. When adding new permissions:

• Path in DB: /rag/knowledge_bases_list/<file>.dspy (WITH /rag/)
• In .ui files: {{entire_url('/rag/knowledge_bases_list/<file>.dspy')}}

DSPY URL encoding — Chinese chars in query params break ahserver

ahserver rejects unencoded non-ASCII characters in URL query strings. entire_url() passes values literally — it does NOT percent-encode. Passing Chinese like &label=根目录 causes Invalid char in url query error.

Fix: Keep all DSPY URL query params ASCII-only. Derive Chinese labels internally:

# ❌ DON'T — Chinese in URL query
url = entire_url('./page.dspy') + '&label=根目录&id=__root__'

# ✅ DO — ASCII-only, derive label inside DSPY
url = entire_url('./page.dspy') + '&id=__root__'
# In page.dspy: ns = params_kw.copy(); label = '根目录' if ns.get('id') == '__root__' else ns.get('label', '')

Always curl-test after adding permissions:

# Login, then verify each endpoint
curl -s -b "$COOKIE" "http://localhost:9181/rag/knowledge_bases_list/file.dspy?_webbricks_=1" | head -c 200

401 = auth, 403 = permission path mismatch, 500 = dspy bug, 200 = OK.

RBAC dual-path hell — cleanup for /knowledge_bases_list/ requires login

After the /rag/ prefix was added, knowledge_bases_list pages should ALL require login (logined role), not public (any role). Use scripts/init_rbac_v2.py which auto-scans wwwroot and assigns logined to all business pages.

This ensures unauthenticated users cannot browse knowledge bases. Public endpoints (login, register, status, bricks) remain in any.

Jinja2 backslash template error — ** inside {{}} breaks .ui rendering

.ui files processed by the bui processor go through Jinja2 template rendering. Backslash-escaped quotes (\") inside {{}} expressions cause TemplateSyntaxError: unexpected char '\\'. This happens when HTML/JS snippets with inline event handlers are embedded in JSON strings.

Fix: Remove backslashes before quotes INSIDE {{...}} blocks:

-- Add logined for all KB paths
INSERT IGNORE INTO rolepermission (id, roleid, permid)
SELECT CONCAT('rp_', p.id), 'logined', p.id
FROM permission p
WHERE p.path LIKE '%knowledge_bases_list%'
AND NOT EXISTS (SELECT 1 FROM rolepermission rp WHERE rp.permid = p.id AND rp.roleid = 'logined');

-- Remove any public (any) access to KB pages
DELETE rp FROM rolepermission rp JOIN permission p ON rp.permid = p.id
WHERE p.path LIKE '%knowledge_bases_list%' AND rp.roleid = 'any';

-- Remove ghost permissions (files that no longer exist)
DELETE FROM rolepermission WHERE permid IN (
  SELECT id FROM permission WHERE path NOT LIKE '/api/%' AND path NOT LIKE '/bricks/%'
  AND path NOT LIKE '/i18n/%' AND path NOT LIKE '/_%'
);

Then flush Redis + restart:

redis-cli FLUSHDB
cd /d/rag/ragserver && bash stop.sh && pkill -9 -f ragserver.py; sleep 2; bash start.sh

Ghost permissions — deleted files leave orphaned DB entries

When a .dspy or .ui file is deleted from wwwroot/, its permission entries in the permission and rolepermission tables remain. These ghost permissions cause confusion: curl returns 500 (file not found) but RBAC passes (403 would mean permission issue). Always verify file existence when debugging 500s:

ls /d/rag/ragserver/pkgs/rag/wwwroot/knowledge_bases_list/get_kb_cards.dspy
# vs
mysql ... -e "SELECT * FROM permission WHERE path LIKE '%get_kb_cards%'"

If the file doesn't exist but the permission does → ghost. Delete the permission.

Git pull overwrites uncommitted server changes — user mandate

User preference (explicit): "代码永远在本地改，提交远程后，测试服务器git pull这样不会出问题". Never manually edit files on the server. Edit locally → git commit → git push → server git pull. Manual server edits are overwritten by git pull and cannot be recovered because they were never tracked.

cd /d/rag/ragserver/pkgs/rag
git stash    # save manual changes before pull
git pull
git stash pop  # restore after

Better: never manually edit on the server — edit locally, push, then pull on server.

Two-SSH-hop deployment pattern (when local can't reach target directly)

When the local machine can't SSH directly to the rag server but can reach an intermediate:

scp /tmp/file apitest@120.48.168.15:/d/apitest/
ssh apitest@120.48.168.15 "scp /d/apitest/file rag@rag.opencomputing.cn:/tmp/"
ssh apitest@120.48.168.15 "ssh rag@rag.opencomputing.cn 'command'"

urlwidget relative URL pitfall — use absolute paths in scripts

When a .ui page is loaded as a sub-widget via urlwidget within the shell, the browser's base URL is the shell page, NOT the sub-widget's URL. Relative URLs like ./upload_file.dspy in XHR/script resolve to the ROOT, not to the sub-widget directory. Always use absolute paths in script XHR calls: /rag/knowledge_bases_list/upload_file.dspy.

Upload completion refresh — put refreshList AFTER if/else

When uploading multiple files, the refresh call must be placed OUTSIDE the if/else block so it fires regardless of success/failure, after all uploads complete:

x.onload=function(){
    done++;
    if(x.status===200){ st.innerText='上传成功 '+done+'/'+total }
    else{ st.innerText='上传失败' }
    if(done>=total)refreshList()  // outside if/else — always runs after last upload
};

Voiceprint ahserver startup — auth bypass needed for dependency compatibility

When starting voiceprint with ahserver (python3 ah.py -p 9087), the imported ahserver may use a different aiohttp_auth version than what's installed. The installed aiohttp_auth.auth module may lack the setup() method, and auth.get_auth() may throw RuntimeError: auth_middleware not installed.

Fix — two changes needed:

1. In ahserver/ahserver/auth_api.py, add return at the start of setupAuth() (skip auth middleware setup)
2. In ahserver/ahserver/processorResource.py, change self.user = await auth.get_auth(request) to self.user = None # auth disabled

Dependencies needed (install once):

pip install aiohttp_auth aiohttp_session aiohttp_cors aiohttp_middlewares openpyxl rsa qrcode asyncssh

Start command:

cd /share/ymq/run/voiceprint
PYTHONPATH=.:sqlor:appPublic:ahserver:longtasks/longtasks \
  nohup python3 ah.py -p 9087 > logs/voiceprint.log 2>&1 &

Verify: curl http://localhost:9087/api/status → should return JSON with "ready": true

Face API parameter format — images (array), NOT image

The face-service accepts {"images": ["base64_str", ...]} — an array of base64 strings. Sending {"image": "..."} (singular, non-array) returns 500.