pipeline_core/skills_library/all/harnessed-module-development/SKILL.md

name	description	author	related_skills	linked_files
harnessed-module-development	Development patterns, pitfalls, and conventions for Sage modules — URL paths, JSON config, TabPanel syntax, API patterns, and uapi gateway integration for vendor APIs	Hermes Agent		references common-pitfalls.md cross-database-etl-pattern.md

Harnessed Module Development Guide

Workflow: Extending Existing Modules

Before modifying any existing Sage module, read and understand the complete existing implementation first.

Pitfall: When asked to add a feature to an existing module (e.g., credit limit to accounting), agents may invent new logic instead of extending existing patterns, modify the wrong layer (sageapi vs sage/pkgs/accounting), or miss existing integration points.

Correct workflow:

1. Load relevant skills (e.g., accounting-module-example) to understand architecture
2. Read ALL related source files to understand existing patterns
3. Identify where the new feature hooks into existing code (e.g., leg_accounting() for credit limit)
4. Follow existing patterns exactly — don't invent new approaches
5. Verify against a checklist of all integration points

User correction example: "accounting有一套完整的记账功能，只是需要你读懂并增加信用额度能力" — read existing code first, never invent new balance-update logic.

External Vendor API Integration (uapi Gateway)

When a module needs to call external vendor/third-party APIs, use the Sage uapi gateway (UpAppApi) instead of writing direct HTTP client code. This is the platform-native pattern. Key pattern: module config table maps vendor → upappid + api_mapping(JSON); UpAppApi.call(upappid, apiname, callerid, params) routes through uapi templates; AK/SK lives in uapi's upappkey table (not the calling module); api_mapping is flexible JSON — zero code changes to add vendors. Do NOT write direct HTTP clients (e.g. volcengine_client.py) — they duplicate uapi's signing, templating, and streaming capabilities. Details: references/uapi-gateway-integration.md; full pattern in "External API Design Patterns" below.

Module Architecture

• Reasoning layer: context analysis, task decomposition, safety checks, planning
• Execution layer: tool calls, memory, skills, workflows, remote skills
• Reasoning feeds execution plans to agent module
• Shared database schema, common RBAC auth, complementary APIs

Module Permission Registration: scripts/load_path.py Pattern

CRITICAL: Each module's RBAC permissions are managed in the module's own scripts/load_path.py, NOT in sage/load_path.py (the latter is a legacy artifact). Follow the product_management/scripts/load_path.py pattern:

module_name/
├── scripts/
│   └── load_path.py     ← RBAC permission registration (run from sage dir)
├── json/
├── models/
└── wwwroot/

scripts/load_path.py Template

#!/usr/bin/env python3
"""module_name RBAC 权限脚本. Usage: cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py"""
import subprocess, os, sys

def find_sage_root():
    for c in [os.path.expanduser("~/repos/sage"), os.path.expanduser("~/sage")]:
        if os.path.isdir(os.path.join(c, "py3")) and os.path.isdir(os.path.join(c, "wwwroot")):
            return c
    return None

SAGE_ROOT = find_sage_root()
if not SAGE_ROOT: print("ERROR: Cannot find Sage root"); sys.exit(1)
PYTHON = os.path.join(SAGE_ROOT, "py3", "bin", "python")
SET_PERM_SCRIPT = os.path.join(SAGE_ROOT, "set_role_perm.py")
MOD = "module_name"

PATHS_ANY = [f"/{MOD}/menu.ui"]
PATHS_LOGINED = [f"/{MOD}", f"/{MOD}/page.ui", f"/{MOD}/api/endpoint.dspy"]  # + all module paths

def run_set_perm(role, path):
    return subprocess.run([PYTHON, SET_PERM_SCRIPT, role, path], capture_output=True, text=True).returncode == 0

def register_role_paths(role, paths):
    count = sum(1 for p in paths if run_set_perm(role, p))
    print(f"  {role}: {count}/{len(paths)} paths registered"); return count

def main():
    total = register_role_paths("any", PATHS_ANY) + register_role_paths("logined", PATHS_LOGINED)
    print(f"Done. Total {total} permission entries registered.")

if __name__ == "__main__":
    main()

Run: cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py

Rules:

• NEVER modify sage/load_path.py for module permissions
• Every new page, API, CRUD directory, and .dspy file needs a path entry
• Paths follow URL convention: /modulename/path (no wwwroot in URLs)
• Use logined for authenticated endpoints, any for public ones
• Restart Sage after running to reload RBAC cache

CRUD JSON: data_filter Pattern

When a CRUD list needs search/filter functionality, define data_filter in the params section:

{
    "tblname": "llm",
    "params": {
        "data_url": "{{entire_url('../api/llm_list.dspy')}}",
        "data_filter": {
            "AND": [
                {"field": "name", "op": "LIKE", "var": "name_input"},
                {"field": "model", "op": "LIKE", "var": "model_input"},
                {"field": "providerid", "op": "=", "var": "providerid_input"},
                {"field": "upappid", "op": "=", "var": "upappid_input"}
            ]
        },
        "filter_labels": {"name_input": "名称", "model_input": "识别名", "providerid_input": "供应商", "upappid_input": "上位系统"}
    }
}

Backend .dspy receives data_filter as a JSON string, parses it, and feeds it to DBFilter:

#!/usr/bin/env python3
import json
from sqlor.filter import DBFilter

result = {'success': False, 'rows': [], 'total': 0}
try:
    dbname = get_module_dbname('module_name')
    page = int(params_kw.get('page', 1)); rows_per_page = int(params_kw.get('rows', 20))
    offset = (page - 1) * rows_per_page
    try:
        filterjson = json.loads(params_kw.get('data_filter')) if params_kw.get('data_filter') else None
    except (json.JSONDecodeError, TypeError):
        filterjson = None
    async with DBPools().sqlorContext(dbname) as sor:
        where_clause, filterdic = '', {}
        if filterjson:
            ns = dict(params_kw)
            for key, val in ns.items():          # auto-add % wildcards for LIKE vars
                if _is_like_var(filterjson, key) and val and '%' not in val:
                    ns[key] = f'%{val}%'
            conds = DBFilter(filterjson).gen(ns)
            if conds: where_clause, filterdic = f' WHERE {conds}', ns
        # count + paginated query: f"select count(*) as cnt from tablename{where_clause}" then data query
        result['success'] = True
except Exception as e:
    result['error'] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)

def _is_like_var(filterjson, varname):
    """Check if a var is used with LIKE operator in the filter tree."""
    if not filterjson: return False
    for key, val in filterjson.items():
        if key.upper() in ('AND', 'OR') and isinstance(val, list):
            for item in val:
                if _is_like_var(item, varname): return True
        elif key.upper() == 'NOT' and isinstance(val, dict):
            if _is_like_var(val, varname): return True
        elif isinstance(val, dict) and val.get('var') == varname:
            if val.get('op', '').upper() == 'LIKE': return True
    return False

Rules:

1. data_filter lives under params; uses AND/OR/NOT tree matching sqlor.filter.DBFilter
2. var names map to URL params from the frontend search form; filter_labels gives display labels
3. LIKE fields need % wildcards — auto-add on backend if not present
4. DBFilter.gen(ns) returns the WHERE clause; ns holds variable values
5. Dropdown fields (providerid, upappid) use browserfields.alters with uitype: "code" + dataurl for code table data

CRITICAL: URL & Path Rules

menu.ui URL Must Match JSON Alias

The menu.ui URLs must match the alias (or tblname if no alias) defined in the JSON CRUD files: {{entire_url('/module/alias_name')}} requires JSON to have "alias": "alias_name" (or "tblname": "alias_name").

• WRONG: menu URL /harnessed_agent/sessions but JSON alias hermes_sessions
• CORRECT: menu URL /harnessed_agent/hermes_sessions matching JSON alias
• .ui wrappers using entire_url('crud_alias') must use the exact alias from the JSON definition

wwwroot is INVISIBLE in URLs

The wwwroot directory is the document root — it NEVER appears in URL paths.

• WRONG: {{entire_url('../wwwroot/api/xxx.dspy')}} / /module/wwwroot/page.ui
• CORRECT: {{entire_url('../api/xxx.dspy')}} / /module/page.ui
• From json/ directory: same-module refs use ../api/endpoint.dspy or ../crud_alias; cross-module uses absolute /module_name/api/endpoint.dspy
• CRUD subtables url must use ../ prefix: "url": "{{entire_url('../handover_items_list')}}" targeting the CRUD alias in another JSON file

.ui File References Must Not Include Module Prefix

When a .ui file in wwwroot/ references another .ui in the same wwwroot/, use just the filename: {{entire_url('memory.ui')}} — NOT {{entire_url('harnessed_agent/memory.ui')}} (resolves to double-prefixed path).

CRITICAL: WSS WebSocket URL Routing — ALL paths include /wss/ prefix

Server logs confirm RBAC checks the FULL path including /wss/ (e.g. [debug] userid=None, path='/wss/harnessed_reasoning/reasoning_console.wss' permission check failed). Use that path verbatim everywhere: frontend entire_url() → {{entire_url('/wss/harnessed_reasoning/reasoning_console.wss')}}; RBAC permission paths and set_role_perm.py args → /wss/harnessed_reasoning/reasoning_console.wss. WRONG: /harnessed_reasoning/reasoning_console.wss (no /wss/) → permission check failed.

CRITICAL: editable Section Required

Every JSON list/crud definition MUST have an editable section — without it the framework doesn't know where to submit create/update/delete forms:

{
    "tblname": "table_name",
    "alias": "table_crud",
    "params": {
        "editable": {
            "new_data_url": "{{entire_url('../api/table_create.dspy')}}",
            "update_data_url": "{{entire_url('../api/table_update.dspy')}}",
            "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}"
        },
        "browserfields": { ... },
        "editexclouded": [ ... ]
    }
}

Pitfall: json/ CRUD configs must reference wwwroot files via relative paths skipping wwwroot — the framework resolves ../api/xxx.dspy from the module root, not from json/.

CRUD JSON Strict Validation Checklist

Validate EVERY field reference against the model definition in models/. Known mismatches:

File	Wrong Field	Correct Field (from model)
opportunities_list.json	org_id	(does not exist — remove)
opportunities_list.json	sales_stage	current_stage
opportunities_list.json	source	source_type
sales_stages_list.json	is_active	is_won_stage / is_lost_stage
stage_history_list.json	changed_by	changed_by_id / changed_by_name

Every CRUD JSON MUST have:

1. tblname root key matching a table in models/
2. params with at least sortby and browserfields
3. editable with new_data_url/update_data_url/delete_data_url (even if read-only)
4. All field names in browserfields.exclouded, browserfields.alters, editexclouded must exist in the model
5. alters use uitype: "code" with data array — never nest style objects
6. subtables[].url uses {{entire_url('../alias')}} with ../ prefix
7. editor.binds[].actiontype ∈ {urlwidget, method, script, registerfunction, event}

When adding model fields, ALWAYS update init/data.json seed data (missing fields → config gaps after fresh deployment).

Model float/decimal Fields

MUST have BOTH length (int) and dec (int) as separate numeric keys. WRONG: "length": "15,2" (string). CORRECT: "length": 15, "dec": 2.

ID Generation: Always Use getID(), Never uuid.uuid4()

id columns are VARCHAR(32); str(uuid.uuid4()).replace('-','') is a 32-char hex string that can exceed column length → DataError: (1406, "Data too long for column 'id' at row 1"):

# WRONG: new_id = str(uuid.uuid4()).replace('-', '')
# CORRECT:
from appPublic.uniqueID import getID
new_id = getID()

Applies to ALL Python backend code (core.py) AND .dspy API files — same scheme as the framework's uniqueID module.

CRITICAL: TabPanel Correct Syntax

Widgettype "TabPanel" (NOT "Tab" — doesn't exist); parameter items array (NOT tabs); each item has name, label, content; content directly embeds a widget description object (e.g. urlwidget); tab_pos: "top" (default)/"bottom"/"left"/"right".

{"widgettype": "TabPanel", "options": {"tab_pos": "top", "items": [
    {"name": "sessions", "label": "推理会话", "icon": "history",
     "content": {"widgettype": "urlwidget", "options": {"url": "{{entire_url('crud_alias_or_file.ui')}}"}}}
]}}

CRUD UI File Pattern

Tabular (list views):

{
    "widgettype": "Tabular",
    "options": {
        "width": "100%", "height": "100%",
        "data_url": "{{entire_url('api/list_endpoint.dspy')}}",
        "data_method": "GET",
        "page_rows": 20,
        "row_options": {
            "fields": [
                {"name": "id", "width": 80, "frozen": true},
                {"name": "field_name", "title": "中文标题", "width": 150},
                {"name": "status", "title": "状态", "width": 100, "uitype": "code",
                 "data": [{"value": "active", "text": "活跃"}, {"value": "inactive", "text": "非活跃"}]}
            ],
            "editexclouded": ["id", "created_at"]
        },
        "editable": {"new_data_url": null, "update_data_url": null, "delete_data_url": null}
    }
}

Form (config/edit views):

{
    "widgettype": "Form",
    "id": "form_id",
    "options": {
        "data_url": "{{entire_url('api/get_endpoint.dspy')}}",
        "data_method": "GET",
        "submit_url": "{{entire_url('api/save_endpoint.dspy')}}",
        "method": "POST",
        "layout": "vertical",
        "fields": [...],
        "buttons": [{"type": "submit", "label": "保存", "variant": "primary"}],
        "maxWidth": "500px"
    },
    "binds": [{"wid": "self", "event": "submited", "actiontype": "script",
               "script": "await bricks.show_resp_message_or_error(event.params)"}]
}

.dspy API Pattern

CRITICAL: ahserver .dspy files return data via return, NOT print() — print() goes to stdout, ahserver receives NoneType. Always return json.dumps(result, ensure_ascii=False, default=str).

List endpoint (rows/total):

#!/usr/bin/env python3
import json
result = {'success': False, 'rows': [], 'total': 0}
try:
    dbname = get_module_dbname('module_name')
    user_id = await get_user()
    sql = """SELECT id, name, status, created_at FROM table_name WHERE user_id = ${user_id}$ ORDER BY created_at DESC"""
    async with DBPools().sqlorContext(dbname) as sor:
        data = await sor.sqlExe(sql, {'user_id': user_id})
        if isinstance(data, dict):
            result['total'] = data.get('total', 0)
            result['rows'] = [dict(r) for r in data.get('rows', [])]
        else:
            result['rows'] = [dict(r) for r in (data or [])]
            result['total'] = len(result['rows'])
        result['success'] = True
except Exception as e:
    result['error'] = str(e)
return json.dumps(result, ensure_ascii=False, default=str)

Get config endpoint: same skeleton; result = {'success': False, 'config': {}}; SELECT * FROM config_table WHERE user_id = ${user_id}$ LIMIT 1; set result['config'] = dict(rows[0]) or defaults.

Save config endpoint (returns Message widget — both success and error):

#!/usr/bin/env python3
import json, uuid, time
result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid', 'type': 'error'}}
try:
    dbname = get_module_dbname('module_name')
    user_id = await get_user()
    now = time.strftime('%Y-%m-%d %H:%M:%S')
    async with DBPools().sqlorContext(dbname) as sor:
        rows = await sor.sqlExe("SELECT id FROM table WHERE user_id = ${user_id}$", {'user_id': user_id})
        if rows:
            await sor.sqlExe("UPDATE table SET ... WHERE id = ${id}$", {...})
        else:
            await sor.sqlExe("INSERT INTO table ... VALUES (...)", {...})
    result = {'widgettype': 'Message', 'options': {'title': 'Success', 'message': '保存成功', 'type': 'success'}}
except Exception as e:
    result['options'] = {'title': 'Error', 'message': '保存失败: ' + str(e), 'type': 'error'}
return json.dumps(result, ensure_ascii=False)

File Organization

module_name/
├── json/                     # CRUD metadata (table_crud.json list, table_edit.json form)
├── models/                   # table.json (canonical, CRUD+DDL) + table.xlsx (source)
├── wwwroot/
│   ├── menu.ui, index.ui, page.ui, table_crud.ui
│   └── api/                  # table_list/create/update/delete.dspy
└── module_name/
    ├── __init__.py           # Empty
    ├── init.py               # load_module() registers to ServerEnv
    └── core.py               # Business logic

Convert xlsx→json with ~/repos/sage/xlsx2json_models.py (see database-table-definition-spec for format).

Debugging Reference (references/)

• module-init-deferred-io.md — deferred I/O, graceful degradation, per-subsystem try/except
• websocket-ui-debugging.md — WS diagnosis; WS Registered = backend fine, problem frontend; "等待连接" causes; wid not find/HTML not registered/ReferenceError
• sage-login-debugging.md — login flow (RC4, users, remember_user); silent password mismatch; RBAC 401; restart after set_role_perm.py
• bricks-ui-pitfalls.md — Html NOT HTML; no Scroll (VBox+CSS); Form buttons no external binds; uitype:"text"+height; raw JS WS; get_session()
• sage-session-debugging.md — remember_user()→auth.remember()→ticket→Redis; wwwroot diff; Jinja2 calls
• sage-auth-and-websocket.md — cookie+Redis; WS auth Sec-WebSocket-Protocol; RBAC path formats; roles; users schema
• sage-deployment-architecture.md — wwwroot gitignored (symlinks); global_menu.ui in dashboard_for_sage; 'NoneType'...be_call = missing bricks static
• sage-background-coroutines.md — add_cleanupctx/add_startup registry; safety matrix; standalone pattern; main-branch only
• sage-testing-environment.md — test URLs/credentials; Chrome CDP; 502 flowchart; RBAC cache refresh
• sage-i18n-system.md — MiniI18N+bricks.js; msg.txt encoding; regex fallback for template .ui files
• sage-environment.md — password encoding; commands; required tables
• vendor-callback-pattern.md — vendor webhook .dspy; POST parsing (JSON+query fallback); idempotency; paths_any; Bearer auto-resolves (no downapp_id); no internal IDs
• debugging-guide.md — deployed vs source; info() vs debug(); 'NoneType' get; non-JSON LLM; key decryption
• event-dispatcher-cache-pattern.md — sqlor C/U/D events; LRU+TTL; naming {db}:{tbl}:{c|u|d}:{before|after}; RBAC this/self bug
• user-isolation-pattern.md — context chain; users/{user_id}/; _get_user_dir(); per-user ws_push; hermes_skills
• reasoning-visualization.md — event flow; per-user WS callback; skill search
• multi-process-cache-invalidation.md — Redis Pub/Sub pattern

Verifying Jinja2 template rendering via curl

When a .ui file uses Jinja2 templates like {{ get_user() }}, verify rendering:

curl -s http://localhost:9180/module/page.ui | grep -o "user_id:.*" | head -1
# 'user_id: None' = correct (not logged in); raw '{{ get_user() }}' = template NOT processed
#   (served as static file, not through ahserver renderer); 'current_user' = hardcoded, wrong

Testing: Use Browser Tools, NOT curl

CRITICAL: When testing Sage web features, ALWAYS use browser tools (browser_navigate, browser_click, browser_type, etc.) — NEVER curl. curl cannot carry the browser's session cookies through the RBAC authentication flow → false negatives. The browser handles cookie sessions (AIOHTTP_SESSION), JS widget rendering (bricks), and form submissions.

browser_navigate(url='http://localhost:9180/module/page.ui')
browser_snapshot()
# Login-required pages: navigate to /index.ui, type username/password (superuser / Kyy@123456), submit, then navigate to target
browser_navigate(url='http://localhost:9180/harnessed_reasoning/reasoning_console.ui')

Common Pitfalls Checklist

• Stay focused on the current module — on "你跑飞啦，停，聚焦在XX模块", stop cross-module investigation immediately; user redirects when ready
• Understand the requirement before coding — "完全错误" = fundamentally misunderstood: stop, re-read, ask
• Be proactive with configuration — with credentials/paths in hand, configure immediately ("该配什么配什么"), don't ask permission per step
• Use existing RBAC modules, don't recreate — never custom login in sage/wwwroot/ when /rbac/user/login.ui exists; don't duplicate existing module files
• Follow established standards & verify all affected files — load bricks-framework, crud-definition-spec, module-development-spec before changes; zero tolerance at critical moments
• entire_url paths must NOT include wwwroot
• JSON CRUD files MUST have editable with new_data_url/update_data_url/delete_data_url
• TabPanel: widgettype TabPanel NOT Tab; items NOT tabs; content directly embeds widget description
• SQL params use ${param}$ in .dspy; sor.sqlExe(sql, ns) ALWAYS needs the 2nd ns arg ({} if none)
• .dspy returns json.dumps() strings; CRUD save endpoints return Message widget format
• get_module_dbname('module_name') for db name; await get_user() for user ID
• json/ files reference wwwroot files via ../ relative paths; entire_url() args MUST be quoted strings (same for subtable url)
• .dspy wrapped by framework in async def myfunc() — py_compile "await outside function" errors are expected/normal
• CRUD .dspy API files go in wwwroot/api/
• DELETE/UPDATE must include AND user_id = ${user_id}$ for multi-user isolation
• Python backend: NEVER sor.sqlExe() with ORDER BY or LIMIT — use sor.R(table, ns_dict) + Python slicing; ALL queries filter by user_id
• sor.R(tablename, ns, filters=None) — 2nd arg ns is a SINGLE dict with BOTH filters AND sort/page options; no ns= keyword, no 3rd-arg filters for normal queries
• NEVER create DBPools() in __init__() — create locally in each function needing DB access
• NEVER db.sqlorContext('default') — always the actual module dbname (e.g. 'harnessed_agent', 'customer_management')
• Debugging: rf.register('password', ...) in app/rf.py must be uncommented (else login silently fails "user name or password error"); decode_password typo config.getConfig() → getConfig(); restart Sage after
• Always report the root cause — zero tolerance for silent failures ("出错了不报告出错原因吗"); explain WHY with logs/curl/error text
• Self-test with available tools before asking the user ("你自己操作浏览器测试"); curl + log analysis as fallback when browser unavailable
• WS Registered in logs but UI "等待连接" → RBAC path missing or without /wss/ (server log shows exact path checked)
• Deployed code runs from site-packages, not repo — use log line numbers to identify version
• hasattr(config.website, 'ssl') is True even when value is None — check if self.conf.website.ssl: before accessing attributes
• Use info() for debug output, not debug() (may be filtered by log level)
• Core tool wrappers must execute REAL operations, not return mock dicts (read_file, write_file, terminal, execute_code, memory, skill_manage, todo)
• Tool wrappers accept context param and use _get_user_dir() for user isolation
• Widgettype casing: Html NOT HTML (else "widgetBuild(): HTML not registered"); Scroll doesn't exist — use VBox + style: "overflow-y: auto;"
• Form internal buttons can't have external binds — standalone Button widgets OUTSIDE the Form for custom handlers (else "desc wid not find")
• Frontend JS in HTML widgets: use {{ get_user() }} for user_id, NOT hardcoded 'current_user' (JS runs client-side, no server session)
• Raw JS WebSocket more reliable than bricks.WebSocket widget (avoids ReferenceError timing — handlers must exist before widget init)

Python Backend: Database Query Patterns

CRITICAL: sor.R() signature is R(tablename, ns, filters=None) — the 2nd arg ns is a SINGLE dict containing BOTH filter conditions AND sort/page options. No ns= keyword needed.

• Sorting — put 'sort' in the same dict: rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'}). WRONG: sor.R('users', {'status': 'active'}, ns={'sort': ...}); WRONG: sqlExe with ORDER BY
• Limit/OFFSET — Python slicing, NOT SQL LIMIT: rows = (rows or [])[:10]; offset: (rows or [])[20:20+10]
• Multi-user isolation — every query filters user_id: await sor.R('hermes_sessions', {'user_id': user_id, 'sort': 'started_at desc'})
• $or conditions go in the ns dict: await sor.R('hermes_skills', {'user_id': user_id, '$or': [{'name': {'$like': '%keyword%'}}, {'description': {'$like': '%keyword%'}}]})

Use sor.R(table, ns_dict)	Use sor.sqlExe(sql, params)
Simple CRUD reads with filtering	INSERT/UPDATE/DELETE operations
Sorting via 'sort' in ns dict	Complex joins or subqueries
Multi-user isolation with user_id filter	When sor.R can't express the query

LLM Client Pattern (harnessed_agent)

CRITICAL: harnessed_agent is an LLM CLIENT, not a server — it calls external LLM provider APIs (aiohttp POST /v1/chat/completions to OpenAI/DashScope/DeepSeek/SiliconFlow); it does NOT serve chat endpoints to others.

llm_client.py registers 5 functions to ServerEnv: llm_chat(messages, model, temperature, ...) → OpenAI response dict; llm_chat_stream(messages, ...) → async generator yielding {delta, finish_reason, raw}; llm_simple(prompt, system) → plain text; llm_list_models() → provider model list; llm_get_config() → current config (key masked).

result = await llm_chat(messages=[{"role": "system", "content": "You are helpful"}, {"role": "user", "content": "Hello"}], model="qwen3-max", temperature=0.7)
# result: {"choices": [{"message": {"content": "..."}}], "usage": {...}}
async for chunk in llm_chat_stream(messages=[...]): text = chunk['delta']
answer = await llm_simple("What is 2+2?", system="Answer briefly")

Provider presets (harnessed_agent_config table): llm_provider (preset name: dashscope default / openai / deepseek / siliconflow / empty=custom); llm_service_url (base URL, auto-filled from preset or custom); llm_api_key (Bearer token); default_model (e.g. qwen-plus); default_temperature / top_p (floats: length=5, dec=2). Preset URLs: dashscope https://dashscope.aliyuncs.com/compatible-mode/v1; openai https://api.openai.com/v1; deepseek https://api.deepseek.com/v1; siliconflow https://api.siliconflow.cn/v1

Resilience REQUIRED: retry with exponential backoff (3 attempts: timeout/500/connection failure); 429 rate limit → read Retry-After, wait, retry; structured logging via appPublic.log (info/warning/error: request params, response timing, token counts); error propagation as OpenAI-compatible {"error": {"message": ..., "type": ..., "code": N}}.

harnessed_reasoning Pattern: LLM-Based Reasoning Engine

harnessed_reasoning is a REAL reasoning engine, not a mock — uses harnessed_agent's LLM client + tool execution. Flow: reasoning_console.ui (Form) → reasoning_submit.dspy → hermes_reason_and_execute() → LLM planning (llm_chat) + tool execution (harnessed_execute_tool) → results stored in DB.

1. Context gathering: harnessed_get_intelligent_memory_context + session search + skill search
2. LLM planning: llm_chat() with system prompt incl. tool descriptions → JSON execution plan
3. Safety check: validates plan against configurable rules (strict/moderate/lenient) — blocks dangerous commands like rm -rf /
4. Tool execution: if safe & execute_immediately=True, harnessed_execute_tool() per action
5. Error recovery: auto-recovers (read_file not found → search_files; permission denied → strip sudo prefix)
6. Session storage: harnessed_reasoning_sessions table

17 tools: read_file, write_file, search_files, patch, terminal, process, execute_code, memory, skill_manage, skill_view, todo, session_search, cronjob, clarify, delegate_task, text_to_speech, vision_analyze

Config (harnessed_reasoning_config): model_name (default qwen3-max), temperature/top_p, system_prompt (overrides default), safety_mode (strict/moderate/lenient), max_reasoning_steps, max_tool_calls_per_step, enable_error_recovery.

Sage Multi-Process Deployment Architecture

SO_REUSEPORT: Multiple Workers Share One Port

ConfiguredServer.run() sets reuse_port=True on Linux — multiple sage.py processes bind the same port; kernel distributes connections (nginx/gunicorn-style). start.sh:

WORKERS=$(nproc)  # auto-detect CPU cores
for (( i=0; i<WORKERS; i++ )); do
    nohup ./py3/bin/python app/sage.py --workdir "$WORKDIR" --port $PORT > "logs/sage_worker_${i}.log" 2>&1 &
done

No load balancer or port range needed.

Background Coroutines Must NOT Run in Every Worker

add_cleanupctx(coro) (async ctx manager: startup+cleanup) / add_startup(coro) (startup only) / asyncio.create_task(...) in module init.py attach to aiohttp hooks — in multi-process mode EVERY worker starts its own copy → duplicate work, double-charging, DB race conditions. Known: llmage add_cleanupctx(start_backend) → backend_accounting() (10s billing loop); unipay add_startup(setup_callback_path) (route registration — safe to duplicate, aiohttp registration is idempotent per process; background tasks MUST be extracted).

Fix: extract background coroutines into standalone programs, start them ONCE in start.sh BEFORE sage.py workers, remove add_cleanupctx/add_startup from init.py.

Standalone program template (from llmage backend_accounting):

#!/usr/bin/env python
import os, sys, asyncio, signal
os.chdir(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.insert(0, 'py3/lib/python3.10/site-packages'); sys.path.insert(0, 'pkgs')
from appPublic.folderUtils import ProgramPath
from appPublic.jsonConfig import getConfig
from sqlor.dbpools import DBPools
from appPublic.log import MyLogger, info, exception
from llmage.accounting import get_accounting_llmusages, llm_accounting, llm_accoung_failed

p = ProgramPath()
config = getConfig(NS={'workdir': os.getcwd(), 'ProgramPath': p})
DBPools(config.databases)   # standalone must init manually (sage.py workers get it from webapp())

async def backend_accounting():
    info('backend accounting started ...')
    while True:
        try:
            lus = await get_accounting_llmusages()
        except Exception as e:
            exception(f'{e}'); lus = []
        for lu in lus:
            try:
                await llm_accounting(lu)
            except Exception as e:
                exception(f'{e}, {lu.id=}'); await llm_accoung_failed(lu.id)
        await asyncio.sleep(10)

def main():
    loop = asyncio.new_event_loop(); asyncio.set_event_loop(loop)
    signal.signal(signal.SIGTERM, lambda s, f: [t.cancel() for t in asyncio.all_tasks(loop)] or loop.stop())
    try:
        loop.run_until_complete(backend_accounting())
    except asyncio.CancelledError:
        pass
    finally:
        loop.close()

if __name__ == '__main__':
    main()

start.sh order: 1) background programs first: nohup $PYTHON bin/backend_accounting.py > logs/backend_accounting.log 2>&1 &, record name:pid in sage_backend.pid; 2) sage workers (plain pid per line in sage.pid). stop.sh: kill workers from sage.pid, then background programs via while IFS=: read name pid; do kill $pid 2>/dev/null; done < sage_backend.pid.

Pitfalls:

• add_cleanupctx = async context manager (yield; startup + cleanup); add_startup = startup only. Both per-process hooks
• PID file: name:pid format for background programs (named stop), plain pid for workers
• Background programs must handle SIGTERM (cancel asyncio tasks) or kill leaves DB connections inconsistent
• Standalone programs must init DBPools(config.databases) manually; use MyLogger with a dedicated log file (they don't inherit Sage's log config)

Sage standalone scripts — DBPools pattern

from appPublic.jsonConfig import getConfig  # NOT appPublic.getConfig
from sqlor.dbpools import DBPools
import asyncio
config = getConfig('.')
db = DBPools(config.databases)
dbname = list(config.databases.keys())[0]
async def run():
    async with db.sqlorContext(dbname) as sor: ...
asyncio.run(run())

Inside Sage server context (.dspy, module code): use get_sor_context(env, 'modulename') instead.

EventDispatcher Cache Invalidation is Process-Scoped — Use Redis Pub/Sub

sqlor C/U/D lifecycle events only fire within the current process. In SO_REUSEPORT deployment, worker A's invalidation never reaches workers B–N → stale cache. Fix: Redis Pub/Sub channel sage:cache:invalidate — each worker keeps a local cache (fast reads) + subscribes; publish on invalidation, all workers evict; add TTL fallback for missed messages. See references/multi-process-cache-invalidation.md. Naming: {dbname}:{tablename}:{c|u|d}:{before|after}. Known bug: RBAC uses this instead of self in userperm.py.

CRITICAL: uapi.headers Template Rendering — No Helper Functions Needed

uapi.headers is a JSON template string; variables render at runtime before each HTTP request: {{apikey}} → decrypted API key from upappkey table; {{jsondata}} → JSON request body; {{response}} → response transformation template.

• Vidu-style Token auth: {"Content-Type": "application/json", "Authorization": "Token {{apikey}}"} — NO helper function; do NOT add token() to uapi/appapi.py, the template engine handles {{apikey}} natively
• OpenAI-compatible: Bearer {{apikey}}; complex signing: {{bearer(apikey)}} helper

External API Design Patterns (uapi detail)

Tables: upapp (vendor definition), uapi (API endpoints per vendor; upappid + name combo unique — contains path, httpmethod, headers JSON template w/ {{apikey}}, data JSON template w/ params, response output transformation), upappkey (API credentials apikey/secretkey, encrypted), uapiio (I/O schemas).

Client → dapi(Bearer) → your_module/api/xxx.dspy
  → vendor_id → lookup upappid+apiname in module config table
  → from uapi.uapi import UpAppApi; resp = await UpAppApi(request).call(upappid, apiname, callerid, params)
  → uapi gateway renders headers/data/response templates → HTTP → vendor

Module config table registers per-vendor mappings: vendor: "volcengine", upappid: "upapp-volcengine-01" (refs upapp.id), apiname_create_session: "CreateVisualValidateSession", apiname_get_result: "GetVisualValidateResult", apiname_create_asset: "CreateAsset" — one apiname per vendor operation.

from uapi.uapi import UpAppApi
async def call_vendor_api(vendor_id, action, params):
    config = await get_vendor_config(vendor_id)
    upappid = config.upappid
    apiname = getattr(config, f'apiname_{action}')
    ua = UpAppApi(request)
    resp = await ua.call(upappid, apiname, await get_user(), params)
    return json.loads(resp.decode('utf-8'))   # response template may have transformed it

Use uapi gateway	Use direct HTTP client
Standard REST/JSON APIs	Non-HTTP protocols (gRPC, WebSocket to vendor)
Vendor has OpenAPI/REST interface	Complex HMAC signing not expressible in templates
Multiple vendors for same op (config-driven, zero code)	One-off API with no reuse pattern
Need response templating/transformation	Streaming/chunked responses with custom parsing

• Bearer token auth: dapi auto-resolves identity — client-facing .dspy APIs get user_id via await get_user() and org_id via await get_userorgid(). NEVER add downapp_id/client_id/manual identification params — the Bearer token IS the identifier
• Never expose internal DB IDs to clients — client API responses/params use only vendor-side identifiers (vendor_group_id not local_group_id)
• Upload endpoints accept vendor-side IDs — validate ownership via rl_org_group(org_id, vendor_group_id), use internal local_group_id only for local FK relationships
• Vendor callbacks are paths_any (vendor POSTs have no session) — NOT paths_logined; callback idempotency mandatory (vendors retry — check existing mapping/status before insert)

Schema Migration Pattern (xlsx models)

When modifying model fields (adding/removing columns):

1. Update the .xlsx in models/
2. Migration script: (a) check old column exists, (b) create new tables/indexes, (c) migrate data, (d) optionally drop old column — make it idempotent (safe to run multiple times)
3. Copy updated files to pkgs/ and reinstall: cd pkgs/module && pip install -e .
4. Restart Sage after code changes

pyproject.toml Dependencies

ONLY declare sqlor and bricks_for_python. Do NOT declare ahserver, apppublic, appbase, rbac — these are installed by build.sh, not pip:

dependencies = ["sqlor", "bricks_for_python"]

Python Backend: DBPools() Lifecycle, Singleton Fork Safety, sqlorContext() Module Name

Three critical rules for all Python backend code (core.py, etc.):

Rule 1: DBPools() must be created in function scope, NEVER in __init__() — WRONG: self.db = DBPools() in __init__; CORRECT: db = DBPools() inside each async method that needs DB access.

Rule 2: DBPools is a Singleton (@SingletonDecorator) — in forked child processes (where .dspy files run) the inherited parent instance persists; DBPools(config.databases) silently discards new args and returns the old instance with empty/stale databases.

Rule 3: sqlorContext() takes a database KEY (e.g. 'crm_db'), NOT a module name — hardcoding 'harnessed_agent' fails (key doesn't exist in config.databases); env.get_module_dbname() resolves modules to the actual key.

Complete required pattern for ALL database access:

from ahserver.serverenv import ServerEnv
from appPublic.jsonConfig import getConfig
from sqlor.dbpools import DBPools

async def my_query():
    env = ServerEnv()
    dbname = env.get_module_dbname('my_module')   # resolves DB key dynamically (e.g. 'crm_db')
    config = getConfig()
    db = DBPools()                                # Singleton instance (fork-safe)
    db.databases = config.databases               # MUST force-set: overrides inherited empty dict
    async with db.sqlorContext(dbname) as sor:
        rows = await sor.R('table', {...})

NEVER hardcode a database name in sqlorContext(). NEVER pass config.databases as a constructor arg to DBPools() — silently ignored due to the Singleton.

harnessed_agent: Tool Permissions, No Mocks, Context Propagation

Internal Calls Must Not Be Blocked

_get_user_permissions() in harnessed_agent/core.py must NOT restrict permissions for empty/missing context — internal workflow calls pass context=None; restricting makes write_file, memory, clarify fail with "Insufficient permissions to execute tool 'X'". Grant full permissions unconditionally:

def _get_user_permissions(self, context):
    # Internal system calls should not be blocked by permission checks
    return ['file_read', 'file_write', 'system_execute', 'system_manage', 'browser_access',
            'ai_vision', 'ai_tts', 'memory_manage', 'memory_read', 'skill_read', 'skill_manage',
            'task_manage', 'task_delegate', 'user_interact', 'schedule_manage', 'config_read']

Tool Implementation: No Mocks

harnessed_agent/tools/base_tools.py wrappers MUST execute real operations, not return status: "mock_implementation" dicts (reasoning engine reports fake success, LLM hallucinates file locations). Real implementations: read_file/write_file/search_files/patch = actual file I/O; terminal = asyncio.create_subprocess_shell; execute_code = temp .py + python3; memory = ~/.hermes/memory.json; skill_view/skills_list = ~/.hermes/skills/; todo = ~/.hermes/todo.json. Browser (browser_*), vision (vision_analyze), TTS (text_to_speech) need external drivers — may legitimately return structured note responses.

Keep base_tools.py Exports in Sync

Tool dicts (file_tools, system_tools, skill_tools, ...) are imported by BOTH __init__.py and registration.py. After modifying base_tools.py, verify all three are consistent — removing/renaming a dict or merging two (e.g. skill_tools into memory_tools) without updating the others → ImportError/KeyError at registration time.

Tool Wrappers Must Accept context for User Isolation

HERMES_DIR = os.path.expanduser("~/.hermes")

def _get_user_dir(base_dir, context=None):
    """User-isolated subdirectory; falls back to global dir if no user context."""
    user_id = (context or {}).get('user_id') or (context or {}).get('userid')
    return os.path.join(base_dir, "users", str(user_id)) if user_id else base_dir

async def wrapped_skill_manage(action, name, context=None, **kwargs):
    user_dir = _get_user_dir(HERMES_DIR, context)
    skills_dir = os.path.join(user_dir, "skills", name)
    # ... file operations in user-isolated directory

Context propagation chain (every step must pass context):

1. reasoning_console.wss → engine.reason_and_execute(user_id=X)
2. _execute_tool() → harnessed_execute_tool(tool, params, context={user_id: X})
3. → agent.execute_tool_call(tool, params, context)
4. → _execute_tool_with_retry() injects context via inspect.signature — CRITICAL: check signature first, else tools without context fail "unexpected keyword argument"
5. Tool wrapper receives context, uses _get_user_dir()

User-isolated structure: ~/.hermes/users/{user_id}/ → skills/ (SKILL.md per subdir), memory.json, todo.json, tmp/ (execute_code temp files). Context-accepting wrappers: wrapped_skill_manage/wrapped_skill_view/wrapped_skills_list → users/{user_id}/skills/; wrapped_memory → memory.json; wrapped_todo → todo.json; wrapped_execute_code → tmp/.

harnessed_execute_tool: Must Accept and Pass context

# CORRECT — accepts and forwards context:
async def harnessed_execute_tool(tool_name, parameters, context=None):
    return await get_harnessed_agent().execute_tool_call(tool_name, parameters, context)

If context is omitted, the user_id embedded in it is lost and tools execute as "anonymous".

_get_current_user_id Must NOT Raise

Return "anonymous" when context is missing, NOT raise ValueError (internal calls/system workflows often lack full context):

def _get_current_user_id(self, context):
    user_id = (context or {}).get('user_id') or (context or {}).get('userid')
    return str(user_id) if user_id else "anonymous"

Table Name: hermes_skills (NOT harnessed_skills)

Model is models/hermes_skills.json; all SQL in core.py must use hermes_skills. harnessed_skills does NOT exist → "table not found".

Per-User WebSocket Callbacks (NOT shared ws_push)

A shared ws_push attribute breaks with concurrent users (they overwrite each other's callbacks):

class HermesReasoningEngine:
    ws_push_callbacks: Dict[str, callable] = {}   # per-user callbacks

    async def _push(self, event_type, data=None, user_id=None):
        if user_id and user_id in self.ws_push_callbacks:
            await self.ws_push_callbacks[user_id]({'event': event_type, 'data': data})

In reason_and_execute(): set self._current_user_id = user_id, pass to all _push() calls; cleanup finally: self._current_user_id = None. In .wss endpoint: engine.ws_push_callbacks[user_id] = callback; cleanup engine.ws_push_callbacks.pop(user_id, None).

Shared Skills Permission: Owner Org Only

Shared skills (~/.hermes/skills/) readable by ALL users, writable ONLY by owner organization users (org_id='0'); non-owner gets "共享技能仅允许所有者机构用户修改". reason_and_execute() must set self._current_org_id from ServerEnv's orgid/org_id attribute and include it in the context dict: context = {"user_id": user_id, "org_id": self._current_org_id, ...}.

user_id Must Be in Context for Tool Execution

If user_id is missing from the context passed to harnessed_execute_tool, tools run as 'anonymous' → permission checks and data isolation failures:

context = await self._get_memory_context(user_id, request, config)
context['user_id'] = user_id  # CRITICAL: must be in context for tool execution
# _get_memory_context initializes: {"user_id": user_id, "memory_entries": [], "recent_sessions": [], "skills": []}

Reasoning Engine: Execution Details

LLM Call: Do NOT Pass model Parameter

harnessed_reasoning/core.py _llm_call() must NOT pass model= to llm_chat() — let llm_chat resolve default_model from harnessed_agent_config:

# WRONG:  result = await env.llm_chat(messages=messages, model=model, ...)
# CORRECT: result = await env.llm_chat(messages=messages, temperature=temperature, max_tokens=max_tokens, **extra)

execute_immediately Parameter Parsing

Frontend may send true (boolean or string), not just '1' — support multiple truthy values:

execute_val = str(params_kw.get('execute_immediately', '1')).lower()
execute_immediately = execute_val in ('1', 'true', 'yes', 'on')

Module Function Signature Consistency

When a .dspy passes user_id to a module function, ALL functions in the call chain must accept it — else unexpected keyword argument 'user_id':

async def hermes_reason_and_execute(request: str, execute_immediately: bool = True, user_id: str = None):
    return await engine.reason_and_execute(request, execute_immediately=execute_immediately, user_id=user_id)

async def reason_and_execute(self, request: str, execute_immediately: bool = True, user_id: str = None):
    if not user_id: user_id = "anonymous"

Non-JSON LLM Response Handling

LLM API may return HTML error page / proxy block — check content-type before parsing and log the body:

if resp.status == 200:
    content_type = resp.content_type
    if 'json' not in content_type:
        err_text = await resp.text()
        error(f"[llm_response] Non-JSON from {url}, Content-Type={content_type}")
        error(f"[llm_response] Body (first 2000): {err_text[:2000]}")
        return {'error': {'message': f'Non-JSON response ({content_type})', 'type': 'content_type_error'}}
    return await resp.json()

Store Session: JSON Serialization Safety

json.dumps(plan) may hit non-serializable types (datetime, custom objects). Clean before serialization (or use default=str):

def clean_plan(obj):
    if isinstance(obj, dict): return {k: clean_plan(v) for k, v in obj.items()}
    if isinstance(obj, list): return [clean_plan(i) for i in obj]
    if isinstance(obj, datetime): return obj.isoformat()
    return obj

data['execution_plan_json'] = json.dumps(clean_plan(plan), ensure_ascii=False)  # or json.dumps(plan, default=str)
await sor.C('harnessed_reasoning_sessions', data)

Database Table Must Exist Before Use

Failed to store session: 'NoneType' object has no attribute 'get' often means the table doesn't exist — sqlor.getTableDesc() returns None for missing tables and C() crashes on None['fields']. Ensure tables are created via build.sh before running.

LLM Config Database Isolation & Debugging

llm_client.py _get_llm_config() reads harnessed_agent_config. When called from another module (e.g. integrated_crm_app calling llm_chat()), lookup can fail (Failed to fetch LLM config from DB 'default': 'NoneType' object has no attribute 'get' → hardcoded model=qwen3-max fallback, NOT the user's configured value). Root causes: (1) db context — env.get_module_dbname('harnessed_agent') throws → falls back to default; (2) table missing in queried database; (3) llm_api_key stored encrypted → must decrypt via env.password_decode().

Correct implementation — try module DB first, then default; decrypt api_key; sort updated_at desc:

async def _get_llm_config():
    dbnames_to_try = ['default']
    try:
        env = ServerEnv()
        module_db = env.get_module_dbname('harnessed_agent')
        if module_db and module_db not in dbnames_to_try:
            dbnames_to_try.insert(0, module_db)
    except Exception as e:
        error(f"[llm_config] Exception: {e}")
    for dbname in dbnames_to_try:
        try:
            async with DBPools().sqlorContext(dbname) as sor:
                ns = {'sort': 'updated_at desc'}
                if user_id: ns['user_id'] = user_id
                rows = (await sor.R('harnessed_agent_config', ns)) or []
                if rows:
                    row = rows[0]
                    if row.get('llm_api_key'):
                        row['llm_api_key'] = ServerEnv().password_decode(row['llm_api_key'])
                    return row
        except Exception as e:
            error(f"Failed to fetch LLM config from DB '{dbname}': {e}")
    return {}

Encrypted API Key Decryption

Fields stored encrypted (like llm_api_key, api_key) must be decrypted before use:

api_key = row.get('llm_api_key', '')
if api_key:
    api_key = ServerEnv().password_decode(api_key)

CRITICAL: Reuse Existing RBAC Login — Do NOT Write Your Own

ALWAYS use the existing RBAC user login system. NEVER create up_login.dspy/login.ui in sage/wwwroot/ or any module's wwwroot/:

• Login page: /rbac/user/login.ui; login handler: /rbac/user/up_login.dspy
• RBAC already handles: RC4 password encryption with config.password_key, account lockout detection, session management via remember_user(), redirect to userinfo, multiple login methods (password, SMS, WeChat)
• All modules authenticate via this shared session; custom login in sage/wwwroot/ breaks the RBAC auth flow → session inconsistencies

Password Handling: Use ServerEnv, NOT rf (RegisterFunction)

# CORRECT — ServerEnv's password_encode from ahserver.globalEnv (reads key from config.password_key, RC4):
from ahserver.globalEnv import password_encode
encrypted_pw = password_encode(params_kw.password)

# WRONG — legacy RF pattern, do NOT use:
await rfexe('password', params_kw)  # rf.register('password', ...) is deprecated; may not be registered

Decrypt with ServerEnv().password_decode(value).

User ID Retrieval

• In .dspy: userid = await get_user() (returns user ID string); org: await get_userorgid()

Sage Authentication & Cookie Pitfalls

Cookie Secure Flag for HTTP Development

ahserver/auth_api.py's EncryptedCookieStorage defaults to secure=True → blocks cookies on HTTP (localhost). Check the SSL value, not just key existence:

ssl_enabled = False
if hasattr(self.conf.website, 'ssl') and self.conf.website.ssl:   # hasattr is True even when value is None!
    ssl_cfg = self.conf.website.ssl
    if hasattr(ssl_cfg, 'crtfile') and hasattr(ssl_cfg, 'keyfile'):
        ssl_enabled = True
storage = EncryptedCookieStorage(secret, secure=ssl_enabled,  # False for HTTP, True for HTTPS
    samesite='Lax', httponly=True, max_age=24*60*60)

Login Form Field Names & Users Table

• Form fields: username and passwd (NOT loginid/password); passwd encrypted via password_encode()
• Table users (NOT user); login matches on username (e.g. superuser), not id; id is the user ID (e.g. user-01) used for RBAC/session
• Schema: id VARCHAR(32) PK, username VARCHAR(100), password VARCHAR(255), orgid VARCHAR(32), user_status VARCHAR(1) ('0'=active), login_fail_count INT, last_login_fail DATETIME

RBAC Permission Roles

Role	Description
anonymous	Unauthenticated users
any	All users (including anonymous)
logined	Authenticated users only
owner.*	Owner organization roles

Feature pages requiring login → logined; public pages (login, registration) → any/anonymous.

Module Navigation & Adding Features

menu.ui Pattern

Sage modules are navigated via menu.ui files, not standalone index.ui pages. Main Sage wwwroot/menu.ui references module submenus:

{"name": "llmage", "label": "模型管理", "submenu": "{{entire_url('/llmage/menu.ui')}}"}

Module menu.ui (Menu widget; "url" = direct nav to page or CRUD alias, "submenu" = nested menu.ui, "items" = inline submenu, all URLs via {{entire_url()}}):

{"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"width": "60%", "height": "75%"}, "items": [
    {"name": "feature1", "label": "功能1", "url": "{{entire_url('/module/feature.ui')}}"},
    {"name": "feature2", "label": "功能2", "url": "{{entire_url('/module/alias_name')}}"}
]}}

Menu with auth check (JSON validation fails on .ui files with Jinja2 templates — expected/normal):

{"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"archor": "cc", "width": "70%", "height": "70%"}, "cwidth": 10, "items": [
{% if get_user() %}{"name": "entry_name", "label": "入口名", "url": "{{entire_url('page.ui')}}"}{% endif %}
]}}

Adding New Module Features

1. Create the .ui file in module's wwwroot/
2. Symlink to Sage wwwroot (critical for local development): cd ~/repos/sage/wwwroot/module_name && ln -sf ~/repos/module_name/wwwroot/new_feature.ui .
3. Add menu entry in module_name/wwwroot/menu.ui
4. Register RBAC: ./py3/bin/python set_role_perm.py logined /module_name/new_feature.ui
5. For API .dspy files: symlink into module_name/api/ and register /module_name/api/new_api.dspy

WebSocket Real-Time Event Push (Reasoning Console Pattern)

WSS URL paths

See "WSS WebSocket URL Routing" under URL & Path Rules — /wss/ prefix mandatory everywhere (UI {{entire_url}}, JS URL, RBAC registration, set_role_perm.py arg). Verify via server log [debug] userid=None, path='/wss/...' permission check failed — use that path verbatim.

UI Layout (two approaches)

A: WebSocket widget + Html handlers (framework-managed): {"widgettype": "WebSocket", "id": "...", "options": {"ws_url": "{{entire_url('/wss/module/endpoint.wss')}}", "with_session": true}, "binds": [{"wid": "self", "event": "onopen", "actiontype": "script", "script": "onWsOpen()"}, {"wid": "self", "event": "ontext", "actiontype": "script", "script": "onWsMessage(event.params)"}]} plus an Html widget defining window.onWsOpen/onWsMessage — fragile ordering (see rule 3). B: Pure Html widget + raw JS WebSocket (recommended, full control) — single Html widget with inline <script> doing new WebSocket(wsUrl, session).

Key rules:

1. Multi-line textarea: Form field with uitype: "text" + height (not single-line Input)
2. Form buttons can't have external binds — standalone Button widgets OUTSIDE the Form for custom click handlers
3. bricks.WebSocket binds are fragile: handlers must exist in global scope BEFORE widget init (Html after WebSocket → ReferenceError: onWsOpen is not defined); Approach B avoids this
4. Widgettype casing: Html NOT HTML; Scroll doesn't exist — use VBox + style: "overflow-y: auto;"
5. Session passing: with_session: true → bricks passes session via Sec-WebSocket-Protocol header → ahserver WebsocketProcessor identifies the user
6. Read Form value: bricks.getWidgetById('input_form', bricks.app).get_value('user_input')
7. User ID in JS: {{ get_user() }} Jinja2 injection — JS has no server session; hardcoded 'current_user' → backend receives the literal string, all ops run as user_id='anonymous'

Bricks widgets may not be ready when the HTML widget's script runs — delay connect and add retry: setTimeout(function() { window.reasoningWS.connect(); }, 500);

Backend: .wss WebSocket Endpoint

Create wwwroot/your_endpoint.wss — defines async def myfunc(request, **kwargs); ahserver's WebsocketProcessor wraps it and provides ws_pool and ws_data in kwargs:

"""Module WebSocket endpoint for real-time event push."""
import json, asyncio, time
from appPublic.uniqueID import getID
from appPublic.log import info, debug, error, exception

_module_ws_sessions = {}   # user_id -> {'ws_pool': ..., 'session_id': ...}

async def myfunc(request, **kwargs):
    ws_pool = kwargs.get('ws_pool'); ws_data = kwargs.get('ws_data')
    try:
        data = json.loads(ws_data) if ws_data else {}
    except:
        data = {}
    cmd = data.get('cmd', '')
    if cmd == 'connect':
        user_id = data.get('user_id', 'anonymous')
        _module_ws_sessions[user_id] = {'ws_pool': ws_pool, 'session_id': data.get('session_id', getID())}
        await ws_pool.sendto(json.dumps({'type': 'connected',
            'session_id': _module_ws_sessions[user_id]['session_id'], 'message': 'WebSocket connected'}))
    elif cmd == 'start_action':
        user_id = data.get('user_id', 'anonymous'); request_text = data.get('request', '')
        if not request_text:
            await ws_pool.sendto(json.dumps({'type': 'error', 'message': 'Empty request'})); return
        await _ws_push(user_id, {'type': 'action_start', 'data': {'request': request_text}})
        asyncio.create_task(_run_action(user_id, request_text))   # non-blocking
    elif cmd == 'ping':
        await ws_pool.sendto(json.dumps({'type': 'pong', 'timestamp': time.time()}))

async def _ws_push(user_id, message):
    session = _module_ws_sessions.get(user_id)
    if session and session.get('ws_pool'):
        try:
            await session['ws_pool'].sendto(json.dumps(message))
        except Exception as e:
            error(f"WS push failed for user {user_id}: {e}")

async def _run_action(user_id, request_text):
    from your_module.core import get_engine
    engine = get_engine()
    engine.ws_push = lambda msg: _ws_push(user_id, msg)   # inject callback
    try:
        result = await engine.run(request_text, user_id=user_id)
        await _ws_push(user_id, {'type': 'action_complete', 'data': {'result': result}})
    except Exception as e:
        await _ws_push(user_id, {'type': 'error', 'data': {'message': str(e)}})
    finally:
        engine.ws_push = None

Python core.py: Event Push Points

class MyEngine:
    ws_push = None  # Async callback, injected by .wss endpoint

    async def _push(self, event_type: str, data: dict = None):
        if self.ws_push:
            try:
                await self.ws_push({'event': event_type, 'data': data or {}, 'timestamp': time.time()})
            except Exception as e:
                error(f"ws_push failed: {e}")

    async def run(self, request: str, user_id: str = None):
        await self._push('start', {'request': request, 'message': 'Starting...'})
        # ... await self._push('step_1', {'message': 'Collecting context'}) ...
        await self._push('complete', {'message': 'Done', 'result': result})

_push MUST be async def and awaited; push at every meaningful state transition (start, step start/complete, error, finish); callback injected by the .wss endpoint, not the engine; use asyncio.create_task() in .wss so the websocket isn't blocked.

Frontend JS WebSocket

var protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
var url = protocol + '//' + window.location.host + '/wss/module_name/endpoint.wss';   // include /wss/!
var session = '';
try { if (window.bricks && window.bricks.app && window.bricks.app.get_session) session = window.bricks.app.get_session(); } catch(e) {}
var ws = session ? new WebSocket(url, session) : new WebSocket(url);

WebSocket(url, protocol) — 2nd param is the sub-protocol string; WebsocketProcessor reads it from Sec-WebSocket-Protocol to identify the session.

Event flow: action_start → step_context → step_plan → step_safety → execution_start → step_N_start → tool_call → tool_result → step_N_complete → execution_complete → action_complete Message format: {"event": "step_name", "data": {"message": "..."}, "timestamp": 1234567890} or type-based: {"type": "error", "data": {"message": "..."}}

For production, extract the real user_id from the websocket handshake headers (cookie/session) rather than trusting the frontend-provided value.

RBAC for WebSocket

python set_role_perm.py "logined" "/wss/harnessed_reasoning/reasoning_console.wss"

"等待连接" / WebSocket never connects — checklist

1. HTML widget JS has hardcoded user_id: 'current_user' → must be {{ get_user() }}
2. RBAC permission for the .wss path missing or registered without /wss/ prefix (server log shows the exact path RBAC checks)
3. Redis not running (Sage sessions depend on redis://127.0.0.1:6379)
4. .wss file missing at wwwroot/endpoint.wss or doesn't define async def myfunc(request, **kwargs)