--- name: harnessed-module-development description: Development patterns, pitfalls, and conventions for Sage modules — URL paths, JSON config, TabPanel syntax, API patterns, and uapi gateway integration for vendor APIs author: Hermes Agent related_skills: [] linked_files: references: - common-pitfalls.md - cross-database-etl-pattern.md --- # Harnessed Module Development Guide ## Workflow: Extending Existing Modules **Before modifying any existing Sage module, read and understand the complete existing implementation first.** Pitfall: When asked to add a feature to an existing module (e.g., credit limit to accounting), agents may invent new logic instead of extending existing patterns, modify the wrong layer (sageapi vs sage/pkgs/accounting), or miss existing integration points. Correct workflow: 1. Load relevant skills (e.g., `accounting-module-example`) to understand architecture 2. Read ALL related source files to understand existing patterns 3. Identify where the new feature hooks into existing code (e.g., `leg_accounting()` for credit limit) 4. Follow existing patterns exactly — don't invent new approaches 5. Verify against a checklist of all integration points User correction example: "accounting有一套完整的记账功能，只是需要你读懂并增加信用额度能力" — read existing code first, never invent new balance-update logic. ## External Vendor API Integration (uapi Gateway) **When a module needs to call external vendor/third-party APIs, use the Sage uapi gateway (`UpAppApi`) instead of writing direct HTTP client code.** This is the platform-native pattern. Key pattern: module config table maps `vendor → upappid + api_mapping(JSON)`; `UpAppApi.call(upappid, apiname, callerid, params)` routes through uapi templates; AK/SK lives in uapi's `upappkey` table (not the calling module); `api_mapping` is flexible JSON — zero code changes to add vendors. **Do NOT write direct HTTP clients** (e.g. `volcengine_client.py`) — they duplicate uapi's signing, templating, and streaming capabilities. Details: `references/uapi-gateway-integration.md`; full pattern in "External API Design Patterns" below. ## Module Architecture - Reasoning layer: context analysis, task decomposition, safety checks, planning - Execution layer: tool calls, memory, skills, workflows, remote skills - Reasoning feeds execution plans to agent module - Shared database schema, common RBAC auth, complementary APIs ## Module Permission Registration: scripts/load_path.py Pattern **CRITICAL: Each module's RBAC permissions are managed in the module's own `scripts/load_path.py`, NOT in `sage/load_path.py`** (the latter is a legacy artifact). Follow the `product_management/scripts/load_path.py` pattern: ``` module_name/ ├── scripts/ │ └── load_path.py ← RBAC permission registration (run from sage dir) ├── json/ ├── models/ └── wwwroot/ ``` ### scripts/load_path.py Template ```python #!/usr/bin/env python3 """module_name RBAC 权限脚本. Usage: cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py""" import subprocess, os, sys def find_sage_root(): for c in [os.path.expanduser("~/repos/sage"), os.path.expanduser("~/sage")]: if os.path.isdir(os.path.join(c, "py3")) and os.path.isdir(os.path.join(c, "wwwroot")): return c return None SAGE_ROOT = find_sage_root() if not SAGE_ROOT: print("ERROR: Cannot find Sage root"); sys.exit(1) PYTHON = os.path.join(SAGE_ROOT, "py3", "bin", "python") SET_PERM_SCRIPT = os.path.join(SAGE_ROOT, "set_role_perm.py") MOD = "module_name" PATHS_ANY = [f"/{MOD}/menu.ui"] PATHS_LOGINED = [f"/{MOD}", f"/{MOD}/page.ui", f"/{MOD}/api/endpoint.dspy"] # + all module paths def run_set_perm(role, path): return subprocess.run([PYTHON, SET_PERM_SCRIPT, role, path], capture_output=True, text=True).returncode == 0 def register_role_paths(role, paths): count = sum(1 for p in paths if run_set_perm(role, p)) print(f" {role}: {count}/{len(paths)} paths registered"); return count def main(): total = register_role_paths("any", PATHS_ANY) + register_role_paths("logined", PATHS_LOGINED) print(f"Done. Total {total} permission entries registered.") if __name__ == "__main__": main() ``` Run: `cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py` Rules: - **NEVER modify `sage/load_path.py`** for module permissions - Every new page, API, CRUD directory, and `.dspy` file needs a path entry - Paths follow URL convention: `/modulename/path` (no `wwwroot` in URLs) - Use `logined` for authenticated endpoints, `any` for public ones - Restart Sage after running to reload RBAC cache ## CRUD JSON: data_filter Pattern When a CRUD list needs search/filter functionality, define `data_filter` in the `params` section: ```json { "tblname": "llm", "params": { "data_url": "{{entire_url('../api/llm_list.dspy')}}", "data_filter": { "AND": [ {"field": "name", "op": "LIKE", "var": "name_input"}, {"field": "model", "op": "LIKE", "var": "model_input"}, {"field": "providerid", "op": "=", "var": "providerid_input"}, {"field": "upappid", "op": "=", "var": "upappid_input"} ] }, "filter_labels": {"name_input": "名称", "model_input": "识别名", "providerid_input": "供应商", "upappid_input": "上位系统"} } } ``` Backend `.dspy` receives `data_filter` as a JSON string, parses it, and feeds it to `DBFilter`: ```python #!/usr/bin/env python3 import json from sqlor.filter import DBFilter result = {'success': False, 'rows': [], 'total': 0} try: dbname = get_module_dbname('module_name') page = int(params_kw.get('page', 1)); rows_per_page = int(params_kw.get('rows', 20)) offset = (page - 1) * rows_per_page try: filterjson = json.loads(params_kw.get('data_filter')) if params_kw.get('data_filter') else None except (json.JSONDecodeError, TypeError): filterjson = None async with DBPools().sqlorContext(dbname) as sor: where_clause, filterdic = '', {} if filterjson: ns = dict(params_kw) for key, val in ns.items(): # auto-add % wildcards for LIKE vars if _is_like_var(filterjson, key) and val and '%' not in val: ns[key] = f'%{val}%' conds = DBFilter(filterjson).gen(ns) if conds: where_clause, filterdic = f' WHERE {conds}', ns # count + paginated query: f"select count(*) as cnt from tablename{where_clause}" then data query result['success'] = True except Exception as e: result['error'] = str(e) return json.dumps(result, ensure_ascii=False, default=str) def _is_like_var(filterjson, varname): """Check if a var is used with LIKE operator in the filter tree.""" if not filterjson: return False for key, val in filterjson.items(): if key.upper() in ('AND', 'OR') and isinstance(val, list): for item in val: if _is_like_var(item, varname): return True elif key.upper() == 'NOT' and isinstance(val, dict): if _is_like_var(val, varname): return True elif isinstance(val, dict) and val.get('var') == varname: if val.get('op', '').upper() == 'LIKE': return True return False ``` Rules: 1. `data_filter` lives under `params`; uses `AND`/`OR`/`NOT` tree matching `sqlor.filter.DBFilter` 2. `var` names map to URL params from the frontend search form; `filter_labels` gives display labels 3. LIKE fields need `%` wildcards — auto-add on backend if not present 4. `DBFilter.gen(ns)` returns the WHERE clause; `ns` holds variable values 5. Dropdown fields (providerid, upappid) use `browserfields.alters` with `uitype: "code"` + `dataurl` for code table data ## CRITICAL: URL & Path Rules ### menu.ui URL Must Match JSON Alias The `menu.ui` URLs must match the `alias` (or `tblname` if no alias) defined in the JSON CRUD files: `{{entire_url('/module/alias_name')}}` requires JSON to have `"alias": "alias_name"` (or `"tblname": "alias_name"`). - WRONG: menu URL `/harnessed_agent/sessions` but JSON alias `hermes_sessions` - CORRECT: menu URL `/harnessed_agent/hermes_sessions` matching JSON alias - `.ui` wrappers using `entire_url('crud_alias')` must use the exact alias from the JSON definition ### `wwwroot` is INVISIBLE in URLs The `wwwroot` directory is the document root — it NEVER appears in URL paths. - WRONG: `{{entire_url('../wwwroot/api/xxx.dspy')}}` / `/module/wwwroot/page.ui` - CORRECT: `{{entire_url('../api/xxx.dspy')}}` / `/module/page.ui` - From `json/` directory: same-module refs use `../api/endpoint.dspy` or `../crud_alias`; cross-module uses absolute `/module_name/api/endpoint.dspy` - CRUD subtables `url` must use `../` prefix: `"url": "{{entire_url('../handover_items_list')}}"` targeting the CRUD alias in another JSON file ### .ui File References Must Not Include Module Prefix When a `.ui` file in `wwwroot/` references another `.ui` in the same `wwwroot/`, use just the filename: `{{entire_url('memory.ui')}}` — NOT `{{entire_url('harnessed_agent/memory.ui')}}` (resolves to double-prefixed path). ### CRITICAL: WSS WebSocket URL Routing — ALL paths include `/wss/` prefix Server logs confirm RBAC checks the FULL path including `/wss/` (e.g. `[debug] userid=None, path='/wss/harnessed_reasoning/reasoning_console.wss' permission check failed`). Use that path verbatim everywhere: frontend `entire_url()` → `{{entire_url('/wss/harnessed_reasoning/reasoning_console.wss')}}`; RBAC permission paths and `set_role_perm.py` args → `/wss/harnessed_reasoning/reasoning_console.wss`. WRONG: `/harnessed_reasoning/reasoning_console.wss` (no `/wss/`) → `permission check failed`. ## CRITICAL: `editable` Section Required Every JSON list/crud definition MUST have an `editable` section — without it the framework doesn't know where to submit create/update/delete forms: ```json { "tblname": "table_name", "alias": "table_crud", "params": { "editable": { "new_data_url": "{{entire_url('../api/table_create.dspy')}}", "update_data_url": "{{entire_url('../api/table_update.dspy')}}", "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}" }, "browserfields": { ... }, "editexclouded": [ ... ] } } ``` Pitfall: `json/` CRUD configs must reference wwwroot files via relative paths skipping `wwwroot` — the framework resolves `../api/xxx.dspy` from the module root, not from `json/`. ## CRUD JSON Strict Validation Checklist Validate EVERY field reference against the model definition in `models/`. Known mismatches: | File | Wrong Field | Correct Field (from model) | |------|-------------|----------------------------| | opportunities_list.json | `org_id` | (does not exist — remove) | | opportunities_list.json | `sales_stage` | `current_stage` | | opportunities_list.json | `source` | `source_type` | | sales_stages_list.json | `is_active` | `is_won_stage` / `is_lost_stage` | | stage_history_list.json | `changed_by` | `changed_by_id` / `changed_by_name` | Every CRUD JSON MUST have: 1. `tblname` root key matching a table in `models/` 2. `params` with at least `sortby` and `browserfields` 3. `editable` with `new_data_url`/`update_data_url`/`delete_data_url` (even if read-only) 4. All field names in `browserfields.exclouded`, `browserfields.alters`, `editexclouded` must exist in the model 5. `alters` use `uitype: "code"` with `data` array — never nest `style` objects 6. `subtables[].url` uses `{{entire_url('../alias')}}` with `../` prefix 7. `editor.binds[].actiontype` ∈ {`urlwidget`, `method`, `script`, `registerfunction`, `event`} When adding model fields, ALWAYS update `init/data.json` seed data (missing fields → config gaps after fresh deployment). ### Model float/decimal Fields MUST have BOTH `length` (int) and `dec` (int) as separate numeric keys. WRONG: `"length": "15,2"` (string). CORRECT: `"length": 15, "dec": 2`. ### ID Generation: Always Use `getID()`, Never `uuid.uuid4()` `id` columns are VARCHAR(32); `str(uuid.uuid4()).replace('-','')` is a 32-char hex string that can exceed column length → `DataError: (1406, "Data too long for column 'id' at row 1")`: ```python # WRONG: new_id = str(uuid.uuid4()).replace('-', '') # CORRECT: from appPublic.uniqueID import getID new_id = getID() ``` Applies to ALL Python backend code (core.py) AND `.dspy` API files — same scheme as the framework's `uniqueID` module. ## CRITICAL: TabPanel Correct Syntax Widgettype `"TabPanel"` (NOT `"Tab"` — doesn't exist); parameter `items` array (NOT `tabs`); each item has `name`, `label`, `content`; `content` directly embeds a widget description object (e.g. urlwidget); `tab_pos`: `"top"` (default)/`"bottom"`/`"left"`/`"right"`. ```json {"widgettype": "TabPanel", "options": {"tab_pos": "top", "items": [ {"name": "sessions", "label": "推理会话", "icon": "history", "content": {"widgettype": "urlwidget", "options": {"url": "{{entire_url('crud_alias_or_file.ui')}}"}}} ]}} ``` ## CRUD UI File Pattern Tabular (list views): ```json { "widgettype": "Tabular", "options": { "width": "100%", "height": "100%", "data_url": "{{entire_url('api/list_endpoint.dspy')}}", "data_method": "GET", "page_rows": 20, "row_options": { "fields": [ {"name": "id", "width": 80, "frozen": true}, {"name": "field_name", "title": "中文标题", "width": 150}, {"name": "status", "title": "状态", "width": 100, "uitype": "code", "data": [{"value": "active", "text": "活跃"}, {"value": "inactive", "text": "非活跃"}]} ], "editexclouded": ["id", "created_at"] }, "editable": {"new_data_url": null, "update_data_url": null, "delete_data_url": null} } } ``` Form (config/edit views): ```json { "widgettype": "Form", "id": "form_id", "options": { "data_url": "{{entire_url('api/get_endpoint.dspy')}}", "data_method": "GET", "submit_url": "{{entire_url('api/save_endpoint.dspy')}}", "method": "POST", "layout": "vertical", "fields": [...], "buttons": [{"type": "submit", "label": "保存", "variant": "primary"}], "maxWidth": "500px" }, "binds": [{"wid": "self", "event": "submited", "actiontype": "script", "script": "await bricks.show_resp_message_or_error(event.params)"}] } ``` ## .dspy API Pattern **CRITICAL: ahserver `.dspy` files return data via `return`, NOT `print()`** — print() goes to stdout, ahserver receives `NoneType`. Always `return json.dumps(result, ensure_ascii=False, default=str)`. List endpoint (rows/total): ```python #!/usr/bin/env python3 import json result = {'success': False, 'rows': [], 'total': 0} try: dbname = get_module_dbname('module_name') user_id = await get_user() sql = """SELECT id, name, status, created_at FROM table_name WHERE user_id = ${user_id}$ ORDER BY created_at DESC""" async with DBPools().sqlorContext(dbname) as sor: data = await sor.sqlExe(sql, {'user_id': user_id}) if isinstance(data, dict): result['total'] = data.get('total', 0) result['rows'] = [dict(r) for r in data.get('rows', [])] else: result['rows'] = [dict(r) for r in (data or [])] result['total'] = len(result['rows']) result['success'] = True except Exception as e: result['error'] = str(e) return json.dumps(result, ensure_ascii=False, default=str) ``` Get config endpoint: same skeleton; `result = {'success': False, 'config': {}}`; `SELECT * FROM config_table WHERE user_id = ${user_id}$ LIMIT 1`; set `result['config'] = dict(rows[0])` or defaults. Save config endpoint (returns Message widget — both success and error): ```python #!/usr/bin/env python3 import json, uuid, time result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid', 'type': 'error'}} try: dbname = get_module_dbname('module_name') user_id = await get_user() now = time.strftime('%Y-%m-%d %H:%M:%S') async with DBPools().sqlorContext(dbname) as sor: rows = await sor.sqlExe("SELECT id FROM table WHERE user_id = ${user_id}$", {'user_id': user_id}) if rows: await sor.sqlExe("UPDATE table SET ... WHERE id = ${id}$", {...}) else: await sor.sqlExe("INSERT INTO table ... VALUES (...)", {...}) result = {'widgettype': 'Message', 'options': {'title': 'Success', 'message': '保存成功', 'type': 'success'}} except Exception as e: result['options'] = {'title': 'Error', 'message': '保存失败: ' + str(e), 'type': 'error'} return json.dumps(result, ensure_ascii=False) ``` ## File Organization ``` module_name/ ├── json/ # CRUD metadata (table_crud.json list, table_edit.json form) ├── models/ # table.json (canonical, CRUD+DDL) + table.xlsx (source) ├── wwwroot/ │ ├── menu.ui, index.ui, page.ui, table_crud.ui │ └── api/ # table_list/create/update/delete.dspy └── module_name/ ├── __init__.py # Empty ├── init.py # load_module() registers to ServerEnv └── core.py # Business logic ``` Convert xlsx→json with `~/repos/sage/xlsx2json_models.py` (see `database-table-definition-spec` for format). ## Debugging Reference (references/) - `module-init-deferred-io.md` — deferred I/O, graceful degradation, per-subsystem try/except - `websocket-ui-debugging.md` — WS diagnosis; `WS Registered` = backend fine, problem frontend; "等待连接" causes; `wid not find`/`HTML not registered`/`ReferenceError` - `sage-login-debugging.md` — login flow (RC4, users, remember_user); silent password mismatch; RBAC 401; restart after set_role_perm.py - `bricks-ui-pitfalls.md` — `Html` NOT `HTML`; no `Scroll` (VBox+CSS); Form buttons no external binds; `uitype:"text"`+height; raw JS WS; get_session() - `sage-session-debugging.md` — remember_user()→auth.remember()→ticket→Redis; wwwroot diff; Jinja2 calls - `sage-auth-and-websocket.md` — cookie+Redis; WS auth Sec-WebSocket-Protocol; RBAC path formats; roles; users schema - `sage-deployment-architecture.md` — wwwroot gitignored (symlinks); global_menu.ui in dashboard_for_sage; `'NoneType'...be_call` = missing bricks static - `sage-background-coroutines.md` — add_cleanupctx/add_startup registry; safety matrix; standalone pattern; main-branch only - `sage-testing-environment.md` — test URLs/credentials; Chrome CDP; 502 flowchart; RBAC cache refresh - `sage-i18n-system.md` — MiniI18N+bricks.js; msg.txt encoding; regex fallback for template .ui files - `sage-environment.md` — password encoding; commands; required tables - `vendor-callback-pattern.md` — vendor webhook `.dspy`; POST parsing (JSON+query fallback); idempotency; paths_any; Bearer auto-resolves (no downapp_id); no internal IDs - `debugging-guide.md` — deployed vs source; info() vs debug(); 'NoneType' get; non-JSON LLM; key decryption - `event-dispatcher-cache-pattern.md` — sqlor C/U/D events; LRU+TTL; naming `{db}:{tbl}:{c|u|d}:{before|after}`; RBAC `this`/`self` bug - `user-isolation-pattern.md` — context chain; users/{user_id}/; _get_user_dir(); per-user ws_push; hermes_skills - `reasoning-visualization.md` — event flow; per-user WS callback; skill search - `multi-process-cache-invalidation.md` — Redis Pub/Sub pattern ### Verifying Jinja2 template rendering via curl When a `.ui` file uses Jinja2 templates like `{{ get_user() }}`, verify rendering: ```bash curl -s http://localhost:9180/module/page.ui | grep -o "user_id:.*" | head -1 # 'user_id: None' = correct (not logged in); raw '{{ get_user() }}' = template NOT processed # (served as static file, not through ahserver renderer); 'current_user' = hardcoded, wrong ``` ## Testing: Use Browser Tools, NOT curl **CRITICAL: When testing Sage web features, ALWAYS use browser tools (`browser_navigate`, `browser_click`, `browser_type`, etc.) — NEVER curl.** curl cannot carry the browser's session cookies through the RBAC authentication flow → false negatives. The browser handles cookie sessions (AIOHTTP_SESSION), JS widget rendering (bricks), and form submissions. ```python browser_navigate(url='http://localhost:9180/module/page.ui') browser_snapshot() # Login-required pages: navigate to /index.ui, type username/password (superuser / Kyy@123456), submit, then navigate to target browser_navigate(url='http://localhost:9180/harnessed_reasoning/reasoning_console.ui') ``` ## Common Pitfalls Checklist - [ ] **Stay focused on the current module** — on "你跑飞啦，停，聚焦在XX模块", stop cross-module investigation immediately; user redirects when ready - [ ] **Understand the requirement before coding** — "完全错误" = fundamentally misunderstood: stop, re-read, ask - [ ] **Be proactive with configuration** — with credentials/paths in hand, configure immediately ("该配什么配什么"), don't ask permission per step - [ ] **Use existing RBAC modules, don't recreate** — never custom login in `sage/wwwroot/` when `/rbac/user/login.ui` exists; don't duplicate existing module files - [ ] **Follow established standards & verify all affected files** — load `bricks-framework`, `crud-definition-spec`, `module-development-spec` before changes; zero tolerance at critical moments - [ ] `entire_url` paths must NOT include `wwwroot` - [ ] JSON CRUD files MUST have `editable` with `new_data_url`/`update_data_url`/`delete_data_url` - [ ] TabPanel: widgettype `TabPanel` NOT `Tab`; `items` NOT `tabs`; `content` directly embeds widget description - [ ] SQL params use `${param}$` in `.dspy`; `sor.sqlExe(sql, ns)` ALWAYS needs the 2nd `ns` arg (`{}` if none) - [ ] `.dspy` returns `json.dumps()` strings; CRUD save endpoints return Message widget format - [ ] `get_module_dbname('module_name')` for db name; `await get_user()` for user ID - [ ] `json/` files reference wwwroot files via `../` relative paths; `entire_url()` args MUST be quoted strings (same for subtable `url`) - [ ] .dspy wrapped by framework in `async def myfunc()` — py_compile "await outside function" errors are expected/normal - [ ] CRUD .dspy API files go in `wwwroot/api/` - [ ] DELETE/UPDATE must include `AND user_id = ${user_id}$` for multi-user isolation - [ ] **Python backend: NEVER `sor.sqlExe()` with ORDER BY or LIMIT** — use `sor.R(table, ns_dict)` + Python slicing; ALL queries filter by user_id - [ ] **`sor.R(tablename, ns, filters=None)`** — 2nd arg `ns` is a SINGLE dict with BOTH filters AND sort/page options; no `ns=` keyword, no 3rd-arg filters for normal queries - [ ] **NEVER create `DBPools()` in `__init__()`** — create locally in each function needing DB access - [ ] **NEVER `db.sqlorContext('default')`** — always the actual module dbname (e.g. `'harnessed_agent'`, `'customer_management'`) - [ ] Debugging: `rf.register('password', ...)` in `app/rf.py` must be uncommented (else login silently fails "user name or password error"); `decode_password` typo `config.getConfig()` → `getConfig()`; restart Sage after - [ ] **Always report the root cause** — zero tolerance for silent failures ("出错了不报告出错原因吗"); explain WHY with logs/curl/error text - [ ] **Self-test with available tools** before asking the user ("你自己操作浏览器测试"); curl + log analysis as fallback when browser unavailable - [ ] `WS Registered` in logs but UI "等待连接" → RBAC path missing or without `/wss/` (server log shows exact path checked) - [ ] Deployed code runs from site-packages, not repo — use log line numbers to identify version - [ ] `hasattr(config.website, 'ssl')` is True even when value is None — check `if self.conf.website.ssl:` before accessing attributes - [ ] Use `info()` for debug output, not `debug()` (may be filtered by log level) - [ ] Core tool wrappers must execute REAL operations, not return mock dicts (read_file, write_file, terminal, execute_code, memory, skill_manage, todo) - [ ] Tool wrappers accept `context` param and use `_get_user_dir()` for user isolation - [ ] Widgettype casing: `Html` NOT `HTML` (else "widgetBuild(): HTML not registered"); `Scroll` doesn't exist — use VBox + `style: "overflow-y: auto;"` - [ ] Form internal buttons can't have external binds — standalone Button widgets OUTSIDE the Form for custom handlers (else "desc wid not find") - [ ] Frontend JS in HTML widgets: use `{{ get_user() }}` for user_id, NOT hardcoded `'current_user'` (JS runs client-side, no server session) - [ ] Raw JS WebSocket more reliable than `bricks.WebSocket` widget (avoids ReferenceError timing — handlers must exist before widget init) ## Python Backend: Database Query Patterns **CRITICAL: `sor.R()` signature is `R(tablename, ns, filters=None)`** — the 2nd arg `ns` is a SINGLE dict containing BOTH filter conditions AND sort/page options. No `ns=` keyword needed. - **Sorting** — put `'sort'` in the same dict: `rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'})`. WRONG: `sor.R('users', {'status': 'active'}, ns={'sort': ...})`; WRONG: `sqlExe` with ORDER BY - **Limit/OFFSET** — Python slicing, NOT SQL LIMIT: `rows = (rows or [])[:10]`; offset: `(rows or [])[20:20+10]` - **Multi-user isolation** — every query filters user_id: `await sor.R('hermes_sessions', {'user_id': user_id, 'sort': 'started_at desc'})` - **$or conditions** go in the ns dict: `await sor.R('hermes_skills', {'user_id': user_id, '$or': [{'name': {'$like': '%keyword%'}}, {'description': {'$like': '%keyword%'}}]})` | Use `sor.R(table, ns_dict)` | Use `sor.sqlExe(sql, params)` | |---|---| | Simple CRUD reads with filtering | INSERT/UPDATE/DELETE operations | | Sorting via `'sort'` in ns dict | Complex joins or subqueries | | Multi-user isolation with user_id filter | When sor.R can't express the query | ## LLM Client Pattern (harnessed_agent) **CRITICAL: harnessed_agent is an LLM CLIENT, not a server** — it calls external LLM provider APIs (`aiohttp POST /v1/chat/completions` to OpenAI/DashScope/DeepSeek/SiliconFlow); it does NOT serve chat endpoints to others. `llm_client.py` registers 5 functions to ServerEnv: `llm_chat(messages, model, temperature, ...)` → OpenAI response dict; `llm_chat_stream(messages, ...)` → async generator yielding `{delta, finish_reason, raw}`; `llm_simple(prompt, system)` → plain text; `llm_list_models()` → provider model list; `llm_get_config()` → current config (key masked). ```python result = await llm_chat(messages=[{"role": "system", "content": "You are helpful"}, {"role": "user", "content": "Hello"}], model="qwen3-max", temperature=0.7) # result: {"choices": [{"message": {"content": "..."}}], "usage": {...}} async for chunk in llm_chat_stream(messages=[...]): text = chunk['delta'] answer = await llm_simple("What is 2+2?", system="Answer briefly") ``` Provider presets (`harnessed_agent_config` table): `llm_provider` (preset name: `dashscope` default / `openai` / `deepseek` / `siliconflow` / empty=custom); `llm_service_url` (base URL, auto-filled from preset or custom); `llm_api_key` (Bearer token); `default_model` (e.g. `qwen-plus`); `default_temperature` / `top_p` (floats: length=5, dec=2). Preset URLs: dashscope `https://dashscope.aliyuncs.com/compatible-mode/v1`; openai `https://api.openai.com/v1`; deepseek `https://api.deepseek.com/v1`; siliconflow `https://api.siliconflow.cn/v1` Resilience REQUIRED: retry with exponential backoff (3 attempts: timeout/500/connection failure); 429 rate limit → read `Retry-After`, wait, retry; structured logging via `appPublic.log` (info/warning/error: request params, response timing, token counts); error propagation as OpenAI-compatible `{"error": {"message": ..., "type": ..., "code": N}}`. ## harnessed_reasoning Pattern: LLM-Based Reasoning Engine **harnessed_reasoning is a REAL reasoning engine, not a mock** — uses harnessed_agent's LLM client + tool execution. Flow: `reasoning_console.ui` (Form) → `reasoning_submit.dspy` → `hermes_reason_and_execute()` → LLM planning (`llm_chat`) + tool execution (`harnessed_execute_tool`) → results stored in DB. 1. **Context gathering**: `harnessed_get_intelligent_memory_context` + session search + skill search 2. **LLM planning**: `llm_chat()` with system prompt incl. tool descriptions → JSON execution plan 3. **Safety check**: validates plan against configurable rules (strict/moderate/lenient) — blocks dangerous commands like `rm -rf /` 4. **Tool execution**: if safe & `execute_immediately=True`, `harnessed_execute_tool()` per action 5. **Error recovery**: auto-recovers (read_file not found → search_files; permission denied → strip sudo prefix) 6. **Session storage**: `harnessed_reasoning_sessions` table 17 tools: read_file, write_file, search_files, patch, terminal, process, execute_code, memory, skill_manage, skill_view, todo, session_search, cronjob, clarify, delegate_task, text_to_speech, vision_analyze Config (`harnessed_reasoning_config`): `model_name` (default `qwen3-max`), `temperature`/`top_p`, `system_prompt` (overrides default), `safety_mode` (strict/moderate/lenient), `max_reasoning_steps`, `max_tool_calls_per_step`, `enable_error_recovery`. ## Sage Multi-Process Deployment Architecture ### SO_REUSEPORT: Multiple Workers Share One Port `ConfiguredServer.run()` sets `reuse_port=True` on Linux — multiple `sage.py` processes bind the same port; kernel distributes connections (nginx/gunicorn-style). `start.sh`: ```bash WORKERS=$(nproc) # auto-detect CPU cores for (( i=0; i "logs/sage_worker_${i}.log" 2>&1 & done ``` No load balancer or port range needed. ### Background Coroutines Must NOT Run in Every Worker `add_cleanupctx(coro)` (async ctx manager: startup+cleanup) / `add_startup(coro)` (startup only) / `asyncio.create_task(...)` in module `init.py` attach to aiohttp hooks — in multi-process mode EVERY worker starts its own copy → duplicate work, double-charging, DB race conditions. Known: `llmage` `add_cleanupctx(start_backend)` → `backend_accounting()` (10s billing loop); `unipay` `add_startup(setup_callback_path)` (route registration — safe to duplicate, aiohttp registration is idempotent per process; background *tasks* MUST be extracted). **Fix**: extract background coroutines into standalone programs, start them ONCE in `start.sh` BEFORE sage.py workers, remove `add_cleanupctx`/`add_startup` from `init.py`. Standalone program template (from llmage `backend_accounting`): ```python #!/usr/bin/env python import os, sys, asyncio, signal os.chdir(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.insert(0, 'py3/lib/python3.10/site-packages'); sys.path.insert(0, 'pkgs') from appPublic.folderUtils import ProgramPath from appPublic.jsonConfig import getConfig from sqlor.dbpools import DBPools from appPublic.log import MyLogger, info, exception from llmage.accounting import get_accounting_llmusages, llm_accounting, llm_accoung_failed p = ProgramPath() config = getConfig(NS={'workdir': os.getcwd(), 'ProgramPath': p}) DBPools(config.databases) # standalone must init manually (sage.py workers get it from webapp()) async def backend_accounting(): info('backend accounting started ...') while True: try: lus = await get_accounting_llmusages() except Exception as e: exception(f'{e}'); lus = [] for lu in lus: try: await llm_accounting(lu) except Exception as e: exception(f'{e}, {lu.id=}'); await llm_accoung_failed(lu.id) await asyncio.sleep(10) def main(): loop = asyncio.new_event_loop(); asyncio.set_event_loop(loop) signal.signal(signal.SIGTERM, lambda s, f: [t.cancel() for t in asyncio.all_tasks(loop)] or loop.stop()) try: loop.run_until_complete(backend_accounting()) except asyncio.CancelledError: pass finally: loop.close() if __name__ == '__main__': main() ``` start.sh order: 1) background programs first: `nohup $PYTHON bin/backend_accounting.py > logs/backend_accounting.log 2>&1 &`, record `name:pid` in `sage_backend.pid`; 2) sage workers (plain pid per line in `sage.pid`). stop.sh: kill workers from sage.pid, then background programs via `while IFS=: read name pid; do kill $pid 2>/dev/null; done < sage_backend.pid`. Pitfalls: - `add_cleanupctx` = async context manager (yield; startup + cleanup); `add_startup` = startup only. Both per-process hooks - PID file: `name:pid` format for background programs (named stop), plain `pid` for workers - Background programs must handle SIGTERM (cancel asyncio tasks) or `kill` leaves DB connections inconsistent - Standalone programs must init `DBPools(config.databases)` manually; use `MyLogger` with a dedicated log file (they don't inherit Sage's log config) ### Sage standalone scripts — DBPools pattern ```python from appPublic.jsonConfig import getConfig # NOT appPublic.getConfig from sqlor.dbpools import DBPools import asyncio config = getConfig('.') db = DBPools(config.databases) dbname = list(config.databases.keys())[0] async def run(): async with db.sqlorContext(dbname) as sor: ... asyncio.run(run()) ``` Inside Sage server context (`.dspy`, module code): use `get_sor_context(env, 'modulename')` instead. ### EventDispatcher Cache Invalidation is Process-Scoped — Use Redis Pub/Sub sqlor C/U/D lifecycle events only fire within the current process. In SO_REUSEPORT deployment, worker A's invalidation never reaches workers B–N → stale cache. **Fix**: Redis Pub/Sub channel `sage:cache:invalidate` — each worker keeps a local cache (fast reads) + subscribes; publish on invalidation, all workers evict; add TTL fallback for missed messages. See `references/multi-process-cache-invalidation.md`. Naming: `{dbname}:{tablename}:{c|u|d}:{before|after}`. Known bug: RBAC uses `this` instead of `self` in userperm.py. ## CRITICAL: uapi.headers Template Rendering — No Helper Functions Needed `uapi.headers` is a JSON **template string**; variables render at runtime before each HTTP request: `{{apikey}}` → decrypted API key from `upappkey` table; `{{jsondata}}` → JSON request body; `{{response}}` → response transformation template. - Vidu-style Token auth: `{"Content-Type": "application/json", "Authorization": "Token {{apikey}}"}` — NO helper function; do NOT add `token()` to `uapi/appapi.py`, the template engine handles `{{apikey}}` natively - OpenAI-compatible: `Bearer {{apikey}}`; complex signing: `{{bearer(apikey)}}` helper ## External API Design Patterns (uapi detail) **Tables**: `upapp` (vendor definition), `uapi` (API endpoints per vendor; `upappid + name` combo unique — contains `path`, `httpmethod`, `headers` JSON template w/ `{{apikey}}`, `data` JSON template w/ params, `response` output transformation), `upappkey` (API credentials `apikey`/`secretkey`, encrypted), `uapiio` (I/O schemas). ``` Client → dapi(Bearer) → your_module/api/xxx.dspy → vendor_id → lookup upappid+apiname in module config table → from uapi.uapi import UpAppApi; resp = await UpAppApi(request).call(upappid, apiname, callerid, params) → uapi gateway renders headers/data/response templates → HTTP → vendor ``` Module config table registers per-vendor mappings: `vendor: "volcengine"`, `upappid: "upapp-volcengine-01"` (refs upapp.id), `apiname_create_session: "CreateVisualValidateSession"`, `apiname_get_result: "GetVisualValidateResult"`, `apiname_create_asset: "CreateAsset"` — one apiname per vendor operation. ```python from uapi.uapi import UpAppApi async def call_vendor_api(vendor_id, action, params): config = await get_vendor_config(vendor_id) upappid = config.upappid apiname = getattr(config, f'apiname_{action}') ua = UpAppApi(request) resp = await ua.call(upappid, apiname, await get_user(), params) return json.loads(resp.decode('utf-8')) # response template may have transformed it ``` | Use uapi gateway | Use direct HTTP client | |---|---| | Standard REST/JSON APIs | Non-HTTP protocols (gRPC, WebSocket to vendor) | | Vendor has OpenAPI/REST interface | Complex HMAC signing not expressible in templates | | Multiple vendors for same op (config-driven, zero code) | One-off API with no reuse pattern | | Need response templating/transformation | Streaming/chunked responses with custom parsing | - **Bearer token auth: dapi auto-resolves identity** — client-facing `.dspy` APIs get `user_id` via `await get_user()` and `org_id` via `await get_userorgid()`. NEVER add `downapp_id`/`client_id`/manual identification params — the Bearer token IS the identifier - **Never expose internal DB IDs to clients** — client API responses/params use only vendor-side identifiers (`vendor_group_id` not `local_group_id`) - **Upload endpoints accept vendor-side IDs** — validate ownership via `rl_org_group(org_id, vendor_group_id)`, use internal `local_group_id` only for local FK relationships - **Vendor callbacks are `paths_any`** (vendor POSTs have no session) — NOT paths_logined; **callback idempotency mandatory** (vendors retry — check existing mapping/status before insert) ## Schema Migration Pattern (xlsx models) When modifying model fields (adding/removing columns): 1. Update the `.xlsx` in `models/` 2. Migration script: (a) check old column exists, (b) create new tables/indexes, (c) migrate data, (d) optionally drop old column — make it idempotent (safe to run multiple times) 3. Copy updated files to `pkgs/` and reinstall: `cd pkgs/module && pip install -e .` 4. Restart Sage after code changes ## pyproject.toml Dependencies ONLY declare `sqlor` and `bricks_for_python`. Do NOT declare `ahserver`, `apppublic`, `appbase`, `rbac` — these are installed by `build.sh`, not pip: ```toml dependencies = ["sqlor", "bricks_for_python"] ``` ## Python Backend: DBPools() Lifecycle, Singleton Fork Safety, sqlorContext() Module Name Three critical rules for all Python backend code (core.py, etc.): **Rule 1: `DBPools()` must be created in function scope, NEVER in `__init__()`** — WRONG: `self.db = DBPools()` in `__init__`; CORRECT: `db = DBPools()` inside each async method that needs DB access. **Rule 2: DBPools is a Singleton (`@SingletonDecorator`)** — in forked child processes (where `.dspy` files run) the inherited parent instance persists; `DBPools(config.databases)` silently discards new args and returns the old instance with empty/stale `databases`. **Rule 3: `sqlorContext()` takes a database KEY (e.g. `'crm_db'`), NOT a module name** — hardcoding `'harnessed_agent'` fails (key doesn't exist in `config.databases`); `env.get_module_dbname()` resolves modules to the actual key. Complete required pattern for ALL database access: ```python from ahserver.serverenv import ServerEnv from appPublic.jsonConfig import getConfig from sqlor.dbpools import DBPools async def my_query(): env = ServerEnv() dbname = env.get_module_dbname('my_module') # resolves DB key dynamically (e.g. 'crm_db') config = getConfig() db = DBPools() # Singleton instance (fork-safe) db.databases = config.databases # MUST force-set: overrides inherited empty dict async with db.sqlorContext(dbname) as sor: rows = await sor.R('table', {...}) ``` **NEVER** hardcode a database name in `sqlorContext()`. **NEVER** pass `config.databases` as a constructor arg to `DBPools()` — silently ignored due to the Singleton. ## harnessed_agent: Tool Permissions, No Mocks, Context Propagation ### Internal Calls Must Not Be Blocked `_get_user_permissions()` in `harnessed_agent/core.py` must NOT restrict permissions for empty/missing context — internal workflow calls pass `context=None`; restricting makes `write_file`, `memory`, `clarify` fail with "Insufficient permissions to execute tool 'X'". Grant full permissions unconditionally: ```python def _get_user_permissions(self, context): # Internal system calls should not be blocked by permission checks return ['file_read', 'file_write', 'system_execute', 'system_manage', 'browser_access', 'ai_vision', 'ai_tts', 'memory_manage', 'memory_read', 'skill_read', 'skill_manage', 'task_manage', 'task_delegate', 'user_interact', 'schedule_manage', 'config_read'] ``` ### Tool Implementation: No Mocks `harnessed_agent/tools/base_tools.py` wrappers MUST execute real operations, not return `status: "mock_implementation"` dicts (reasoning engine reports fake success, LLM hallucinates file locations). Real implementations: read_file/write_file/search_files/patch = actual file I/O; terminal = `asyncio.create_subprocess_shell`; execute_code = temp `.py` + `python3`; memory = `~/.hermes/memory.json`; skill_view/skills_list = `~/.hermes/skills/`; todo = `~/.hermes/todo.json`. Browser (`browser_*`), vision (`vision_analyze`), TTS (`text_to_speech`) need external drivers — may legitimately return structured `note` responses. ### Keep base_tools.py Exports in Sync Tool dicts (`file_tools`, `system_tools`, `skill_tools`, ...) are imported by BOTH `__init__.py` and `registration.py`. After modifying `base_tools.py`, verify all three are consistent — removing/renaming a dict or merging two (e.g. skill_tools into memory_tools) without updating the others → `ImportError`/`KeyError` at registration time. ### Tool Wrappers Must Accept `context` for User Isolation ```python HERMES_DIR = os.path.expanduser("~/.hermes") def _get_user_dir(base_dir, context=None): """User-isolated subdirectory; falls back to global dir if no user context.""" user_id = (context or {}).get('user_id') or (context or {}).get('userid') return os.path.join(base_dir, "users", str(user_id)) if user_id else base_dir async def wrapped_skill_manage(action, name, context=None, **kwargs): user_dir = _get_user_dir(HERMES_DIR, context) skills_dir = os.path.join(user_dir, "skills", name) # ... file operations in user-isolated directory ``` **Context propagation chain** (every step must pass context): 1. `reasoning_console.wss` → `engine.reason_and_execute(user_id=X)` 2. `_execute_tool()` → `harnessed_execute_tool(tool, params, context={user_id: X})` 3. → `agent.execute_tool_call(tool, params, context)` 4. → `_execute_tool_with_retry()` injects `context` via `inspect.signature` — **CRITICAL: check signature first**, else tools without context fail "unexpected keyword argument" 5. Tool wrapper receives `context`, uses `_get_user_dir()` User-isolated structure: `~/.hermes/users/{user_id}/` → `skills/` (SKILL.md per subdir), `memory.json`, `todo.json`, `tmp/` (execute_code temp files). Context-accepting wrappers: `wrapped_skill_manage`/`wrapped_skill_view`/`wrapped_skills_list` → `users/{user_id}/skills/`; `wrapped_memory` → memory.json; `wrapped_todo` → todo.json; `wrapped_execute_code` → tmp/. ### harnessed_execute_tool: Must Accept and Pass `context` ```python # CORRECT — accepts and forwards context: async def harnessed_execute_tool(tool_name, parameters, context=None): return await get_harnessed_agent().execute_tool_call(tool_name, parameters, context) ``` If `context` is omitted, the `user_id` embedded in it is lost and tools execute as `"anonymous"`. ### _get_current_user_id Must NOT Raise Return `"anonymous"` when context is missing, NOT raise ValueError (internal calls/system workflows often lack full context): ```python def _get_current_user_id(self, context): user_id = (context or {}).get('user_id') or (context or {}).get('userid') return str(user_id) if user_id else "anonymous" ``` ### Table Name: hermes_skills (NOT harnessed_skills) Model is `models/hermes_skills.json`; all SQL in core.py must use `hermes_skills`. `harnessed_skills` does NOT exist → "table not found". ### Per-User WebSocket Callbacks (NOT shared ws_push) A shared `ws_push` attribute breaks with concurrent users (they overwrite each other's callbacks): ```python class HermesReasoningEngine: ws_push_callbacks: Dict[str, callable] = {} # per-user callbacks async def _push(self, event_type, data=None, user_id=None): if user_id and user_id in self.ws_push_callbacks: await self.ws_push_callbacks[user_id]({'event': event_type, 'data': data}) ``` In `reason_and_execute()`: set `self._current_user_id = user_id`, pass to all `_push()` calls; cleanup `finally: self._current_user_id = None`. In `.wss` endpoint: `engine.ws_push_callbacks[user_id] = callback`; cleanup `engine.ws_push_callbacks.pop(user_id, None)`. ### Shared Skills Permission: Owner Org Only Shared skills (`~/.hermes/skills/`) readable by ALL users, writable ONLY by owner organization users (`org_id='0'`); non-owner gets `"共享技能仅允许所有者机构用户修改"`. `reason_and_execute()` must set `self._current_org_id` from ServerEnv's `orgid`/`org_id` attribute and include it in the context dict: `context = {"user_id": user_id, "org_id": self._current_org_id, ...}`. ### user_id Must Be in Context for Tool Execution If `user_id` is missing from the context passed to `harnessed_execute_tool`, tools run as `'anonymous'` → permission checks and data isolation failures: ```python context = await self._get_memory_context(user_id, request, config) context['user_id'] = user_id # CRITICAL: must be in context for tool execution # _get_memory_context initializes: {"user_id": user_id, "memory_entries": [], "recent_sessions": [], "skills": []} ``` ## Reasoning Engine: Execution Details ### LLM Call: Do NOT Pass `model` Parameter `harnessed_reasoning/core.py` `_llm_call()` must NOT pass `model=` to `llm_chat()` — let `llm_chat` resolve `default_model` from `harnessed_agent_config`: ```python # WRONG: result = await env.llm_chat(messages=messages, model=model, ...) # CORRECT: result = await env.llm_chat(messages=messages, temperature=temperature, max_tokens=max_tokens, **extra) ``` ### execute_immediately Parameter Parsing Frontend may send `true` (boolean or string), not just `'1'` — support multiple truthy values: ```python execute_val = str(params_kw.get('execute_immediately', '1')).lower() execute_immediately = execute_val in ('1', 'true', 'yes', 'on') ``` ### Module Function Signature Consistency When a `.dspy` passes `user_id` to a module function, ALL functions in the call chain must accept it — else `unexpected keyword argument 'user_id'`: ```python async def hermes_reason_and_execute(request: str, execute_immediately: bool = True, user_id: str = None): return await engine.reason_and_execute(request, execute_immediately=execute_immediately, user_id=user_id) async def reason_and_execute(self, request: str, execute_immediately: bool = True, user_id: str = None): if not user_id: user_id = "anonymous" ``` ### Non-JSON LLM Response Handling LLM API may return HTML error page / proxy block — check content-type before parsing and log the body: ```python if resp.status == 200: content_type = resp.content_type if 'json' not in content_type: err_text = await resp.text() error(f"[llm_response] Non-JSON from {url}, Content-Type={content_type}") error(f"[llm_response] Body (first 2000): {err_text[:2000]}") return {'error': {'message': f'Non-JSON response ({content_type})', 'type': 'content_type_error'}} return await resp.json() ``` ### Store Session: JSON Serialization Safety `json.dumps(plan)` may hit non-serializable types (datetime, custom objects). Clean before serialization (or use `default=str`): ```python def clean_plan(obj): if isinstance(obj, dict): return {k: clean_plan(v) for k, v in obj.items()} if isinstance(obj, list): return [clean_plan(i) for i in obj] if isinstance(obj, datetime): return obj.isoformat() return obj data['execution_plan_json'] = json.dumps(clean_plan(plan), ensure_ascii=False) # or json.dumps(plan, default=str) await sor.C('harnessed_reasoning_sessions', data) ``` ### Database Table Must Exist Before Use `Failed to store session: 'NoneType' object has no attribute 'get'` often means the table doesn't exist — `sqlor.getTableDesc()` returns `None` for missing tables and `C()` crashes on `None['fields']`. Ensure tables are created via `build.sh` before running. ### LLM Config Database Isolation & Debugging `llm_client.py` `_get_llm_config()` reads `harnessed_agent_config`. When called from another module (e.g. `integrated_crm_app` calling `llm_chat()`), lookup can fail (`Failed to fetch LLM config from DB 'default': 'NoneType' object has no attribute 'get'` → hardcoded `model=qwen3-max` fallback, NOT the user's configured value). Root causes: (1) db context — `env.get_module_dbname('harnessed_agent')` throws → falls back to `default`; (2) table missing in queried database; (3) `llm_api_key` stored encrypted → must decrypt via `env.password_decode()`. Correct implementation — try module DB first, then `default`; decrypt api_key; sort `updated_at desc`: ```python async def _get_llm_config(): dbnames_to_try = ['default'] try: env = ServerEnv() module_db = env.get_module_dbname('harnessed_agent') if module_db and module_db not in dbnames_to_try: dbnames_to_try.insert(0, module_db) except Exception as e: error(f"[llm_config] Exception: {e}") for dbname in dbnames_to_try: try: async with DBPools().sqlorContext(dbname) as sor: ns = {'sort': 'updated_at desc'} if user_id: ns['user_id'] = user_id rows = (await sor.R('harnessed_agent_config', ns)) or [] if rows: row = rows[0] if row.get('llm_api_key'): row['llm_api_key'] = ServerEnv().password_decode(row['llm_api_key']) return row except Exception as e: error(f"Failed to fetch LLM config from DB '{dbname}': {e}") return {} ``` ### Encrypted API Key Decryption Fields stored encrypted (like `llm_api_key`, `api_key`) must be decrypted before use: ```python api_key = row.get('llm_api_key', '') if api_key: api_key = ServerEnv().password_decode(api_key) ``` ## CRITICAL: Reuse Existing RBAC Login — Do NOT Write Your Own ALWAYS use the existing RBAC user login system. NEVER create `up_login.dspy`/`login.ui` in `sage/wwwroot/` or any module's `wwwroot/`: - Login page: `/rbac/user/login.ui`; login handler: `/rbac/user/up_login.dspy` - RBAC already handles: RC4 password encryption with `config.password_key`, account lockout detection, session management via `remember_user()`, redirect to userinfo, multiple login methods (password, SMS, WeChat) - All modules authenticate via this shared session; custom login in `sage/wwwroot/` breaks the RBAC auth flow → session inconsistencies ## Password Handling: Use ServerEnv, NOT rf (RegisterFunction) ```python # CORRECT — ServerEnv's password_encode from ahserver.globalEnv (reads key from config.password_key, RC4): from ahserver.globalEnv import password_encode encrypted_pw = password_encode(params_kw.password) # WRONG — legacy RF pattern, do NOT use: await rfexe('password', params_kw) # rf.register('password', ...) is deprecated; may not be registered ``` Decrypt with `ServerEnv().password_decode(value)`. ## User ID Retrieval - In `.dspy`: `userid = await get_user()` (returns user ID string); org: `await get_userorgid()` ## Sage Authentication & Cookie Pitfalls ### Cookie Secure Flag for HTTP Development `ahserver/auth_api.py`'s `EncryptedCookieStorage` defaults to `secure=True` → blocks cookies on HTTP (localhost). Check the SSL value, not just key existence: ```python ssl_enabled = False if hasattr(self.conf.website, 'ssl') and self.conf.website.ssl: # hasattr is True even when value is None! ssl_cfg = self.conf.website.ssl if hasattr(ssl_cfg, 'crtfile') and hasattr(ssl_cfg, 'keyfile'): ssl_enabled = True storage = EncryptedCookieStorage(secret, secure=ssl_enabled, # False for HTTP, True for HTTPS samesite='Lax', httponly=True, max_age=24*60*60) ``` ### Login Form Field Names & Users Table - Form fields: `username` and `passwd` (NOT `loginid`/`password`); `passwd` encrypted via `password_encode()` - Table `users` (NOT `user`); login matches on `username` (e.g. `superuser`), not `id`; `id` is the user ID (e.g. `user-01`) used for RBAC/session - Schema: `id VARCHAR(32) PK`, `username VARCHAR(100)`, `password VARCHAR(255)`, `orgid VARCHAR(32)`, `user_status VARCHAR(1)` ('0'=active), `login_fail_count INT`, `last_login_fail DATETIME` ### RBAC Permission Roles | Role | Description | |------|-------------| | `anonymous` | Unauthenticated users | | `any` | All users (including anonymous) | | `logined` | Authenticated users only | | `owner.*` | Owner organization roles | Feature pages requiring login → `logined`; public pages (login, registration) → `any`/`anonymous`. ## Module Navigation & Adding Features ### menu.ui Pattern Sage modules are navigated via `menu.ui` files, not standalone `index.ui` pages. Main Sage `wwwroot/menu.ui` references module submenus: ```json {"name": "llmage", "label": "模型管理", "submenu": "{{entire_url('/llmage/menu.ui')}}"} ``` Module `menu.ui` (Menu widget; `"url"` = direct nav to page or CRUD alias, `"submenu"` = nested menu.ui, `"items"` = inline submenu, all URLs via `{{entire_url()}}`): ```json {"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"width": "60%", "height": "75%"}, "items": [ {"name": "feature1", "label": "功能1", "url": "{{entire_url('/module/feature.ui')}}"}, {"name": "feature2", "label": "功能2", "url": "{{entire_url('/module/alias_name')}}"} ]}} ``` Menu with auth check (JSON validation fails on `.ui` files with Jinja2 templates — expected/normal): ```json {"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"archor": "cc", "width": "70%", "height": "70%"}, "cwidth": 10, "items": [ {% if get_user() %}{"name": "entry_name", "label": "入口名", "url": "{{entire_url('page.ui')}}"}{% endif %} ]}} ``` ### Adding New Module Features 1. Create the `.ui` file in module's `wwwroot/` 2. **Symlink to Sage wwwroot** (critical for local development): `cd ~/repos/sage/wwwroot/module_name && ln -sf ~/repos/module_name/wwwroot/new_feature.ui .` 3. Add menu entry in `module_name/wwwroot/menu.ui` 4. Register RBAC: `./py3/bin/python set_role_perm.py logined /module_name/new_feature.ui` 5. For API `.dspy` files: symlink into `module_name/api/` and register `/module_name/api/new_api.dspy` ## WebSocket Real-Time Event Push (Reasoning Console Pattern) ### WSS URL paths See "WSS WebSocket URL Routing" under URL & Path Rules — `/wss/` prefix mandatory everywhere (UI `{{entire_url}}`, JS URL, RBAC registration, `set_role_perm.py` arg). Verify via server log `[debug] userid=None, path='/wss/...' permission check failed` — use that path verbatim. ### UI Layout (two approaches) **A: WebSocket widget + Html handlers** (framework-managed): `{"widgettype": "WebSocket", "id": "...", "options": {"ws_url": "{{entire_url('/wss/module/endpoint.wss')}}", "with_session": true}, "binds": [{"wid": "self", "event": "onopen", "actiontype": "script", "script": "onWsOpen()"}, {"wid": "self", "event": "ontext", "actiontype": "script", "script": "onWsMessage(event.params)"}]}` plus an Html widget defining `window.onWsOpen`/`onWsMessage` — fragile ordering (see rule 3). **B: Pure Html widget + raw JS WebSocket (recommended, full control)** — single Html widget with inline `