From 93079c1a544e96b6d060154b29e0e7224285f6e2 Mon Sep 17 00:00:00 2001 From: yumoqing Date: Fri, 21 Aug 2026 12:24:26 +0800 Subject: [PATCH] =?UTF-8?q?refactor(skills):=20=E7=B2=BE=E7=82=BC=207=20?= =?UTF-8?q?=E4=B8=AA=E5=BC=80=E5=8F=91=E6=8A=80=E8=83=BD=E6=96=87=E6=9C=AC?= =?UTF-8?q?=EF=BC=88=E5=8E=8B=E7=BC=A9=20WRONG/CORRECT=20=E5=8F=8C?= =?UTF-8?q?=E4=BB=A3=E7=A0=81=E5=9D=97=E4=B8=BA=E8=A7=84=E5=88=99+?= =?UTF-8?q?=E8=AD=A6=E7=A4=BA=EF=BC=8C=E8=A7=84=E5=88=99100%=E4=BF=9D?= =?UTF-8?q?=E7=95=99=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit module-development-spec 99KB→57KB、harnessed-module-development 101KB→62KB、 crud-definition-spec 78KB→46KB、dspy-file-implementation-spec 51KB→26KB、 ahserver 50KB→26KB、sqlor-database-module 49KB→26KB、database-table-definition-spec 20KB→19KB 合计 448KB→263KB，压缩 41%，节省约 92k token --- skills_library/all/ahserver/SKILL.md | 1285 ++-------- .../all/crud-definition-spec/SKILL.md | 1529 ++--------- .../database-table-definition-spec/SKILL.md | 2 - .../dspy-file-implementation-spec/SKILL.md | 1055 ++------ .../all/harnessed-module-development/SKILL.md | 2261 ++++------------- .../all/module-development-spec/SKILL.md | 1763 ++----------- .../all/sqlor-database-module/SKILL.md | 1017 ++------ 7 files changed, 1535 insertions(+), 7377 deletions(-) diff --git a/skills_library/all/ahserver/SKILL.md b/skills_library/all/ahserver/SKILL.md index f6dd9c5..393d8ae 100644 --- a/skills_library/all/ahserver/SKILL.md +++ b/skills_library/all/ahserver/SKILL.md @@ -6,7 +6,7 @@ author: "yu moqing" tags: ["aiohttp", "web-framework", "async", "python", "http-server"] dependencies: - uvloop - - httptools + - httptools - redis - asyncio - aiofiles @@ -30,208 +30,61 @@ dependencies: # ahserver Web Application Framework ## Overview -ahserver is an asynchronous HTTP(S) server built on top of the aiohttp framework. It provides a comprehensive set of features for building modern web applications including: - -- User authorization and authentication support -- HTTPS support with SSL/TLS configuration -- Multiple database connection pools (MySQL, PostgreSQL, Oracle, SQL Server) -- Built-in processors for various file types (.dspy, .tmpl, .md, .xlsxds, .sqlds) -- Internationalization (i18n) support -- File upload handling with automatic storage -- Background task support -- RESTful API capabilities +Async HTTP(S) server built on aiohttp. Features: auth/authorization, HTTPS (SSL/TLS), multi-DB connection pools (MySQL/PostgreSQL/Oracle/SQL Server), processors for `.dspy/.tmpl/.md/.xlsxds/.sqlds/.wss`, i18n, auto-stored file uploads, background tasks, RESTful APIs. ## Installation - -### Prerequisites -- Python 3.8+ -- Required dependencies from setup.cfg - -### Install from source -```bash -cd /tmp/ahserver -pip install -e . -``` - -### Python 3.12+ Compatibility Fix -For Python 3.12+, aioredis requires a compatibility fix: - -```bash -pip install packaging -``` - -Then modify aioredis files: -- **aioredis/connection.py line 11**: Replace with `from packaging.version import Version as StrictVersion` -- **aioredis/exceptions.py line 14**: Replace with `class TimeoutError(asyncio.TimeoutError, RedisError):` +- Python 3.8+. From source: `cd /tmp/ahserver && pip install -e .` +- **Python 3.12+ aioredis fix**: `pip install packaging`, then edit `aioredis/connection.py:11` → `from packaging.version import Version as StrictVersion`; `aioredis/exceptions.py:14` → `class TimeoutError(asyncio.TimeoutError, RedisError):` ## Project Structure ``` your-app/ -├── ah.py # Main application entry point -├── conf/ -│ └── config.json # Configuration file -├── i18n/ # Internationalization files -└── app/ # Your application files +├── ah.py # entry point +├── conf/config.json +├── i18n/ +└── app/ ├── index.html - ├── api/ # API endpoints (directory pattern) - │ ├── status/ - │ │ └── index.dspy # → /api/status - │ └── submit/ - │ └── index.dspy # → /api/submit - ├── template.tmpl # Jinja2 templates - └── data.sqlds # SQL data sources + ├── api/status/index.dspy # → /api/status (directory pattern, 1.2.0+) + ├── template.tmpl # Jinja2 + └── data.sqlds ``` - -⚠️ **1.2.0+ requires directory/index.dspy pattern for APIs** — flat `api/status.dspy` files no longer resolve to `/api/status`. See `references/api-routing-patterns.md`. +⚠️ **1.2.0+ requires `directory/index.dspy` for APIs** — flat `api/status.dspy` no longer resolves to `/api/status`. See `references/api-routing-patterns.md`. ## Configuration (conf/config.json) +Top-level keys: `password_key`, `databases`, `website`, `langMapping`. +- `website`: `paths: [["$[workdir]$/app", ""]]`, `host: "0.0.0.0"`, `port: 8080`, `coding: "utf-8"`, optional `ssl: {crtfile, keyfile}`, `indexes: ["index.html","index.ui","index.tmpl","index.dspy","index.md"]`, `processors: [[".xlsxds","xlsxds"],[".sqlds","sqlds"],[".tmpl","tmpl"],[".dspy","dspy"],[".md","md"]]`. +- `langMapping`: `{"zh-Hans-CN": "zh-cn", "en-US": "en"}`. -### Basic Configuration Template -```json -{ - "password_key": "YOUR_24_CHAR_AES_KEY_HERE", - "databases": { - "your_db": { - "driver": "aiomysql", - "async_mode": true, - "coding": "utf8", - "dbname": "your_database", - "kwargs": { - "user": "username", - "db": "your_database", - "password": "encrypted_password", - "host": "localhost" - } - } - }, - "website": { - "paths": [ - ["$[workdir]$/app", ""] - ], - "host": "0.0.0.0", - "port": 8080, - "coding": "utf-8", - "ssl": { - "crtfile": "$[workdir]$/conf/cert.pem", - "keyfile": "$[workdir]$/conf/key.pem" - }, - "indexes": [ - "index.html", - "index.ui", - "index.tmpl", - "index.dspy", - "index.md" - ], - "processors": [ - [".xlsxds", "xlsxds"], - [".sqlds", "sqlds"], - [".tmpl", "tmpl"], - [".dspy", "dspy"], - [".md", "md"] - ] - }, - "langMapping": { - "zh-Hans-CN": "zh-cn", - "en-US": "en" - } -} -``` +⚠️ **`password_key` must be a TOP-LEVEL field** (same level as `databases`). sqlor reads it via `getConfig().password_key` to AES-decrypt DB passwords. Without it, `self.unpassword()` returns None and connection fails. -### Database Configuration Examples +### Database entries (all use `coding: "utf8"` + `dbname`; kwargs differ by driver) +| Driver | kwargs | +|--------|--------| +| MySQL/MariaDB: `mysql.connector` | user, db, password, host | +| PostgreSQL: `psycopg2` | database, user, password, host, port | +| Oracle: `cx_Oracle` | user, host, dsn (e.g. `10.0.185.137:1521/SAMPLEDB`) | +| SQL Server: `pymssql` | user, database, password, server, port (1433), charset (`utf8`) | -⚠️ **`password_key`** must be a top-level config field (same level as `databases`). sqlor reads it via `getConfig().password_key` to AES-decrypt database passwords. Without it, `self.unpassword()` returns None and connection fails. - -#### MySQL/MariaDB -```json -"mysql_db": { - "driver": "mysql.connector", - "coding": "utf8", - "dbname": "sampledb", - "kwargs": { - "user": "user1", - "db": "sampledb", - "password": "***", - "host": "localhost" - } -} -``` - -#### PostgreSQL -```json -"postgres_db": { - "driver": "psycopg2", - "dbname": "testdb", - "coding": "utf8", - "kwargs": { - "database": "testdb", - "user": "postgres", - "password": "***", - "host": "127.0.0.1", - "port": "5432" - } -} -``` - -#### Oracle -```json -"oracle_db": { - "driver": "cx_Oracle", - "coding": "utf8", - "dbname": "sampledb", - "kwargs": { - "user": "user1", - "host": "localhost", - "dsn": "10.0.185.137:1521/SAMPLEDB" - } -} -``` - -#### SQL Server -```json -"mssql_db": { - "driver": "pymssql", - "coding": "utf8", - "dbname": "sampledb", - "kwargs": { - "user": "user1", - "database": "sampledb", - "password": "***", - "server": "localhost", - "port": 1433, - "charset": "utf8" - } -} -``` +Passwords are AES-ECB base64 (see Password Encryption below). ## Usage Examples -### Basic Application Setup (ah.py) - -**Startup hook pattern** — version-dependent API: - +### Basic Application Setup (ah.py) — startup hooks | Version | API | Import | |---------|-----|--------| -| ≤1.0.x | `RegisterCoroutine().register('ahapp_built', callback)` | `from ahserver.configuredServer import RegisterCoroutine` | -| ≥1.2.0 | `add_startup(callback)` | `from ahserver.configuredServer import add_startup` | +| ≤1.0.x | `RegisterCoroutine().register('ahapp_built', cb)` | `ahserver.configuredServer` | +| ≥1.2.0 | `add_startup(cb)` | `ahserver.configuredServer` | -⚠️ **`RegisterCoroutine` was REMOVED in 1.2.0.** `add_startup` is the replacement. The callback signature changed: old receives `(app)`, new receives `(*args, **kw)`. +⚠️ **`RegisterCoroutine` REMOVED in 1.2.0** → use `add_startup`. Callback signature changed: old `(app)` → new `(*args, **kw)`. -**ahserver ≥1.2.0 (recommended):** ```python -import asyncio from ahserver.webapp import webapp from ahserver.serverenv import ServerEnv from ahserver.configuredServer import add_startup -async def on_app_built(*args, **kw): - """Called after auth middleware and processors are set up.""" +async def on_app_built(*args, **kw): # runs after auth middleware + processors are set up asyncio.ensure_future(my_background_task()) -async def my_background_task(): - while True: - await asyncio.sleep(10) - print('Background task running...') - def init(): env = ServerEnv() env.get_module_dbname = lambda m: 'your_db_name' @@ -241,256 +94,70 @@ if __name__ == '__main__': webapp(init) ``` -**ahserver ≤1.0.x (legacy):** -```python -from ahserver.configuredServer import RegisterCoroutine - -def init(): - rc = RegisterCoroutine() - rc.register('ahapp_built', on_app_built) -``` - -### Authentication API Implementation +### Authentication API ```python from ahserver.auth_api import AuthAPI - class MyAuthAPI(AuthAPI): - def needAuth(self, path): - # Return True if path requires authentication - return path.startswith('/admin') - - async def getPermissionNeed(self, path): - if path.startswith('/admin'): - return 'admin' - return 'user' - - async def checkUserPassword(self, user_id, password): - # Implement your authentication logic - return user_id == 'admin' and password == 'secret' - - async def getUserPermissions(self, user): - # Return user permissions - if user == 'admin': - return ['admin', 'user'] - return ['user'] - -if __name__ == '__main__': - server = ConfiguredServer(MyAuthAPI) - server.run() + def needAuth(self, path): ... # True → require auth + async def getPermissionNeed(self, path): ... # permission name for path + async def checkUserPassword(self, user_id, password): ... + async def getUserPermissions(self, user): ... # list of permission names +# run: ConfiguredServer(MyAuthAPI).run() ``` ## Processor Types -### .dspy Files (Dynamic Python Scripts) -Execute Python code dynamically with full access to ahserver environment. +### .dspy (Dynamic Python Scripts) +**Execution model:** code runs directly in script scope; the framework captures the script's TOP-LEVEL `return` as the response. -**Execution model:** Code runs directly in script scope — no wrapper function. Use `return` to send the response. Do NOT wrap code in `async def run()` — the script's top-level `return` is what the framework captures. - -⚠️ **`result` variable alone does NOT work.** The framework captures the script's return value, not a variable. Always use `return json.dumps(...)` at the top level: +⚠️ Setting a `result` variable alone does NOT work (framework sees NoneType). Do NOT wrap code in `async def run()` — its return goes nowhere at script scope. Always return at top level: ```python -# ✅ CORRECT return json.dumps({'status': 'success', 'data': result}, ensure_ascii=False) - -# ❌ WRONG — framework sees NoneType -result = json.dumps({'status': 'success'}) # No return! - -# ❌ WRONG — async def run() wrapper prevents top-level return -async def run(): - result = json.dumps({'status': 'success'}) - return result # This return goes nowhere at script scope ``` -**Example (api/user.dspy):** -```python -# Get user from session -user_id = await get_user() -if user_id is None: - await redirect('/login') - -# Database CRUD operations -db = DBPools() -async with db.sqlorContext('your_db') as sor: - # Create - await sor.C('users', {'id': uuid(), 'name': 'John'}) - - # Read - users = await sor.R('users', {'name': 'John'}) - - # Update - await sor.U('users', {'id': user_id, 'name': 'Jane'}) - - # Delete - await sor.D('users', {'id': user_id}) - - # Raw SQL - results = await sor.sqlExe("SELECT * FROM users WHERE id=${id}$", {'id': user_id}) - - # Paged query - paged_results = await sor.sqlPaging( - "SELECT * FROM users WHERE name LIKE ${search}$", - {'search': '%john%', 'page': 1, 'pagerows': 20, 'sort': 'name'} - ) - -return results -``` - -### File Upload Handling - -**Multipart uploads are auto-handled by ahserver.** Files are saved by `FileStorage` before your .dspy runs. `params_kw` contains the **relative web_path**, not the file object. +In-scope helpers: `await get_user()`, `await redirect('/login')`, `uuid()`, `DBPools()`; CRUD via `async with db.sqlorContext('dbname') as sor:` → `await sor.C('tbl', ns)`, `sor.R('tbl', ns)`, `sor.U('tbl', ns)`, `sor.D('tbl', {'id':...})`, `sor.sqlExe("SELECT ... WHERE id=${id}$", {'id':...})` (placeholders `${name}$`), `sor.sqlPaging(sql, {'search':..., 'page':1, 'pagerows':20, 'sort':'name'})`. +### File Upload +Multipart uploads auto-handled: files are saved by `FileStorage` BEFORE the .dspy runs; `params_kw` contains the **relative web_path**, not the file object. ```python from ahserver.filestorage import FileStorage - -web_path = params_kw.get('audio_file') # e.g., '/66/34/59/64/file.mp3' -fs = FileStorage() -absolute_path = fs.realPath(web_path) # e.g., '/tmp/66/34/59/64/file.mp3' +abs_path = FileStorage().realPath(params_kw.get('audio_file')) # '/66/34/59/64/file.mp3' → '/tmp/66/34/59/64/file.mp3' ``` +See `references/file-upload-patterns.md` (examples, curl testing, debugging). -See `references/file-upload-patterns.md` for complete examples, curl testing, and debugging tips. +### .tmpl / .sqlds / .xlsxds / .md +- `.tmpl`: Jinja2 template rendered with request context (see Jinja2 filter pitfall below). +- `.sqlds`: SQL query exposed as data source; `.xlsxds`: Excel as structured data; `.md`: rendered as HTML. -### .tmpl Files (Jinja2 Templates) -Render HTML templates with dynamic data. - -**Example (template/user.tmpl):** -```html - - -User Profile - -

Welcome {{ username }}!

-

User ID: {{ user_id }}

- {% if permissions %} -

Permissions: {{ permissions|join(', ') }}

- {% endif %} - - -``` - -### .sqlds Files (SQL Data Sources) -Define SQL queries that can be executed as data sources. - -**Example (data/users.sqlds):** -```sql -SELECT - id, - name, - email, - created_at -FROM users -WHERE active = 1 -ORDER BY created_at DESC -``` - -### .xlsxds Files (Excel Data Sources) -Serve Excel files as structured data sources. - -### .md Files (Markdown) -Render Markdown files as HTML. - -### .wss Files (WebSocket Endpoints) -Python scripts that handle WebSocket connections. Must contain `async def myfunc(request, **kwargs)`. - -**Framework behavior** (`websocketProcessor.py`): -- Reads the client's `Sec-WebSocket-Protocol` header as cookie for user authentication -- Calls `get_user()` to authenticate the user -- Injects `ws_pool` (WsPool instance) and `ws_data` (client message text) into kwargs -- Executes `myfunc` on each incoming TEXT message -- `ws_pool.sendto(data, id=None)` pushes JSON messages to the client - -**Example (endpoint.wss):** +### .wss (WebSocket Endpoints) +Must contain `async def myfunc(request, **kwargs)`. Behavior (`websocketProcessor.py`): reads client `Sec-WebSocket-Protocol` header as cookie → `get_user()` auth; injects `ws_pool` (WsPool) and `ws_data` (message text) into kwargs; runs `myfunc` per incoming TEXT message; `ws_pool.sendto(data, id=None)` pushes JSON. ```python -import json async def myfunc(request, **kwargs): - ws_pool = kwargs.get('ws_pool') - ws_data = kwargs.get('ws_data') - data = json.loads(ws_data) if ws_data else {} - - if data.get('cmd') == 'connect': - await ws_pool.sendto(json.dumps({'type': 'connected', 'message': 'OK'})) - elif data.get('cmd') == 'ping': - await ws_pool.sendto(json.dumps({'type': 'pong'})) -``` - -**Frontend JS must pass cookie for authentication:** -```js -new WebSocket(url, document.cookie); // correct -new WebSocket(url); // WRONG - auth fails + data = json.loads(kwargs.get('ws_data') or '{}') + if data.get('cmd') == 'ping': + await kwargs['ws_pool'].sendto(json.dumps({'type': 'pong'})) ``` +⚠️ Frontend must pass the cookie: `new WebSocket(url, document.cookie)` — without it auth fails. ## Global Environment Variables +- **Session (async):** `get_user()`, `remember_user(userid, username='', userorgid='')`, `forget_user()`, `redirect(url)`, `entire_url(url)` (full URL with scheme/host/port), `gethost()` (client IP), `path_call(path, **kw)` (call other server resources). +- **Globals:** `configValue(k)` (e.g. `configValue('.website.port')`), `DBPools()` (sqlor pools), `uuid()`, `curDatetime()`, `str2date(dstr)`, `str2datetime(dstr)`, `server_error(errcode)` (400/401/403/404/500...). +- **Built-in modules:** `time`, `datetime`, `random`, `json`, `ArgsConvert`, `DictObject`. -### Session Functions -- `get_user()`: Get current user ID (async) -- `remember_user(userid, username='', userorgid='')`: Set session user info (async) -- `forget_user()`: Clear session user info (async) -- `redirect(url)`: Redirect to URL (async) -- `entire_url(url)`: Convert to full URL with scheme/host/port -- `gethost()`: Get client IP address -- `path_call(path, **kw)`: Call other server resources (async) - -### Global Functions -- `configValue(k)`: Get configuration value (e.g., `configValue('.website.port')`) -- `DBPools()`: Get database connection pool (uses sqlor framework) -- `uuid()`: Generate UUID -- `curDatetime()`: Get current datetime -- `str2date(dstr)`, `str2datetime(dstr)`: Parse date strings -- `server_error(errcode)`: Raise HTTP error (400, 401, 403, 404, 500, etc.) - -### Built-in Modules -- `time`, `datetime`, `random`, `json` -- `ArgsConvert`, `DictObject` - -## CRUD Operations with SQLor - -All CRUD operations require a table with an `id` field as primary key. - -### Create (Insert) +## CRUD with SQLor +All CRUD requires a table with `id` primary key. Pattern: ```python db = DBPools() async with db.sqlorContext('dbname') as sor: - ns = {'id': uuid(), 'field1': 'value1'} - recs = await sor.C('table_name', ns) -``` - -### Read (Select) -```python -ns = params_kw.copy() # Get parameters from client -db = DBPools() -async with db.sqlorContext('dbname') as sor: - recs = await sor.R('table_name', ns) -``` - -### Update -```python -ns = params_kw.copy() -db = DBPools() -async with db.sqlorContext('dbname') as sor: - await sor.U('table_name', ns) -``` - -### Delete -```python -ns = {'id': params_kw.id} -db = DBPools() -async with db.sqlorContext('dbname') as sor: - await sor.D('table_name', ns) -``` - -### Paged Read -```python -ns = params_kw.copy() -ns.setdefault('page', 1) -ns.setdefault('sort', 'id desc') -db = DBPools() -async with db.sqlorContext('dbname') as sor: - recs = await sor.RP('table_name', ns) - # Returns: {"total": total_records, "rows": data_list} + await sor.C('tbl', {'id': uuid(), 'field1': 'value1'}) # create + recs = await sor.R('tbl', params_kw.copy()) # read (client params) + await sor.U('tbl', params_kw.copy()) # update + await sor.D('tbl', {'id': params_kw.id}) # delete + recs = await sor.RP('tbl', ns) # paged → {'total': n, 'rows': [...]}; ns: page/sort/pagerows ``` ## Running Behind Nginx - -When running behind nginx, configure nginx to forward these headers: - +Forward these headers: ``` proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Scheme $scheme; @@ -499,138 +166,26 @@ proxy_set_header X-Forwarded-Url $request_uri; proxy_set_header X-Forwarded-Prepath ""; ``` -## Performance Diagnostics and Optimization +## Performance +Middleware chain per request: `real_ip_middleware → session_middleware → auth_api.checkAuth() → ProcessorResource._handle() → handler`. -See `references/performance-benchmark-may2026.md` for complete benchmark results and applied fixes. -See `references/performance-diagnostics.md` for detailed middleware chain breakdown. +Per-request overhead (local SSD): Redis session GET 0.06ms (only if `session_redis`); `info()` log flush 0.01ms (production NFS/cloud: **10–100ms**); `_handle()` closures + isHtml 0.22ms; network + file I/O 1.23ms. ⚠️ Production latency can be 10–50x higher (NFS/cloud disk flush, I/O contention, multiple log calls per request). -### Middleware Chain Breakdown +- **Log flush (P0+P3 FIXED, commits `574ef00`/`5238a08`, `appPublic/log.py`):** file opened once + kept open; `threading.Queue(maxsize=10000)` + daemon thread for non-blocking writes; only `exception`/`critical` flush immediately, others every ~1s idle; queue-full drops oldest instead of blocking the loop; **public API (`info()` etc.) unchanged**. ⚠️ On local SSD flush ≈ 0.01ms — if prod disk is local SSD, log flush is NOT the bottleneck. +- **isHtml() (FIXED, `processorResource.py:398`):** now reads only first 512 bytes (`f.read(512)` + `decode('utf-8', errors='ignore')`) instead of the whole file (was reading e.g. 48KB bricks.js / 1MB echarts.min.js). +- **Static file fast path (APPLIED, `574ef00`):** at very top of `ProcessorResource._handle()` (before parse_request/closures/i18n/isHtml): if `request.path.lower()` ends with a static ext (`.js .css .png .jpg .jpeg .gif .ico .svg .woff .woff2 .ttf .eot .map .webp .bmp .mp3 .mp4 .webm .ogg .wav`) → `self.parse_request(request)` then `return await super()._handle(request)`. ⚠️ `parse_request` MUST run first — `super()` needs `self._preurl` set by it. +- **Session on static files:** with `session_redis`, every request incl. static triggers a Redis GET. Consider `EncryptedCookieStorage` instead of `RedisStorage` for static-heavy apps, or bypass session for known static paths. -Every request passes through this middleware chain (in order): +### timecost log format (`auth_api.py`) +`timecost=client(IP) user_id access /path cost TOTAL, (AUTH_MS)` — TOTAL = full handler execution; AUTH_MS = auth/permission check only. **Diagnostic rule:** AUTH_MS small (1–3ms) but total slow → bottleneck is in the handler / `.dspy`, NOT RBAC. -``` -real_ip_middleware → session_middleware → auth_api.checkAuth() → ProcessorResource._handle() → handler -``` - -**Per-request overhead breakdown (local SSD):** - -| Layer | Avg Latency | Notes | -|-------|------------|-------| -| Redis session GET | 0.06ms | Only if `session_redis` is configured | -| `info()` log flush | 0.01ms | Local SSD; production NFS/cloud: **10-100ms** | -| `_handle()` closure + isHtml | 0.22ms | 20+ closures built per request | -| Network + file I/O | 1.23ms | aiohttp FileResponse baseline | - -**⚠️ Production latency can be 10-50x higher** due to: -- NFS/cloud disk flush latency (10-200ms per `info()` call) -- Concurrent requests competing for disk I/O -- Multiple log calls per request (`info` + `debug` + `exception`) - -### Bottleneck #1: Synchronous Log Flush (CRITICAL — PARTIALLY FIXED) - -**File:** `appPublic/log.py` - -**Status (P0 fix applied):** The `info()` call at `auth_api.py:166` was removed (commit `574ef00`). This eliminates the mandatory per-request log flush. - -**Status (P3 fix applied):** `appPublic/log.py` was rewritten with async queue + persistent file handle (commit `5238a08`): -- File opened once, kept open during runtime -- `threading.Queue(maxsize=10000)` + background daemon thread for non-blocking writes -- Only `exception`/`critical` trigger immediate flush; others flushed periodically (every 1s idle) -- Queue-full protection: drops oldest entry instead of blocking the event loop -- **All public API (`info()`, `debug()`, etc.) remains identical** — zero interface change - -**Original problem:** Each call to `info()`, `debug()`, `warning()`, etc. performed: -```python -def log(self, levelname, message, frame_info): - self.open_logger() # codecs.open(logfile, 'a', 'utf-8') ← SYNC - self.logger.write(s) # ← SYNC - self.logger.flush() # ← BLOCKING flush to disk - self.close_logger() # codecs.close() ← SYNC -``` - -**⚠️ Important finding:** On local SSD, each log flush is ~0.01ms. If production disk is also local SSD (not NFS), log flush is **not** the 17-second bottleneck. The real cause may be elsewhere (network proxy, DNS, connection pool exhaustion, GIL contention, or something outside ahserver). - -### Benchmark methodology - -### Bottleneck #2: isHtml() Reads Entire File - -**File:** `processorResource.py:398` - -The `isHtml()` method reads the **entire file content** to check if it starts with ``: - -```python -# BAD — reads entire file (e.g. 48KB bricks.js or 1MB echarts.min.js) -async with aiofiles.open(fn,'r',encoding='utf-8') as f: - b = await f.read() - while b[0] in ['\n',' ','\t']: - b = b[1:] - if b.lower().startswith(''): - return True -``` - -**Fix (already applied in local repo):** Only read first 512 bytes: -```python -# GOOD — reads only header -async with aiofiles.open(fn,'rb') as f: - b = await f.read(512) - b = b.decode('utf-8', errors='ignore') -``` - -### Bottleneck #3: Static File Fast Path (APPLIED) - -**Status:** Committed to `ahserver` main (commit `574ef00`). - -Added at the **very top** of `ProcessorResource._handle()` (before `parse_request` and all closures): -```python -async def _handle(self,request:Request) -> StreamResponse: - # Fast path for static assets: skip auth closures, i18n, url2processor, isHtml - static_exts = ('.js', '.css', '.png', '.jpg', '.jpeg', '.gif', '.ico', - '.svg', '.woff', '.woff2', '.ttf', '.eot', '.map', - '.webp', '.bmp', '.mp3', '.mp4', '.webm', '.ogg', '.wav') - path_lower = request.path.lower() - if any(path_lower.endswith(ext) for ext in static_exts): - self.parse_request(request) - return await super()._handle(request) - # ... rest of _handle() unchanged (closures, i18n, url2processor, etc.) -``` - -**Key detail:** `parse_request(request)` must be called before `super()._handle(request)` because `super()` needs `self._preurl` which is set by `parse_request`. - -### Bottleneck #4: Session Loading for Static Files - -When `session_redis` is configured, every request (including static files) triggers a Redis GET: -```python -data_bytes = await self._redis.get(self.cookie_name + "_" + key) -``` - -For anonymous/static requests this is unnecessary overhead. Consider: -- Using `EncryptedCookieStorage` instead of `RedisStorage` for static-heavy apps -- Adding a session bypass for known static paths in middleware - -### auth_api.py timecost Log Format - -Every request logs a `timecost` line in `auth_api.py`: - -``` -timecost=client(IP) user_id access /path cost TOTAL, (AUTH_MS) -``` - -- `TOTAL` (before comma): total handler execution time in seconds -- `AUTH_MS` (in parentheses): **only the auth/permission check** time - -**Diagnostic rule:** If `AUTH_MS` is small (1-3ms) but the total response is slow, the bottleneck is **NOT** in RBAC/auth — it's in the handler itself. Check `processorResource.py` or the specific `.dspy` handler. - -### Unauthenticated static files (3parties, css, js) - -Static files under `/bricks/3parties/`, `/bricks/css/`, `/bricks/*.js` require `any` role permission. If `any` role doesn't have these paths in `load_path.py`, all unauthenticated requests return `need login` (not 403 — it redirects to login). Check `rbac/check_perm.py` logs: `userid=None, path='...' permission check failed,userroles=['anonymous', 'any']`. +### Unauthenticated static files +`/bricks/3parties/`, `/bricks/css/`, `/bricks/*.js` require `any` role permission in `load_path.py`; otherwise unauthenticated requests return `need login` (redirect, not 403). Check `rbac/check_perm.py` logs: `userid=None, path='...' permission check failed,userroles=['anonymous', 'any']`. ## Pitfalls -### ⚠️ CRITICAL: StreamResponse breaks aiohttp_auth ticket reissue — users kicked out while active - -**Problem:** `aiohttp_auth`'s `process_response()` checks `isinstance(response, web.Response)` before calling `remember_ticket()`. ahserver returns `StreamResponse`, so the check always fails and tickets are **never renewed**. Users get logged out after `session_max_time` (default 2h) regardless of activity. - -**Fix:** In `auth_api.py` `checkAuth` middleware, after `ret = await handler(request)`, manually reissue: +### ⚠️ CRITICAL: StreamResponse breaks aiohttp_auth ticket reissue — active users get kicked out +`aiohttp_auth.process_response()` renews tickets only for `web.Response` instances; ahserver returns `StreamResponse`, so the check always fails and tickets are **never renewed** — users logged out after `session_max_time` (default 2h) regardless of activity. Fix in `auth_api.py` `checkAuth` after `ret = await handler(request)`: ```python from aiohttp_auth.auth.ticket_auth import _REISSUE_KEY if _REISSUE_KEY in request: @@ -638,658 +193,120 @@ if _REISSUE_KEY in request: if policy and hasattr(policy, 'remember_ticket'): await policy.remember_ticket(request, request[_REISSUE_KEY]) ``` +Diagnostic: users "kicked out while actively using the app" → verify this fix is present. See `references/session-timeout-architecture.md`. -**Diagnostic:** If users report "kicked out while actively using the app", check if this fix is present in `auth_api.py`. Without it, `process_response` silently skips reissue for every StreamResponse. +### ⚠️ 1.2.0+ routing: use directory/index.dspy OR apply extensionless-URL fallback patch +≤1.0.x auto-resolved `/api/status` → `/api/status.dspy`; 1.2.0 removed auto-extension → extensionless URLs raise `Exception: ... invalid path` (HTTP 500). `bricks/i18n.js` hardcodes `/i18n_getmsgs` (no extension) → every Bricks app 500s on load without a fix. -See `references/session-timeout-architecture.md` for full two-layer session architecture. +**Recommended: directory pattern** (`api/status/index.dspy`, resolved natively via `indexes` config). Migration: `cd app/api && for f in *.dspy; do mkdir -p ${f%.dspy} && mv $f ${f%.dspy}/index.dspy; done`. -### ⚠️ .dspy routing changed in 1.2.0+ — use directory/index.dspy pattern OR apply extensionless URL fallback patch +**Fallback patch (commit `7a297e9`, 2026-06-25)** — only for hardcoded extensionless URLs: +1. `processorResource.py` `_handle()`: when `url2file(path)` returns None, try `url2file(path + ext)` for each registered processor extension; then after `url2processor`, if `request_filename` is set and processor has no `real_path`, set `processor.real_path = self.request_filename`. +2. `baseProcessor.py` `set_run_env()`: compute `real_path` only if not already set (plain `url2file(request.path)` returns None for extensionless URLs). +Why two patches: `BaseProcessor.__init__` doesn't set `real_path`; patch 1 sets it before `handle()`, patch 2 stops `set_run_env` overwriting it. +Verify: `curl -u admin:pass http://localhost:9090/i18n_getmsgs` → HTTP 200 `{"success": true, "msgs": {...}}`; if 500 with log `TypeError: expected str, bytes or os.PathLike object, not NoneType`, patch 2 is missing. +❌ **Not recommended: nginx rewrite** (`rewrite ^/api/(.*)$ /api/$1.dspy break;`) — adds complexity, bypasses framework resolution. -**Problem:** In ahserver ≤1.0.x, requesting `/api/status` would automatically resolve to `/api/status.dspy`. In 1.2.0, auto-extension is removed — requests without extension raise `Exception: ... invalid path` (HTTP 500). +### ⚠️ aligner service: own venv + numpy<2 + PYTHONPATH +Aligner (`/data/ymq/aligner/`) uses its own venv `/data/ymq/aligner/py3/` (NOT `/share/vllm-0.8.5`); requires `ctc_segmentation` and `numpy<2` (numpy>=2 breaks ctc_segmentation). Start with `PYTHONPATH=/data/ymq/aligner/app:/data/ymq/aligner` — `aligner.py` does `from align import AlignEngine`; without PYTHONPATH: `ModuleNotFoundError: No module named 'align'`. Listens on 8080, `POST /api/align {audio_path, text}`. -**Quick fix — Extensionless URL fallback patch (2 files):** +### ⚠️ demucs pitfalls +- `/share/vllm-0.8.5` venv is root-owned → `sudo /share/vllm-0.8.5/bin/pip install demucs`; binary `/share/vllm-0.8.5/bin/demucs`; use `-n htdemucs` (4-stem) or `-n htdemucs_ft` (fine-tuned, better but slower). +- demucs 4.0.1 + newer torchaudio: `torchaudio.save()` → `save_with_torchcodec()` fails `RuntimeError: Could not load libtorchcodec` (needs `libnvrtc.so.13`, CUDA runtime, not on shared lib path). Separation completes 100% but save crashes. Fix: wrapper script monkey-patching `torchaudio.save` to `soundfile` (`sf.write(uri, wav.T if wav.shape[0]<=wav.shape[1] else wav, sample_rate)`); requires `soundfile` (present in `/share/vllm-0.8.5`); invoke: `/share/vllm-0.8.5/bin/python /tmp/demucs_wrapper.py --two-stems=vocals -o `. -The frontend `bricks/i18n.js` always requests `/i18n_getmsgs` (no extension). Without this patch, every Bricks app gets 500 on page load. +### ⚠️ GPU services: independent venv + longtasks pattern, NEVER FastAPI +**Rule:** all GPU services on the media server (ymq@opencomputing.net) MUST use ahserver + longtasks, not FastAPI. Each service needs its OWN venv — never install into `/share/vllm-0.8.5` (diffusers 0.35.2 + old transformers → `HybridCache` import error). **CRITICAL: search existing code FIRST** (media-server, aligner, songrate; `session_search(query="longtasks ahserver deploy")`) — reuse proven implementations, don't reinvent. +**longtasks quirks:** `submit_task(payload)` returns a **dict** `{'task_id': '...'}` (not a string) → `task_id = result.get('task_id')`; `get_status(task_id)` → `PENDING|RUNNING|SUCCEEDED|FAILED`. +**Template (ah.py):** subclass `LongTasks.process_task(payload, workid=None)` (payload may be JSON string — `json.loads` it); in `on_app_built`: `if env.longtasks: schedule_once(0.1, env.longtasks.run)`; in `init()`: `env.longtasks = MyTasks('redis://127.0.0.1:6379', 'myqueue', worker_cnt=1, stuck_seconds=3600)`. +**Setup checklist:** `python3 -m venv ~/my-service/venv` → `pip install ahserver appPublic sqlor longtasks aiohttp` → write ah.py + .dspy routes → `conf/config.json` (port, paths, processors) → `nohup python ah.py > service.log 2>&1 &`. +**GitHub is blocked on the GPU server:** SOCKS5 tunnel `ssh -N -D 1086 ymq@proxy-server` → clone with `git -c http.proxy=socks5h://127.0.0.1:1086 -c https.proxy=socks5h://127.0.0.1:1086 clone ` → `scp -r` to server. +See `references/gpu-service-longtasks-pattern.md` (wan22 example — DEPRECATED, replaced by wan27); `references/ssh-access-map.md` (passwordless hosts, server inventory). -**Patch 1: `processorResource.py` `_handle()` method** — after `url2file` returns None, try appending each registered processor extension: -```python -self.request_filename = self.url2file(str(request.path)) -# Fallback: try adding processor extensions for extensionless URLs -if not self.request_filename: - for ext, _ in self.y_processors: - candidate = self.url2file(str(request.path) + ext) - if candidate: - self.request_filename = candidate - break -``` - -Also add real_path preservation after processor creation: -```python -processor = self.url2processor(request, str(request.url), self.request_filename) -if processor: - # Fix real_path when fallback added an extension - if self.request_filename and (not hasattr(processor, 'real_path') or not processor.real_path): - processor.real_path = self.request_filename - ret = await processor.handle(request) -``` - -**Patch 2: `baseProcessor.py` `set_run_env()` method** — preserve pre-set real_path: -```python -# Only calculate real_path if not already set (e.g., by fallback in _handle) -if not hasattr(self, 'real_path') or self.real_path is None: - self.real_path = self.resource.url2file(request.path) -``` - -**Why two patches:** `BaseProcessor.__init__` does NOT set `real_path`. It's set in `set_run_env()` (line 71) by calling `url2file(request.path)` — which returns None for extensionless URLs. The processorResource patch sets `real_path` on the processor object before `handle()` is called. The baseProcessor patch prevents `set_run_env` from overwriting it. - -**Applied in:** ahserver commit `7a297e9` (2026-06-25) - -**Verification:** Test that `curl -u admin:pass http://localhost:9090/i18n_getmsgs` returns `{"success": true, "msgs": {...}}` (HTTP 200) instead of 500. If still 500, check `logs/app.log` for `TypeError: expected str, bytes or os.PathLike object, not NoneType` — indicates baseProcessor patch is missing. - -**Recommended approach** for new APIs: still use directory/index.dspy pattern. The patch above is primarily needed for `i18n_getmsgs` which is hardcoded in bricks.js and cannot be changed to `/i18n_getmsgs.dspy`. - -**✅ Recommended fix: directory + index.dspy pattern** - -Convert `app/api/status.dspy` → `app/api/status/index.dspy`. The framework's `indexes` config (`["index.html", "index.dspy"]`) handles directory resolution natively: - -``` -app/api/ -├── status/index.dspy # GET /api/status -├── demucs/index.dspy # GET /api/demucs -└── pipeline/ - ├── submit/index.dspy # POST /api/pipeline/submit - └── status/index.dspy # GET /api/pipeline/status -``` - -**Migration script:** -```bash -cd app/api && for f in *.dspy; do mkdir -p ${f%.dspy} && mv $f ${f%.dspy}/index.dspy; done -``` - -**❌ NOT recommended: nginx rewrite** — adds complexity, bypasses framework resolution: -```nginx -# Avoid this unless you have a specific reason -location /api/ { rewrite ^/api/(.*)$ /api/$1.dspy break; proxy_pass http://backend; } -``` - -See `references/api-routing-patterns.md` for complete routing guide including nginx proxy path stripping, startswiths for high-traffic APIs, and version migration cheat sheet. - -### ⚠️ aligner service requires own venv and numpy<2 - -**Problem:** The aligner service at `/data/ymq/aligner/` uses its own Python venv (`/data/ymq/aligner/py3/`), NOT the shared `/share/vllm-0.8.5` venv. It requires `ctc_segmentation` and `numpy<2`. numpy>=2 breaks ctc_segmentation. - -**Fix:** -```bash -/data/ymq/aligner/py3/bin/pip install ctc_segmentation 'numpy<2' -# Start (CRITICAL: PYTHONPATH must include app/ for relative imports): -cd /data/ymq/aligner && PYTHONPATH=/data/ymq/aligner/app:/data/ymq/aligner \ - nohup /data/ymq/aligner/py3/bin/python app/aligner.py > /data/ymq/logs/aligner.log 2>&1 & -# Listens on port 8080, POST /api/align with {audio_path, text} -``` - -**Pitfall:** `aligner.py` does `from align import AlignEngine` — this requires `PYTHONPATH` to include the `app/` directory. Without it: `ModuleNotFoundError: No module named 'align'`. - -### ⚠️ demucs requires sudo pip install on shared venv - -**Problem:** `/share/vllm-0.8.5` venv is owned by root. `pip install demucs` fails with Permission denied. - -**Fix:** `sudo /share/vllm-0.8.5/bin/pip install demucs`. Binary at `/share/vllm-0.8.5/bin/demucs`. Use `-n htdemucs` (4-stem model) or `-n htdemucs_ft` (fine-tuned, higher quality but slower). - -### ⚠️ demucs save fails with torchcodec/libnvrtc error - -**Problem:** demucs 4.0.1 + newer torchaudio tries `torchaudio.save()` which calls `save_with_torchcodec()`. This fails with `RuntimeError: Could not load libtorchcodec` because it needs `libnvrtc.so.13` (CUDA runtime) which may not be in the shared library path. Separation completes (100%) but saving crashes. - -**Fix:** Create a wrapper script that monkey-patches `torchaudio.save` to use `soundfile` instead: - -```python -#!/usr/bin/env python3 -# /tmp/demucs_wrapper.py — use this instead of the demucs binary -import sys -import soundfile as sf -import torch -import torchaudio - -_original_save = torchaudio.save - -def patched_save(uri, src, sample_rate, **kwargs): - if isinstance(uri, str): - wav = src.cpu().numpy() - if wav.shape[0] <= wav.shape[1]: - wav = wav.T # soundfile expects (samples, channels) - sf.write(uri, wav, sample_rate) - else: - _original_save(uri, src, sample_rate, **kwargs) - -torchaudio.save = patched_save - -from demucs.separate import main -sys.exit(main()) -``` - -**Usage in .dspy:** -```python -cmd = ['/share/vllm-0.8.5/bin/python', '/tmp/demucs_wrapper.py', - '--two-stems=vocals', '-o', output_dir, input_file] -``` - -**Requires:** `soundfile` package installed (`pip install soundfile`). Already available in `/share/vllm-0.8.5` as of June 2026. - -### ⚠️ GPU service deployment: always use independent venv + longtasks, never FastAPI - -**Rule:** All GPU services on the media server (ymq@opencomputing.net) MUST use ahserver + longtasks pattern, NOT FastAPI or other frameworks. Each service needs its own Python venv — never install into `/share/vllm-0.8.5` (shared venv has version conflicts: diffusers 0.35.2 + old transformers causes `HybridCache` import error). - -**Why:** The longtasks pattern (Redis queue + async worker) is already proven for aligner, songrate, media-server, and fastwhisper. Adding FastAPI introduces dependency conflicts and duplicates the async task infrastructure that longtasks already provides. - -**CRITICAL: Check existing code FIRST.** Before writing a new GPU service, search session history and existing services (media-server, aligner, songrate) for patterns. The user expects you to reuse proven implementations, not reinvent from scratch. Example: `session_search(query="longtasks ahserver deploy")` or `cat ~/media-server/ah.py`. - -**longtasks API quirks:** -- `submit_task(payload)` returns a **dict** with `task_id` key: `{'task_id': 'abc123'}` — not a plain string -- To get the task ID: `result = await longtasks.submit_task(payload); task_id = result.get('task_id')` -- `get_status(task_id)` returns task state: `PENDING`, `RUNNING`, `SUCCEEDED`, `FAILED` - -**Template (ah.py for a longtasks-based GPU service):** -```python -# -*- coding:utf-8 -*- -from ahserver.webapp import webapp -from ahserver.serverenv import ServerEnv -from ahserver.configuredServer import add_startup -from longtasks.longtasks import LongTasks, schedule_once -from appPublic.log import debug -import json - -class MyTasks(LongTasks): - async def process_task(self, payload, workid=None): - if isinstance(payload, str): payload = json.loads(payload) - # dispatch to handler... - -async def on_app_built(app): - env = ServerEnv() - if env.longtasks: - schedule_once(0.1, env.longtasks.run) - -def init(): - env = ServerEnv() - env.longtasks = MyTasks('redis://127.0.0.1:6379', 'myqueue', worker_cnt=1, stuck_seconds=3600) - add_startup(on_app_built) - -if __name__ == '__main__': - webapp(init) -``` - -**Setup checklist:** -1. `python3 -m venv ~/my-service/venv` (independent venv) -2. `pip install ahserver appPublic sqlor longtasks aiohttp` in the venv -3. Write ah.py + .dspy routes following the pattern above -4. `conf/config.json` with port, paths, processors -5. Start: `cd ~/my-service && source venv/bin/activate && nohup python ah.py > service.log 2>&1 &` - -**Pitfall: GitHub is blocked on the GPU server.** Two solutions: -1. Clone repos locally with SOCKS5 proxy, then `scp` to server: - def init(): - env = ServerEnv() - env.longtasks = MyTasks('redis://127.0.0.1:6379', 'myqueue', worker_cnt=1, stuck_seconds=3600) - add_startup(on_app_built) - - if __name__ == '__main__': - webapp(init) - ``` - - **Setup checklist:** - 1. `python3 -m venv ~/my-service/venv` (independent venv) - 2. `pip install ahserver appPublic sqlor longtasks aiohttp` in the venv - 3. Write ah.py + .dspy routes following the pattern above - 4. `conf/config.json` with port, paths, processors - 5. Start: `cd ~/my-service && source venv/bin/activate && nohup python ah.py > service.log 2>&1 &` - - **Pitfall:** GitHub is blocked on the GPU server. Solutions: - 1. SOCKS5 proxy tunnel via jump host: - ```bash - # Start tunnel (on local machine): - ssh -N -D 1086 ymq@proxy-server - ``` - 2. Clone repo locally with proxy: - ```bash - git -c http.proxy=socks5h://127.0.0.1:1086 \ - -c https.proxy=socks5h://127.0.0.1:1086 \ - clone https://github.com/user/repo.git - ``` - 3. Copy to server: - ```bash - scp -r repo user@server:~/destination/ - ``` - - See `references/gpu-service-longtasks-pattern.md` for the wan22 video generation service example (DEPRECATED — replaced by wan27). - See `references/wan22-deployment.md` for Wan2.2 model-specific deployment issues and solutions (DEPRECATED). - See `references/ssh-access-map.md` for SSH access architecture (which domains are passwordless, server inventory). - -### ⚠️ SSH connection throttling on GPU server - -### ⚠️ startswiths config: must use `registerfunction` key, not `path` - -**Problem:** Adding a `startswiths` entry with `{"leading": "/api/foo", "path": "/app/api/foo/"}` causes `KeyError: 'registerfunction'` on every request to that route. - -**Cause:** `FunctionProcessor.path_call()` unconditionally reads `self.config_opts['registerfunction']`. The `path` key is NOT a valid alternative — it's used by a different routing mechanism (processor-based routing via `paths` config). - -**Fix:** Always use `registerfunction` for `startswiths` entries: +### ⚠️ startswiths config: must use `registerfunction`, NOT `path` +`{"leading": "/api/foo", "path": ...}` → `KeyError: 'registerfunction'` on EVERY request (FunctionProcessor.path_call() unconditionally reads `config_opts['registerfunction']`; `path` belongs to a different routing mechanism). Correct: ```json -"startswiths": [ - {"leading": "/idfile", "registerfunction": "idfile"}, - {"leading": "/api/asr", "registerfunction": "asr"}, - {"leading": "/api/transcribe", "registerfunction": "transcribe"} -] +"startswiths": [{"leading": "/idfile", "registerfunction": "idfile"}] ``` +Prefer directory/index.dspy for new APIs; use startswiths + registerfunction only for Python-registered function handlers (`RegisterFunction`). -For new API endpoints, use the **directory/index.dspy pattern** instead of `startswiths` — the processor-based routing handles `app/api/foo/index.dspy` automatically via `paths` + `indexes` config. Only use `startswiths` with `registerfunction` for function-based handlers registered via `RegisterFunction` (like `idfile`, `asr`, etc. that are defined in Python, not .dspy files). +### ⚠️ idfile download endpoint: explicit import AND config required +`/idfile?path=...` → 500 "invalid path" unless BOTH: (1) `from ahserver import filedownload` in ah.py (registers `idfile`/`download` via RegisterFunction at import time; `path_download()` resolves `path` via `FileStorage().realPath()` → file_response); (2) `startswiths` entry `{"leading": "/idfile", "registerfunction": "idfile"}`. +⚠️ Chinese chars in `path` query MUST be URL-encoded (`%E9%94%99%E9%A2%91`, not `错频`) — server rejects unencoded. -### ⚠️ idfile download endpoint requires explicit setup +### ⚠️ appPublic.worker missing schedule functions (legacy ≤5.2.x / ahserver ≤1.0.8) +Only exports `AsyncWorker` — `from appPublic.worker import schedule_once` → ImportError. Fixed in appPublic ≥5.3.0 (`pip install --upgrade apppublic`). Workaround: define local `get_event_loop()` (= `asyncio.get_event_loop()`), `schedule_once(delay, coro_func)` and `schedule_interval(interval, coro_func)` as `asyncio.ensure_future` wrappers. Applies to longtasks and any `appPublic.worker` scheduling code; clear `__pycache__` after patching. -**Problem:** Requesting `/idfile?path=...` returns 500 "invalid path" even though the file exists. +### ⚠️ stream_response re-raises ClientConnectionResetError — noisy logs +`ahserver/globalEnv.py stream_response()` wraps ALL write errors in a generic `Exception` and re-raises, including `ClientConnectionResetError` (client disconnected mid-stream) → every client timeout during SSE floods logs with stack traces. Fix (commit `387726e`): catch `ClientConnectionResetError` separately and `break` the streaming loop; other exceptions still logged (`write error{e=}, {d=}`) and re-raised. -**Cause:** The `idfile` endpoint is registered by `ahserver/filedownload.py`, but it must be explicitly imported AND configured. - -**Fix — TWO steps required:** - -1. Import in `ah.py`: +### ⚠️ `@routes` decorator removed in 1.2.0+ — use `app.router.add_route()` +`from ahserver.webapp import webapp, routes, add_startup` → `ImportError: cannot import name 'routes'`; leftover `@routes.get(...)` → `NameError`. Fix: register inside `on_app_built`: ```python -from ahserver import filedownload # Registers 'idfile' and 'download' with RegisterFunction -``` - -2. Add to `conf/config.json`: -```json -"startswiths": [ - {"leading": "/idfile", "registerfunction": "idfile"}, - {"leading": "/api/...", "registerfunction": "..."} -] -``` - -**How it works:** `filedownload.py` calls `rf.register('idfile', path_download)` at import time. `path_download()` resolves the `path` query param via `FileStorage().realPath()` and returns a `file_response`. The `startswiths` config routes `/idfile` to `FunctionProcessor` which looks up the registered function. - -**URL encoding:** Chinese characters in the `path` query param MUST be URL-encoded (`%E9%94%99%E9%A2%91` not `错频`). The server rejects unencoded Chinese in query strings. - -### ⚠️ appPublic.worker missing schedule functions (legacy versions only) - -**Problem:** `appPublic.worker` in versions ≤5.2.x (e.g. bundled with ahserver ≤1.0.8) only exports `AsyncWorker` — no `get_event_loop`, `schedule_once`, or `schedule_interval`. Code that does `from appPublic.worker import schedule_once` will crash with `ImportError`. - -**Fixed in appPublic ≥5.3.0.** All three functions are now available. When upgrading, also upgrade apppublic: `pip install --upgrade apppublic`. - -**Workaround for environments stuck on old versions:** -```python -def get_event_loop(): - return asyncio.get_event_loop() - -def schedule_once(delay, coro_func): - async def _delayed(): - await asyncio.sleep(delay) - await coro_func() - asyncio.ensure_future(_delayed()) - -def schedule_interval(interval, coro_func): - async def _loop(): - while True: - await asyncio.sleep(interval) - await coro_func() - asyncio.ensure_future(_loop()) -``` - -**Applies to:** longtasks module and any code using `appPublic.worker` scheduling. After patching, clear `__pycache__` dirs. - -### ⚠️ stream_response re-raises ClientConnectionResetError — noisy logs + unnecessary exceptions - -**Problem:** `ahserver/globalEnv.py` `stream_response()` wraps all write errors in a generic `Exception` and re-raises, including `ClientConnectionResetError` (client disconnected mid-stream). This floods logs with stack traces every time a client times out and disconnects during SSE streaming. - -**Traceback pattern:** -``` -aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport -→ globalEnv.py:130 raise e -→ Exception: write errore=ClientConnectionResetError('Cannot write to closing transport'), d='data: ...' -``` - -**Fix (commit 387726e):** Catch `ClientConnectionResetError` separately and `break` the loop: -```python -from aiohttp.client_exceptions import ClientConnectionResetError - -async for d in async_data_generator(): - try: - await res.write(...) - except ClientConnectionResetError: - # Client disconnected — not a server error, stop streaming - break - except Exception as e: - e = Exception(f'write error{e=}, {d=}') - exception(f'{e}\n{format_exc()}') - raise e -``` - -**When this matters:** High-concurrency load tests against streaming endpoints. Without this fix, every client timeout generates a full stack trace in server logs, making it impossible to find real errors. - -### ⚠️ `@routes` decorator removed in ahserver 1.2.0+ — use `app.router.add_route()` instead - -**Problem:** Code using `from ahserver.webapp import webapp, routes, add_startup` crashes with `ImportError: cannot import name 'routes'`. Code that fixes the import but keeps `@routes.get(...)` decorators crashes with `NameError: name 'routes' is not defined`. - -**Cause:** `routes` (an aiohttp `web.RouteTableDef`) was removed from `ahserver.webapp` in 1.2.0. The new pattern registers routes inside the `on_app_built` callback. - -**Fix — replace @routes decorators with app.router.add_route():** -```python -# ❌ OLD (ahserver ≤1.0.x) -from ahserver.webapp import webapp, routes, add_startup - -@routes.get('/api/health') -async def health(request): - return {'status': 'ok'} - -# ✅ NEW (ahserver ≥1.2.0) -from ahserver.webapp import webapp -from ahserver.configuredServer import add_startup - -async def health(request): - return {'status': 'ok'} - async def on_app_built(app): app.router.add_route('GET', '/api/health', health) - # ... other routes ... - -def init(): - add_startup(on_app_built) - -if __name__ == '__main__': - webapp(init) ``` +Migration: `grep -rn '@routes\.' /data/ymq/*/ah.py`, remove decorator lines, register handlers in `on_app_built`. -**Migration regex** for bulk-fixing old services: -```bash -# Find all @routes decorators -grep -rn '@routes\.' /data/ymq/*/ah.py -# Remove decorator lines, keep function defs, register in on_app_built -``` +### ⚠️ CRITICAL: `@` inside comments can truncate files on production deploy +Incident 2026-05-26: `auth_api.py` comment `# redis = await aioredis.from_url("redis://127.0.0.1:6379")` preceded by `@web.middleware` text — the deployment process truncated the file at that line, losing ~30 lines (`aiohttp_session.setup(app, storage)`, `auth.setup(app, policy)`, `app.middlewares.append(self.checkAuth)`). Symptom: all requests 500, `checkAuth` never fires, no auth logs. +Prevention: never put `@something` patterns in comments (esp. strings containing `@`); reference URLs without the `@user:pass` portion. After deploying auth_api.py changes ALWAYS verify: `grep -c "middlewares.append" auth_api.py` ≥ 1; `grep -E "middlewares.append|setup\(app|aiohttp_session.setup" auth_api.py` → 3+ lines. -### ⚠️ CRITICAL: Comments containing `@` patterns can truncate files on production +### ⚠️ Jinja2 in .ui/.tmpl: limited filter set +Non-standard filters (e.g. `|ternary`) → `jinja2.exceptions.TemplateAssertionError: No filter named 'ternary'`. Only standard Jinja2 filters + ahserver built-ins (`entire_url()`, `get_user()`, `configValue()`) are registered. Use if/else: `{{get_user().nick_name if get_user() else '登录'}}`. Rule: keep Jinja2 expressions simple in `.ui` JSON templates; complex conditionals handled client-side in JS. -**Incident (2026-05-26):** Line 134 of `auth_api.py` contained: -```python -# redis = await aioredis.from_url("redis://127.0.0.1:6379") -``` -When this comment was preceded by `@web.middleware` text (from the previous line's content), the deployment process truncated the file at this line, losing ~30 lines of critical code including: -- `aiohttp_session.setup(app, storage)` -- `auth.setup(app, policy)` -- `app.middlewares.append(self.checkAuth)` +### ⚠️ Never modify sage/conf/config.json during feature development +It is the PRODUCTION config for the Sage platform. Config changes (DB password, session settings) are handled manually by the user. -**Symptom:** All requests return 500. `checkAuth` middleware never fires — no auth logs appear. The site is completely broken. +### Verify auth_api.py after any edit +`wc -l ahserver/auth_api.py` ≈ 193 (not fewer); `grep "middlewares.append"` present; `python3 -c "import ast; ast.parse(open('ahserver/auth_api.py').read())"` parses. -**Prevention:** -- Never put `@something` patterns inside comments in Python files, especially strings containing `@` -- If you must reference a URL with `@` in a comment, use a placeholder like `redis://127.0.0.1:6379` without the `@user:pass` portion -- After deploying `auth_api.py` changes, **always verify** that `setupAuth()` is intact: `grep -c "middlewares.append" auth_api.py` should return >= 1 - -**How to verify middleware is registered:** -```bash -# Check that setupAuth() contains all critical calls -grep -E "middlewares.append|setup\(app|aiohttp_session.setup" auth_api.py -# Should output 3+ lines. If fewer, file is truncated. -``` - -### ⚠️ Jinja2 in .ui/.tmpl files — limited filter set - -**Problem:** Using `|ternary()` or other non-standard Jinja2 filters in `.ui` files causes `jinja2.exceptions.TemplateAssertionError: No filter named 'ternary'`. - -**Cause:** ahserver's Jinja2 environment does not register custom filters like `ternary`. Only standard Jinja2 filters are available plus ahserver's built-in functions (`entire_url()`, `get_user()`, `configValue()`). - -**Fix:** Use standard Jinja2 constructs: -``` -{# BAD — ternary filter doesn't exist #} -"{{entire_url('/path')|ternary(get_user().nick_name, '登录')}}" - -{# GOOD — use if/else expression #} -"{{get_user().nick_name if get_user() else '登录'}}" - -{# GOOD — just use static text #} -"管理" -``` - -**Rule:** In `.ui` JSON templates, keep Jinja2 expressions simple. Complex conditionals should be handled client-side in JavaScript. - -### ⚠️ Do not modify sage/conf/config.json during feature development - -`conf/config.json` is the **production configuration** for the Sage platform. Feature development sessions must never modify it. If a task requires config changes (DB password, session settings, etc.), the user will handle production config changes manually. - -### Verifying auth_api.py after any edit - -After modifying `auth_api.py`, always verify: -```bash -# Line count should be ~193 (not fewer) -wc -l ahserver/auth_api.py -# Must have middleware registration -grep "middlewares.append" ahserver/auth_api.py -# Must have complete setupAuth method -python3 -c "import ast; ast.parse(open('ahserver/auth_api.py').read())" -``` - -## Common Issues and Solutions - -### Python 3.12+ Compatibility -As mentioned earlier, modify aioredis files for Python 3.12+ compatibility. +## Common Issues ### Password Encryption - -Database passwords in `conf/config.json` are AES-ECB base64 encoded using the config's `password_key`: - +DB passwords in `conf/config.json` are AES-ECB base64 using `password_key`: ```python from appPublic.aes import aes_encode_b64 -key = getConfig().password_key # from config.json -encoded = aes_encode_b64(key, 'plaintext_password') -``` - -The `sqlor/sor.py` class calls `self.unpassword()` in `__init__` which decodes the password via `aes_decode_b64(key, password)`. **The password must be valid AES-ECB base64** — plain text or RC4-encoded passwords will cause `ValueError: The length of the provided data is not a multiple of the block length`. - -For quick testing, encode the password directly: -```bash -python3 -c "from appPublic.aes import aes_encode_b64; print(aes_encode_b64('YOUR_PASSWORD_KEY', 'test'))" -``` - -Also available: the legacy RC4 encoding via `appPublic.rc4`: -```bash -python -m ahserver.dbpassword /path/to/your/app password123 +encoded = aes_encode_b64(getConfig().password_key, 'plaintext_password') ``` +`sqlor/sor.py` calls `self.unpassword()` in `__init__` (aes_decode_b64). ⚠️ Password MUST be valid AES-ECB base64 — plain text or RC4-encoded → `ValueError: The length of the provided data is not a multiple of the block length`. Quick encode: `python3 -c "from appPublic.aes import aes_encode_b64; print(aes_encode_b64('YOUR_PASSWORD_KEY', 'test'))"`. Legacy RC4: `python -m ahserver.dbpassword /path/to/app password123`. ### Debugging -- Check logs in console output -- Use `format_exc()` for detailed error information -- Access request context via `request._run_ns` +Check console logs; `format_exc()` for detailed errors; request context via `request._run_ns`. -## Performance Optimization - -- Uses uvloop and httptools for high performance -- Connection pooling for databases -- Async file operations with aiofiles -- Efficient JSON handling with ujson - -### Benchmark methodology - -To compare ahserver vs pure aiohttp performance, create a test that measures both with identical static files: - -```python -# Pure aiohttp baseline -app = web.Application() -app.router.add_static("/static/", TEST_DIR) - -# ahserver full chain (session + auth + ProcessorResource) -app = AHApp(client_max_size=1000000000) -res = ProcessorResource("/", TEST_DIR, processors={}) -app.router.register_resource(res) -auth = AuthAPI() -await auth.setupAuth(app) -``` - -**Key finding from May 2026 benchmark:** On local SSD with Redis session storage and `clientinfo` log level: -- Pure aiohttp: **avg 1.23ms** for 48KB static file -- ahserver (before fixes): **avg 1.13ms** — actually faster than aiohttp baseline -- ahserver (after all 3 fixes): **avg 1.10ms** — only 0.03ms improvement - -**This means the 17-second delay in production is NOT caused by ahserver code overhead.** The bottleneck is likely: -- Network/proxy layer (Nginx reverse proxy, TLS handshake, keep-alive issues) -- DNS resolution -- Connection pool exhaustion -- Something in the production deployment not present in local testing - -**Recommended production diagnostics:** -1. Use browser Network panel to check TTFB (Time To First Byte) -2. Check if delay is in connection establishment or data transfer -3. Add timing middleware to measure each layer: - ```python - @web.middleware - async def timing_middleware(request, handler): - t0 = time.time() - resp = await handler(request) - t1 = time.time() - print(f'[{request.path}] total={t1-t0:.3f}s') - return resp - ``` -4. Compare response time with and without Nginx in the path -4. Check `auth_api.py` timecost log: if `AUTH_MS` is small but total is slow, bottleneck is NOT in ahserver +## Performance: benchmark conclusions +May 2026 (local SSD, Redis session storage, clientinfo level, 48KB static file): pure aiohttp **1.23ms** vs ahserver before fixes **1.13ms** / after 3 fixes **1.10ms**. → ahserver code overhead is negligible; a 17s production delay is NOT caused by ahserver. Likely causes: network/proxy layer (nginx reverse proxy, TLS handshake, keep-alive), DNS, connection pool exhaustion, deployment differences. +Diagnostics: (1) browser Network panel TTFB; (2) timing middleware (`@web.middleware` logging `time.time()` around handler); (3) compare with/without nginx in path; (4) timecost AUTH_MS rule above. See `references/performance-benchmark-may2026.md`, `references/performance-diagnostics.md`. ## Multi-Service Deployment - -See `references/multi-service-media-platform.md` for the architecture pattern used when deploying multiple ahserver instances behind nginx — covers IP filtering via geo block, async (longtasks) vs sync (lock) service split, port allocation conventions, and inter-service localhost calls. - -### Pipeline Orchestration Principle - -**Services are capability centers, not agents.** When building pipelines that span multiple services (e.g., KTV song production: lyrics → music → video → subtitle → merge), the orchestration logic belongs in the calling agent/process, not embedded in one of the services. - -Each service should: -- Expose atomic capabilities via REST APIs -- Accept file uploads or paths as input -- Return results (file paths, URLs, status) -- Not know about upstream/downstream steps - -The agent/orchestrator should: -- Call services in sequence -- Handle state transitions and error recovery -- **Be the file router** — download from one service, upload to the next. Services must NOT directly copy files between each other, even when co-located on the same machine. This ensures the pipeline works across distributed deployments. -- Implement retry/threshold logic - -**Why:** This keeps services focused, testable, and reusable. The agent has full context and can adapt the pipeline based on intermediate results. - -See `references/nginx-proxy-ssl-template.md` for nginx proxy config template (SSL on non-standard port, geo-block IP filtering, X-Forwarded headers) and Let's Encrypt DNS-01 certificate workflow for environments where ports 80/443 are unavailable. - -See `references/api-routing-patterns.md` for the complete guide to ahserver URL routing: directory/index.dspy pattern, nginx proxy_pass trailing slash semantics, startswiths vs processor lookup, and 1.0.x → 1.2.0 migration cheat sheet. +**Services are capability centers, not agents.** Each service: exposes atomic capabilities via REST, accepts uploads/paths as input, returns results (paths/URLs/status), knows nothing about upstream/downstream steps. The orchestrating agent: calls services in sequence, handles state transitions/error recovery/retry, and **is the file router** — downloads from one service, uploads to the next. Services must NOT copy files directly between each other, even co-located (keeps pipelines distributable). See `references/multi-service-media-platform.md` (geo-block IP filtering, async longtasks vs sync lock split, port conventions, inter-service localhost calls); `references/nginx-proxy-ssl-template.md` (SSL non-standard port, X-Forwarded headers, Let's Encrypt DNS-01 where 80/443 unavailable). ## Session & Cookie Timeout - -See `references/session-timeout-architecture.md` for the two-layer session system (aiohttp_session + aiohttp_auth ticket), timeout/renewal semantics, and how to configure sliding-window ticket renewal. +Two-layer session system (aiohttp_session + aiohttp_auth ticket) — timeout/renewal semantics and sliding-window ticket renewal: see `references/session-timeout-architecture.md`. ## Hot Reload - -ahserver supports hot-reloading of cached resources without restart. Each worker process independently watches file mtimes via `stat()`, so it works safely with `reuse_port=True` multi-process deployment — no Redis pub/sub, signals, or cross-process coordination needed. - -### What auto-reloads without config -- **.dspy files** — read from disk on every request (no cache) -- **.md files** — read from disk on every request (no cache) -- **.tmpl / .ui files** — Jinja2 `auto_reload` checks mtime natively - -### What needs hot_reload config -- **config.json** — `JsonConfig` is a singleton; cleared on change, next `getConfig()` reloads -- **i18n/*/msg.txt** — `MiniI18N` is a singleton + `ServerEnv.myi18n` cache; both cleared on change - -### Configuration (conf/config.json) +Built-in (`ahserver/hotreload.py`): watches file mtimes via `stat()`, multi-process safe with `reuse_port=True` (each worker runs its own FileWatcher; no Redis pub/sub, signals, or cross-process coordination). +- **Auto (no config):** `.dspy`/`.md` read from disk every request (no cache); `.tmpl`/`.ui` via Jinja2 `auto_reload` (mtime check). +- **Needs hot_reload config:** `conf/config.json` (JsonConfig singleton; cleared on change, next `getConfig()` reloads) and `i18n/*/msg.txt` (MiniI18N singleton + `ServerEnv.myi18n`; both cleared). ```json -"hot_reload": true // enable with default 2s interval +"hot_reload": true // enable, default 2s interval "hot_reload": {"enabled": true, "interval": 5} // custom interval -// omit or false // disabled (default) +// omit or false → disabled (default) ``` - -### Architecture -Module: `ahserver/hotreload.py` -- `FileWatcher` — tracks mtime per path, returns changed paths -- `HotReloader` — on change: sets `JsonConfig.instance = None` and `MiniI18N.instance = None` (SingletonDecorator pattern) -- `hot_reload_task(app, reloader)` — asyncio background task registered via `app.on_startup` -- Throttled by configurable interval (default 2s) to avoid excessive stat() calls +- **`GET /__hot_reload__`** (only when enabled) triggers `invalidate_all_caches()`: rbac (UserPermissions.ur_caches + rp_caches → LRU.clear() + invalidate_rp_cache()), pricing (PricingProgram.pricing_data → clear()), uapi (UAPIData.apidata + org_users → clear()), llmage (_uapi_cache + _uapiio_cache → invalidate). Each module cleared independently (try/except — one failure doesn't block others). +- **Cache TTLs:** rbac.ur_caches 5min, rbac.rp_caches 10min (DB events), pricing.pricing_data none (DB events), llmage._uapi_cache 5min (invalidate_uapi_cache()), **uapi.UAPIData: NO TTL, NO auto-invalidation — use `/__hot_reload__`**. +⚠️ Multi-process caveat: with `reuse_port=True` `/__hot_reload__` only clears the single worker handling the request — hit multiple times or rely on the file-mtime watcher (works in all workers independently). ## Version History +- **1.3.0:** hot-reload module, file watching, `/__hot_reload__` endpoint, `invalidate_all_caches()`. +- **1.2.0 (BREAKING):** `RegisterCoroutine` REMOVED → `add_startup(callback)`; `.dspy` auto-extension REMOVED (`/api/status` no longer → `/api/status.dspy`; needs directory/index.dspy, startswiths, or the fallback patch); added uvloop + httptools; `appPublic.worker` now exports `schedule_once`/`schedule_interval`/`get_event_loop` (was only `AsyncWorker` in ≤5.2.x). +- **1.0.8:** added `server_error(errcode)` global; `request._run_ns` for global env access. -### 1.3.0 -- Hot-reload module, file watching, /__hot_reload__ endpoint, invalidate_all_caches() - -### 1.2.0 (BREAKING CHANGES) -- `RegisterCoroutine` REMOVED → use `add_startup(callback)` instead -- `.dspy` auto-extension REMOVED → `/api/status` no longer resolves to `/api/status.dspy`; needs nginx rewrite or explicit `startswiths` in config -- Added uvloop and httptools for performance improvement -- `appPublic.worker` now exports `schedule_once`, `schedule_interval`, `get_event_loop` (was only `AsyncWorker` in ≤5.2.x) - -### 1.0.8 -- Added `server_error(errcode)` global function for HTTP error handling -- Added `request._run_ns` for accessing global environment variables - -## API Testing - -See `references/api-testing-patterns.md` for testing ahserver APIs with Python urllib instead of curl — avoids shell escaping issues with Bearer tokens and JSON payloads. - -## vLLM Integration - -See `references/vllm-integration-pattern.md` for integrating vLLM's AsyncLLMEngine with ahserver for high-throughput LLM inference. Covers: -- Complete server implementation using AsyncLLMEngine + stream_response -- Multi-GPU deployment pattern (8 GPUs × 2 instances) -- Critical parameters: `--max-model-len`, `--max-num-seqs`, `--gpu-memory-utilization` -- Common pitfalls: heredoc variable expansion, ClientConnectionResetError handling, dead DB connections after OOM -- Migration guide from transformers-based inference to vLLM - -## Hot Reload - -ahserver has a built-in hot-reload system (`ahserver/hotreload.py`) that watches file mtimes and clears cached resources. Multi-process safe — each worker process runs its own FileWatcher, no cross-process coordination needed. - -### Configuration - -```json -"hot_reload": true -// or with custom interval: -"hot_reload": {"enabled": true, "interval": 5} -``` - -### What Gets Hot-Reloaded - -**Automatic (file mtime detection):** -| File | Cache Cleared | -|------|--------------| -| `conf/config.json` | JsonConfig singleton + all module caches | -| `i18n/*/msg.txt` | MiniI18N singleton + ServerEnv.myi18n | -| `.tmpl`/`.ui` files | Jinja2 auto_reload (built-in) | -| `.dspy`/`.md` files | No cache — read from disk every request | - -**Manual (HTTP endpoint, only when hot_reload enabled):** - -`GET /__hot_reload__` — triggers `invalidate_all_caches()` which clears: - -| Module | Cache Object | Clear Method | -|--------|-------------|-------------| -| rbac | UserPermissions.ur_caches + rp_caches | LRU.clear() + invalidate_rp_cache() | -| pricing | PricingProgram.pricing_data | dict.clear() | -| uapi | UAPIData.apidata + org_users | dict.clear() | -| llmage | _uapi_cache + _uapiio_cache | invalidate_uapi_cache() | - -Each module cleared independently with try/except — one module's import failure won't block others. - -### Multi-Process Caveat - -With `reuse_port=True`, `GET /__hot_reload__` only clears the single worker handling the request. Hit multiple times or rely on file-based hot-reload which works across all workers independently. - -### Module Cache Architecture - -| Module | TTL | Auto-invalidation | -|--------|-----|-------------------| -| rbac.ur_caches | 5min | DB events | -| rbac.rp_caches | 10min | DB events | -| pricing.pricing_data | none | DB events | -| uapi.UAPIData | **none** | **none** — use /__hot_reload__ | -| llmage._uapi_cache | 5min | invalidate_uapi_cache() | - -⚠️ uapi.UAPIData has no TTL and no auto-invalidation. - -## Version History - -### 1.3.0 -- Hot-reload module, file watching, /__hot_reload__ endpoint, invalidate_all_caches() - -### 1.2.0 -- Added uvloop and httptools for performance improvement - -### 1.0.8 -- Added `server_error(errcode)` global function for HTTP error handling -- Added `request._run_ns` for accessing global environment variables \ No newline at end of file +## References +- `references/api-routing-patterns.md` — directory/index.dspy, nginx proxy_pass trailing slash semantics, startswiths vs processor lookup, 1.0.x → 1.2.0 migration cheat sheet. +- `references/api-testing-patterns.md` — test ahserver APIs with Python urllib instead of curl (avoids shell escaping issues with Bearer tokens / JSON payloads). +- `references/file-upload-patterns.md` — upload examples, curl testing, debugging. +- `references/session-timeout-architecture.md` — two-layer session, timeout/renewal. +- `references/performance-benchmark-may2026.md`, `references/performance-diagnostics.md` — benchmark results, middleware chain breakdown. +- `references/multi-service-media-platform.md`, `references/nginx-proxy-ssl-template.md`, `references/ssh-access-map.md` — multi-instance nginx deployment, SSL template, SSH access map. +- `references/vllm-integration-pattern.md` — AsyncLLMEngine + stream_response for high-throughput inference; 8 GPUs × 2 instances; critical params `--max-model-len`/`--max-num-seqs`/`--gpu-memory-utilization`; pitfalls: heredoc variable expansion, ClientConnectionResetError, dead DB connections after OOM; transformers → vLLM migration. +- `references/gpu-service-longtasks-pattern.md` (wan22 example — DEPRECATED, replaced by wan27); `references/wan22-deployment.md` (DEPRECATED). diff --git a/skills_library/all/crud-definition-spec/SKILL.md b/skills_library/all/crud-definition-spec/SKILL.md index 8ad663c..be41221 100644 --- a/skills_library/all/crud-definition-spec/SKILL.md +++ b/skills_library/all/crud-definition-spec/SKILL.md @@ -12,24 +12,23 @@ trigger_conditions: # CRUD Definition Specification ## Overview -This skill defines the standardized JSON format for CRUD (Create, Read, Update, Delete) operations that integrate with the bricks-framework frontend and sqlor-database-module backend. The framework automatically selects between list view and tree view based on table structure. +Standardized JSON format for CRUD operations integrating bricks-framework (frontend) and sqlor-database-module (backend). The framework auto-selects list vs tree view from table structure. Files live in the module's `json/` directory and are generated into `wwwroot//` UIs + dspy endpoints via `xls2ddl.xls2crud`. ## View Type Determination -- **Tree View**: Use when table has a self-referencing foreign key (parent-child relationship where one field points to another record's id in the same table) -- **List View**: Use for all other tables without hierarchical relationships +- **Tree View**: table has a self-referencing foreign key (one field points to another record's id in the same table) +- **List View**: all other tables (no hierarchical relationships) ## Common Root Properties (Both View Types) ```json { - "tblname": "table_name", // Required: Actual table name - "alias": "optional_alias", // Optional: Used to create multiple CRUD interfaces for same table - "title": "Display Title", // Optional: If omitted, uses table title from table definition - "params": { ... } // Required: View-specific parameters + "tblname": "table_name", // Required: actual table name + "alias": "optional_alias", // Optional: multiple CRUD interfaces for same table + "title": "Display Title", // Optional: defaults to table title + "params": { ... } // Required: view-specific parameters } ``` ## List View CRUD Specification - ### Complete Structure ```json { @@ -48,284 +47,170 @@ This skill defines the standardized JSON format for CRUD (Create, Read, Update, }, "confidential_fields": ["field1", "field2"], "editor": { - "binds": [ - { - "wid": "source_field_id", // Required: Source widget ID (field name) - "event": "changed", // Required: Event type (typically "changed") - "actiontype": "script", // Required: Action type ("script" for JS) - "target": "target_field_id", // Required: Target widget ID - "script": "// JavaScript code" // Required: JS script content - } - ] + "binds": [{ + "wid": "source_field_id", // Required: source widget ID (field name) + "event": "changed", // Required: event type (typically "changed") + "actiontype": "script", // Required: action type ("script" for JS) + "target": "target_field_id", // Required: target widget ID + "script": "// JS code" // Required: JS script content + }] }, "browserfields": { - "exclouded": ["id"], // Optional: Fields to exclude from display + "exclouded": ["id"], // Optional: fields excluded from display "alters": { "field_name": { - "uitype": "code", // Required: UI type ("code" for dropdown/select) - // OR use dataurl approach: - // "dataurl": "api/endpoint", - // "datamethod": "GET", - // "dataparams": {"param": "value"}, - "data": [ // Required when uitype="code": Option data - { - "value": "v1", // Required: Actual stored value - "text": "Display Text" // Required: Display text for option - } - ] + "uitype": "code", // Required: "code" = dropdown/select + "data": [{"value": "v1", "text": "Display Text"}], // Required for uitype="code" (static) + // OR dynamic: "dataurl": "api/endpoint", "datamethod": "GET", "dataparams": {"param": "value"} } } }, - "editexclouded": ["readonly_field"], // Optional: Fields excluded from edit forms - "subtables": [ // Optional: Foreign key relationships - { - "field": "foreign_key_field", // Required: Foreign key field name - "title": "Subtable Title", // Optional: Uses subtable title if omitted - "url": "{{entire_url(subtable_alias)}}", // Required when alias defined - "subtable": "related_table_name" // Required: Related table name - } - ] + "editexclouded": ["readonly_field"], // Optional: fields excluded from edit form + "subtables": [{ // Optional: foreign key relationships + "field": "foreign_key_field", // Required: FK field name + "title": "Subtable Title", // Optional: defaults to subtable title + "url": "{{entire_url(subtable_alias)}}", // Required when alias defined + "subtable": "related_table_name" // Required: related table name + }] } } ``` ## Tree View CRUD Specification - ### Complete Structure ```json { "tblname": "hierarchical_table", - "alias": "optional_alias", - "uitype": "tree", // Required: Must be "tree" for tree view + "alias": "optional_alias", + "uitype": "tree", // Required: must be "tree" for tree view "title": "Display Title", "params": { - "idField": "id", // Required: Node ID field (typically "id") - "textField": "display_field", // Required: Field used for node display text - "sortby": ["field1 desc", "field2"], // Optional: Sort fields for tree nodes - "confidential_fields": ["field1", "field2"], // Optional: Sensitive field names - "browserfields": { - "alters": {} // Optional: Field attribute modifications - }, - "logined_userorgid": "org_id_field", // Optional: Organization filtering field - "logined_userid": "user_id_field", // Optional: User filtering field - "editable": true, // Required: true=editable, false=read-only - "edit_exclouded_fields": ["system_field"], // Optional: Fields excluded from editing - "parentField": "parent_id", // Required: Field containing parent node reference - "subtables": [ // Optional: Foreign key relationships - { - "field": "foreign_key_field", - "title": "Subtable Title", - "url": "{{entire_url(subtable_alias)}}", - "subtable": "related_table_name" - } - ] + "idField": "id", // Required: node ID field + "textField": "display_field", // Required: node display text field + "sortby": ["field1 desc", "field2"], // Optional: sort fields for tree nodes + "confidential_fields": ["field1", "field2"], // Optional: sensitive fields + "browserfields": {"alters": {}}, // Optional: field attribute modifications + "logined_userorgid": "org_id_field", // Optional: org filtering field + "logined_userid": "user_id_field", // Optional: user filtering field + "editable": true, // Required: true=editable, false=read-only + "edit_exclouded_fields": ["system_field"], // Optional: excluded from editing + "parentField": "parent_id", // Required: parent node reference field + "subtables": [...] // Optional: same structure as list view } } ``` -### Filter/Search Integration (data_filter) - -CRUD definitions can include a `data_filter` field in `params` to enable search/filter UI. The filter definition follows the `sqlor/filter.py` DBFilter JSON format: - +## Filter/Search Integration (data_filter) +`data_filter` in `params` follows the `sqlor/filter.py` DBFilter JSON format: ```json -{ - "tblname": "llm", - "params": { - "data_filter": { - "AND": [ - {"field": "model", "op": "LIKE", "var": "model_input"}, - {"field": "ppid", "op": "=", "var": "ppid_input"}, - {"field": "status", "op": "=", "const": "1"} - ] - } - } -} +{"tblname": "llm", "params": {"data_filter": {"AND": [ + {"field": "model", "op": "LIKE", "var": "model_input"}, + {"field": "ppid", "op": "=", "var": "ppid_input"}, + {"field": "status", "op": "=", "const": "1"} +]}}} ``` - -**Frontend behavior (bricks-framework DataViewer):** -When a CRUD JSON has `data_filter` in `params`, the DataViewer automatically adds a "搜索" button to the toolbar. Clicking it opens a `PopupWindow` containing a `Form` widget. The form fields are dynamically generated from the `data_filter` definition: -- Each `var` in the filter tree becomes a form input field -- Fields with `browserfields.alters[field].uitype == "code"` render as dropdowns -- Custom labels via `filter_labels: { "var_name": "显示名" }` in params -- Custom button text via `filter_label: "自定义按钮名"` in params -- Custom popup title via `filter_title: "自定义标题"` in params -- **No inline search form is rendered** — the filter UI is exclusively triggered by the toolbar button - -Form submit collects user values → sends `data_filter` (JSON string) + each `var` value as URL params → backend `.dspy` uses `DBFilter.gen(ns)` for SQL WHERE clause. +**Frontend behavior**: With `data_filter` present, DataViewer adds a "搜索" toolbar button → PopupWindow with a Form. Fields are generated from the filter tree: each `var` becomes a form input; fields with `browserfields.alters[field].uitype == "code"` render as dropdowns. Customize via `filter_labels: {"var_name": "显示名"}` (labels), `filter_label` (button text), `filter_title` (popup title). **No inline search form is rendered** — filter UI is exclusively toolbar-triggered. Form submit sends `data_filter` (JSON string) + each `var` value as URL params → backend `.dspy` uses `DBFilter.gen(ns)` for the SQL WHERE clause. **Key rules:** -- `var`: parameter name — value comes from user input in the popup form -- `const`: hardcoded value — does not require user input, does NOT generate a form field -- `op`: SQL operator — supported: `=`, `!=`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `LIKE`, `NOT LIKE`, `IS NULL`, `IS NOT NULL` -- Logical operators: `AND` (array, length ≥ 2), `OR` (array, length ≥ 2), `NOT` (single dict) -- OR/AND can be nested -- `const` conditions are included in the sent `data_filter` JSON but do not generate form inputs -- Empty filter values are excluded from the request params (not sent as empty strings) +- `var`: parameter name — value from user input in the popup form +- `const`: hardcoded value — no user input, does NOT generate a form field +- `op` supported: `=`, `!=`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `LIKE`, `NOT LIKE`, `IS NULL`, `IS NOT NULL` +- Logical operators: `AND` (array, length ≥ 2), `OR` (array, length ≥ 2), `NOT` (single dict); OR/AND nestable +- `const` conditions are included in the sent `data_filter` JSON but generate no form inputs +- Empty filter values are excluded from request params (not sent as empty strings) ## File Management Requirements - -### Storage Location - All CRUD definition files **must** be stored in the `json/` directory of the module - Each table gets one or more JSON files (multiple if using aliases) - -### Naming Convention -- Filename format: `{table_name}.json` or `{alias}.json` -- Examples: - - Table `users` → `json/users.json` - - Alias `user_admin` for users table → `json/user_admin.json` +- Naming: `{table_name}.json` or `{alias}.json` — e.g. `users` → `json/users.json`; alias `user_admin` → `json/user_admin.json` ## Key Implementation Notes - -### Field Exclusion Patterns -- **browserfields.exclouded**: Hides fields in read-only/list view -- **editexclouded**: Hides fields in edit forms (list view) -- **edit_exclouded_fields**: Hides fields in edit forms (tree view) - -### Dynamic Data Loading -For dropdown/select fields, you can either: -1. **Static data**: Use `data` array with value/text pairs -2. **Dynamic data**: Use `dataurl`, `datamethod`, and `dataparams` properties -3. **Cross-module data**: When options come from another module's database, see `references/cross-module-dataurl.md` for the multi-DB query pattern - -### Event Binding -- Only supported in list view editor.binds -- Uses JavaScript for dynamic form behavior (e.g., cascading dropdowns) -- `wid` = source field, `target` = destination field - -### Security Considerations -- Always specify `confidential_fields` for sensitive data -- Use `logined_userorgid` and `logined_userid` for proper data isolation -- Set `editable: false` for read-only tree views when appropriate +- **Field exclusion**: `browserfields.exclouded` hides fields in read-only/list view; `editexclouded` hides in edit forms (list view); `edit_exclouded_fields` hides in edit forms (tree view) +- **Dynamic data loading**: static `data` array (value/text pairs) | dynamic `dataurl` + `datamethod` + `dataparams` | cross-module data via `references/cross-module-dataurl.md` (multi-DB query pattern) +- **Event binding**: only in list view `editor.binds`, uses JavaScript for dynamic form behavior (e.g. cascading dropdowns); `wid` = source field, `target` = destination field +- **Security**: always specify `confidential_fields` for sensitive data; use `logined_userorgid`/`logined_userid` for data isolation; set `editable: false` for read-only tree views ## Integration Requirements -- Works with `bricks-framework` for frontend rendering -- Integrates with `sqlor-database-module` for backend operations -- References table definitions from `models/` directory -- Follows module structure defined in `module-development-spec` skill -- Each `data_url` / `editable` URL in CRUD JSON must have a matching `.dspy` endpoint file — see `references/api-endpoint-patterns.md` -- CRUD files are generated from JSON definitions via `xls2ddl.xls2crud` — see `references/xls2crud-generation.md` +- Frontend: `bricks-framework`; backend: `sqlor-database-module`; references table definitions in `models/` +- Follows module structure from `module-development-spec` skill +- Each `data_url` / `editable` URL must have a matching `.dspy` endpoint — see `references/api-endpoint-patterns.md` +- CRUD files are generated from JSON via `xls2ddl.xls2crud` — see `references/xls2crud-generation.md` ## Validation Checklist - [ ] View type correctly chosen (tree vs list based on table relationships) - [ ] Required fields present for chosen view type -- [ ] `tblname` value exactly matches a table defined in table definition +- [ ] `tblname` exactly matches a table defined in table definition - [ ] `params` dict exists and is non-empty - [ ] `editable` paragraph exists with `new_data_url`, `update_data_url`, `delete_data_url` - [ ] `data_url` exists and points to a valid `.dspy` list endpoint - [ ] Every field in `browserfields.exclouded` exists in the model's field list - [ ] Every field in `browserfields.alters` keys exists in the model's field list - [ ] Every field in `editexclouded` / `edit_exclouded_fields` exists in the model's field list -- [ ] All NOT NULL DEFAULT columns that aren't user-editable are in `editexclouded` (prevents "cannot be null" errors on form submit) +- [ ] All NOT NULL DEFAULT columns that aren't user-editable are in `editexclouded` (prevents "cannot be null" on submit) - [ ] `alters` entries use `uitype: "code"` with `dataurl` (endpoint returns plain `[{value,text}]` array) or `data` array (static) -- [ ] **`alters` entries with `valueField`/`textField` MUST also have `uitype: "code"`** (without it, text mapping is silently ignored — see Pitfall 35) +- [ ] **`alters` entries with `valueField`/`textField` MUST also have `uitype: "code"`** (else text mapping silently ignored — see Pitfall 33) - [ ] `subtables[].url` uses `{{entire_url('../alias')}}` with `../` prefix, no `wwwroot` in path - [ ] `editor.binds[].actiontype` is one of: urlwidget, method, script, registerfunction, event - [ ] `entire_url()` arguments are quoted strings - [ ] No forbidden root keys: `tablename` (use `tblname`), `grid`, `form`, `name`, `type`, `components` - [ ] **No Jinja2 control blocks in CRUD JSON** (`{% if %}`, `{% for %}`). `{{entire_url(...)}}` in strings is OK. See `references/crud-json-rules.md`. - [ ] File stored in correct `json/` directory with `{table_name}.json` naming -- All referenced fields exist in table definition (`models/` directory) -- Security fields properly configured (`confidential_fields` + `browserfields.exclouded`) -- Subtable references are valid: `field` and `subtable` keys present, `subtable` value matches an existing table in `models/`, and a corresponding wwwroot directory or CRUD config exists for it -- Every `data_url` / `editable` URL has a matching `.dspy` endpoint file in `wwwroot/api/` -- If `data_filter` present, corresponding `.dspy` list endpoint uses `DBFilter` to parse it -- **CRUD endpoint audit**: Run `scripts/verify-crud-endpoints.py` from `~/repos/` to check all create/update/delete `.dspy` files for json.dumps wrapping, wrong return format, and sor.U argument count. Fix all failures before commit. +- [ ] All referenced fields exist in table definition (`models/` directory) +- [ ] Security fields properly configured (`confidential_fields` + `browserfields.exclouded`) +- [ ] Subtable refs valid: `field` and `subtable` keys present, `subtable` matches an existing table in `models/`, and a corresponding wwwroot directory or CRUD config exists +- [ ] Every `data_url` / `editable` URL has a matching `.dspy` endpoint file in `wwwroot/api/` +- [ ] If `data_filter` present, corresponding `.dspy` list endpoint uses `DBFilter` to parse it +- [ ] **CRUD endpoint audit**: Run `scripts/verify-crud-endpoints.py` from `~/repos/` to check all create/update/delete `.dspy` files for json.dumps wrapping, wrong return format, and sor.U argument count. Fix all failures before commit. + +## Forbidden Root Keys (DO NOT USE) +The following keys do not exist in the CRUD spec and will cause failures: +- `tablename` → use `tblname` +- `grid` → does not exist; CRUD files do NOT support `fields`/`joins`/`select_fields` (no SQL joins, no cross-table fields) — use `browserfields.exclouded` + `alters` +- `form` → does not exist — use `editexclouded` + `alters` with uitype +- `name`, `type`, `components` → not CRUD keys; such files are `.ui` files in `wwwroot/`, not CRUD JSON (see Pitfall 9) ## Common Pitfalls -### WRONG Format Patterns (DO NOT USE) -The following patterns are **not** part of the CRUD spec and will cause failures: - -```json -// WRONG - these keys do not exist in the spec -{ - "tablename": "...", // Should be "tblname" - "grid": { // "grid" key does not exist - "fields": [...], // Use browserfields.exclouded + alters instead - "joins": [...], // CRUD files do NOT support SQL joins - "select_fields": [...] // Cross-table fields are not allowed - }, - "form": { // "form" key does not exist - "fields": [ - { "widget": "text" } // Use editexclouded + alters with uitype instead - ] - } -} -``` - ### Pitfall 1: Root key is `tblname`, not `tablename` -- **Wrong**: `"tablename": "users"` -- **Correct**: `"tblname": "users"` +`"tablename": "users"` is invalid — must be `"tblname": "users"`. ### Pitfall 2: CRUD files reference ONLY the base table -CRUD definition files do NOT support SQL joins, select_fields, or cross-table field references. All fields referenced in `browserfields.exclouded`, `editexclouded`, and `alters` must exist in the table definition (`models/` directory) for the table specified in `tblname`. - -- **Wrong**: Referencing `contract_number` when `tblname` is `financial_vouchers` (that field is in the `contract` table) -- **Correct**: Only reference fields that exist in `financial_vouchers` table definition (e.g., `contract_id`, `voucher_number`, `amount`) +No SQL joins, `select_fields`, or cross-table field references. All fields in `browserfields.exclouded`, `editexclouded`, and `alters` must exist in the table definition (`models/`) for the table in `tblname`. Example: for `financial_vouchers`, reference only fields like `contract_id`, `voucher_number`, `amount` — never `contract_number` (that field is in the `contract` table). ### Pitfall 3: Dropdown fields use `alters` with `uitype: "code"` -Dropdown/select fields must be defined in `browserfields.alters`, not in a `form` section: +Dropdowns go in `browserfields.alters`, never a `form` section. +- Static: `"alters": {"status": {"uitype": "code", "data": [{"value": "1", "text": "Active"}, {"value": "0", "text": "Inactive"}]}}` +- Dynamic: `"providerid": {"uitype": "code", "dataurl": "{{entire_url('../api/get_organizations.dspy')}}"}` -**Static data (inline options):** -```json -"params": {"browserfields": {"alters": {"status": { - "uitype": "code", - "data": [{"value": "1", "text": "Active"}, {"value": "0", "text": "Inactive"}] -}}}} -``` - -**Dynamic data (API endpoint):** -```json -"params": {"browserfields": {"alters": {"providerid": { - "uitype": "code", - "dataurl": "{{entire_url('../api/get_organizations.dspy')}}" -}}}} -``` - -- `dataurl` — API endpoint URL (must use `{{entire_url('...')}}` with quoted string) -- The endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object -- **Prefer appcodes over inline `data` arrays** — fixed options should go into `appcodes`/`appcodes_kv` via model `codes` definitions. See `references/appcodes-pattern.md` for the full pattern. -- The endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object -- On error or empty data, return `[]` -- `data_field` is **deprecated** — it was part of an older nested-response pattern (`{"data": {"organizations": [...]}}`). Do not use it. -- `valueField` and `textField` are **NOT deprecated** — they are required when the data source returns keys other than `value`/`text`. See Pitfall 26 for details. - -**⚠️ Deprecated nested-response pattern (DO NOT USE):** -```json -// WRONG — data_field is deprecated (nested response wrapping) -"providerid": { - "uitype": "code", - "dataurl": "...", - "data_field": "organizations" // DEPRECATED — do not use -} -``` -The endpoint must NOT return `{"success": true, "data": {"organizations": [...]}}`. Bricks Form's UiCode parses the response directly as an array. Wrapped formats cause the dropdown to silently not render. +Rules: +- `dataurl` must use `{{entire_url('...')}}` with a quoted string +- Endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object; on error or empty data return `[]` +- **Prefer `appcodes`/`appcodes_kv` (model `codes` definitions) over inline `data` arrays** for fixed options — see `references/appcodes-pattern.md` +- `data_field` is **deprecated** (old nested-response pattern `{"data": {"organizations": [...]}}`) — DO NOT use; wrapped formats make the dropdown silently not render +- `valueField`/`textField` are **NOT deprecated** — required when the data source returns keys other than `value`/`text` (see Pitfall 34) ### Pitfall 4: Field hiding uses `exclouded`/`editexclouded`, not field-level `"hidden": true` -- **Wrong**: `{"name": "org_id", "widget": "hidden"}` -- **Correct**: `"editexclouded": ["org_id"]` (hides in edit form), `"browserfields": {"exclouded": ["org_id"]}` (hides in list view) +No `{"name": "org_id", "widget": "hidden"}`. Use `"editexclouded": ["org_id"]` (hides in edit form) and `"browserfields": {"exclouded": ["org_id"]}` (hides in list view). ### Pitfall 5: Always load this skill BEFORE creating/modifying CRUD files -Never guess the CRUD format. Load `crud-definition-spec` first, then follow the structure exactly. The user has zero tolerance for guessed or improvised formats — they expect strict adherence to the spec with production-ready output, not experimental or guessed formats. If you're unsure about any property name or structure, load this skill and follow the examples verbatim. +Never guess the CRUD format — the user has zero tolerance for guessed/improvised formats. If unsure about any property, load this skill and follow the examples verbatim. ### Pitfall 6: Confidential fields should be hidden in browser view -Sensitive fields like API keys, passwords, or secret tokens should be listed in both `confidential_fields` and `browserfields.exclouded`. The `confidential_fields` array triggers server-side redaction, while `exclouded` removes them from the browser grid entirely. - -### Pitfall 7: Every CRUD JSON file MUST have an `editable` paragraph -Every CRUD JSON definition file — including list-only views — must include the `editable` paragraph with `new_data_url`, `update_data_url`, and `delete_data_url`. Without it, the framework cannot process form submissions. +Sensitive fields (API keys, passwords, tokens) must be in BOTH `confidential_fields` (server-side redaction) AND `browserfields.exclouded` (removed from the grid). +### Pitfall 7: `editable` paragraph required for ALL CRUD files (even list-only views) +Every CRUD JSON file — including list-only views — MUST include `editable` with `new_data_url`, `update_data_url`, `delete_data_url` (optionally `get_data_url`). Without it the framework cannot process form submissions. URLs use `{{entire_url('../api/xxx.dspy')}}` format with `../` prefix: ```json -"params": { - "editable": { - "new_data_url": "{{entire_url('../api/table_create.dspy')}}", - "update_data_url": "{{entire_url('../api/table_update.dspy')}}", - "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}" - } -} +"params": {"editable": { + "new_data_url": "{{entire_url('../api/table_create.dspy')}}", + "update_data_url": "{{entire_url('../api/table_update.dspy')}}", + "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}" +}} ``` +It must be an **object** — never the string `"default"` (see Pitfall 41). -### Pitfall 8: Field references must match model definitions exactly -All field names in `browserfields.exclouded`, `browserfields.alters`, and `editexclouded` must exactly match field names defined in the table's model JSON (`models/` directory). Common mismatches: +### Pitfall 8: Field references MUST match model definitions exactly +All names in `browserfields.exclouded`, `browserfields.alters` keys, and `editexclouded` must be exact matches to fields in the table's model JSON (`models/`). Even a one-character difference fails. Common mismatches: | Wrong Field | Correct Field | |-------------|---------------| @@ -335,367 +220,121 @@ All field names in `browserfields.exclouded`, `browserfields.alters`, and `edite | `is_active` | `is_won_stage` / `is_lost_stage` | | `changed_by` | `changed_by_id` / `changed_by_name` | -### Pitfall 9: Non-CRUD format files must not be placed in `json/` directory -Files using custom structures like `{"name": "...", "title": "...", "type": "page", "components": [...]}` are NOT CRUD definitions and will cause framework failures. These should be `.ui` files in `wwwroot/`, not JSON files in `json/`. +### Pitfall 9: Non-CRUD files must not be placed in `json/` directory +Files with custom structures (`{"name": ..., "title": ..., "type": "page", "components": [...]}`) are NOT CRUD definitions and cause framework failures. They belong as `.ui` files in `wwwroot/`. -### Pitfall 9: Validate ALL json/ files when touching any file in the directory -The user has zero tolerance for non-CRUD files being left in or added to the `json/` directory. When the task involves any CRUD file modification, you MUST scan every `.json` file in that directory against this spec — not just the files directly involved in the task. - -### Pitfall 10: ALL field references MUST exactly match model field names -Every field name used in `browserfields.exclouded`, `browserfields.alters`, and `editexclouded` MUST be an exact match to a field defined in the table's model JSON (`models/` directory). Even a one-character difference will cause failures. - -- **Wrong**: `alters` key `"sales_stage"` when the model field is `"current_stage"` -- **Wrong**: `exclouded` includes `"org_id"` when that field doesn't exist in the model -- **Wrong**: `alters` key `"is_active"` when the model has `"is_won_stage"` and `"is_lost_stage"` but no `"is_active"` -- **Correct**: Use only names that appear in the model's `fields` array +### Pitfall 10: Validate ALL json/ files when touching any file in the directory +When the task involves any CRUD file modification, scan EVERY `.json` file in `json/` against this spec — not just the files directly involved. Zero tolerance for non-CRUD files left in or added to `json/`. ### Pitfall 11: ID values must use `appPublic.uniqueID.getID()`, not `uuid.uuid4()` -Database `id` columns are typically VARCHAR(32). `uuid.uuid4().replace('-', '')` produces a 32-char hex string that may exceed the column length depending on the database encoding. Always use: - +Database `id` columns are typically VARCHAR(32); `uuid.uuid4().replace('-', '')` may exceed the column length. Always use: ```python from appPublic.uniqueID import getID new_id = getID() ``` +Applies to both `.dspy` API files and Python backend code. -This applies to both `.dspy` API files and Python backend code. - -### Pitfall 14: Hand-written `get_*_list.dspy` SHADOWS framework auto-generated list endpoints -When a module has CRUD JSON definitions in `json/` (e.g., `rl_vendor_config_list.json`), the Sage CRUD framework **automatically generates** list endpoints. If you also have a hand-written `wwwroot/api/get_{table}_list.dspy`, it **shadows** (overrides) the framework-generated one. This causes: -- **500 errors**: hand-written code may apply filters (e.g., `org_id`) on fields that don't exist in the table -- **403 errors**: hand-written files bypass the framework's RBAC and `logined_userorgid` handling -- **Silent data leakage**: hand-written code misses `confidential_fields` redaction - -**Rule**: If `json/{table}_list.json` exists, do NOT create `wwwroot/api/get_{table}_list.dspy`. The framework handles list queries from the JSON definition. Only create hand-written `.dspy` for custom business logic endpoints (create, update, delete, client-specific actions). - -**How to detect**: If a list endpoint returns 500 or 403, check: -1. Does `json/{alias}.json` exist for this table? -2. Does `wwwroot/api/get_{table}_list.dspy` also exist? -3. If both exist → DELETE the hand-written dspy, the framework auto-generates it - -### Pitfall 15: `editable` section is required for ALL CRUD files (even list-only views) -Every CRUD JSON file in the `json/` directory MUST have an `editable` paragraph with `new_data_url`, `update_data_url`, and `delete_data_url`. This is not optional — even files that only display lists need it, because the framework expects it for form submission handling. The URLs must use `{{entire_url('../api/xxx.dspy')}}` format with `../` prefix. +### Pitfall 12: Hand-written `get_*_list.dspy` SHADOWS framework auto-generated list endpoints +When `json/{table}_list.json` exists, the CRUD framework auto-generates the list endpoint. A hand-written `wwwroot/api/get_{table}_list.dspy` shadows it → **500 errors** (filters on nonexistent fields), **403 errors** (bypasses RBAC and `logined_userorgid` handling), **silent data leakage** (misses `confidential_fields` redaction). +**Rule**: if `json/{table}_list.json` exists, do NOT create `get_{table}_list.dspy`. Only write hand-written dspy for custom business endpoints (create, update, delete, client-specific actions). +**Detect**: list endpoint 500/403 → (1) does `json/{alias}.json` exist? (2) does `wwwroot/api/get_{table}_list.dspy` also exist? (3) if both → DELETE the hand-written dspy. ### Pitfall 13: `subtables[].subtable` must reference an existing table with accessible UI -The `subtables` section references a related table's CRUD UI. The value of `subtable` must be a real table name that has: -1. A table definition file in `models/` directory -2. A corresponding wwwroot directory (or at minimum, the CRUD is properly configured) - -A phantom reference to a non-existent table (e.g., `"subtable": "llmtype"` where `llmtype` has no model or wwwroot) causes the UI to break silently — the subtable tab renders but shows no data or errors. - -**CRITICAL: Explicit `url` when default path lacks RBAC permissions.** The framework generates a default path like `/module/subtable_name` for subtable UIs. If that path is NOT registered in RBAC permissions, the subtab will return 401. Solution: add an explicit `url` in the subtables entry pointing to a `.ui` file that already exists in wwwroot: - +The `subtable` value needs: (1) a table definition in `models/`, (2) a corresponding wwwroot directory (or properly configured CRUD). A phantom reference (e.g. `"subtable": "llmtype"` with no model/wwwroot) breaks the UI silently — the tab renders but shows no data. +**CRITICAL — explicit `url` when default path lacks RBAC permissions**: the framework's default path `/module/subtable_name` may not be registered in RBAC → subtab returns 401. Always set an explicit `url` to a `.ui` file that already exists: ```json -"subtables": [ - { - "field": "llmid", - "title": "能力映射", - "url": "{{entire_url('./llm_api_map_manage.ui')}}", - "subtable": "llm_api_map" - } -] +"subtables": [{"field": "llmid", "title": "能力映射", + "url": "{{entire_url('./llm_api_map_manage.ui')}}", "subtable": "llm_api_map"}] ``` +Verify: `find wwwroot -type d` (CRUD dir for subtable), `ls models/.json`, `grep -r subtable_name wwwroot` (existing `.ui` files). **Default: always set `url` explicitly to a concrete `.ui` file** rather than relying on framework path generation. -This bypasses the framework's auto-generated path and uses a known-permitted route. Always check that the referenced `.ui` file actually exists before setting the URL. +### Pitfall 14: NEVER replace CRUD auto-generated endpoints with custom scripts +Auto-generated `get_{table}.dspy` handles RBAC, `logined_userorgid`, `confidential_fields` redaction, and DBFilter parsing correctly; custom scripts often violate `.dspy` conventions and create maintenance burden. **Wrong**: creating `wwwroot/api/llm_list.dspy` to replace `get_llm.dspy` just to add `_text` fields. **Correct**: keep the auto-generated endpoint and fix the `dataurl` API to return `[{field_name, field_name_text}]` instead. Only propose custom scripts for a genuine special requirement the framework cannot handle — and **discuss the approach first**. +Validate configs: `python references/validate_crud.py /json/ --model-dir /models/` (forbidden keys, missing required keys, nonexistent field refs, improper alters syntax). Module-wide audit: follow `references/crud-audit-procedure.md`. -**How to verify**: -1. `find /path/to/module/wwwroot -type d` — check if a CRUD directory exists for the subtable -2. `ls /path/to/module/models/.json` — confirm table definition exists -3. `grep -r "subtable_name" /path/to/module/wwwroot` — check for existing `.ui` files -4. If no CRUD directory exists but a standalone `.ui` file does (e.g., `xxx_manage.ui`), use `"url": "{{entire_url('./xxx_manage.ui')}}"` instead of relying on auto-generated path -5. **Default**: always set `url` explicitly to a concrete `.ui` file rather than depending on framework path generation — this avoids RBAC 401 errors entirely - -### Pitfall 19: NEVER replace CRUD auto-generated endpoints with custom scripts - -The CRUD framework automatically generates `get_{table}.dspy` endpoints from JSON definitions. These are **base framework functionality** — stable, tested, and maintained. Do NOT replace them with hand-written scripts unless there is a genuine special requirement. - -**Wrong approach**: Creating `wwwroot/api/llm_list.dspy` to replace `get_llm.dspy` just to add `_text` fields. - -**Correct approach**: Keep using the CRUD auto-generated `get_llm.dspy`, and fix the `dataurl` API to return `[{field_name, field_name_text}]` format instead. - -**Why this matters**: -- Base framework endpoints handle RBAC, `logined_userorgid`, `confidential_fields` redaction, and DBFilter parsing correctly -- Custom scripts often violate `.dspy` file conventions (imports, dict access patterns, etc.) -- Replacing stable framework code with custom scripts creates maintenance burden — "today this way, tomorrow that way" makes the system unmaintainable - -**When to propose custom scripts**: Only when there is a genuine special requirement that the CRUD framework cannot handle. Even then, **discuss the approach first** before implementing — get confirmation that the deviation is necessary and the proposed solution is acceptable. - -```bash -python references/validate_crud.py /json/ --model-dir /models/ -``` - -It checks for forbidden keys, missing required keys, field references that don't exist in the model, and improper alters syntax. - -### Systematic Module Audit -When auditing an entire module's CRUD configs (e.g., "check all supplychain CRUD"), follow `references/crud-audit-procedure.md` — it has a step-by-step checklist for cross-referencing json/*.json against models/*.json, verifying alters coverage, editexclouded completeness, dataurl existence, and data_filter format correctness. +### Pitfall 15: NEVER manually add `data_url` to CRUD JSON +The framework auto-generates `get_{table}.dspy`; adding `"data_url": "{{entire_url('../api/get_llm.dspy')}}"` overrides it with a non-existent path → 500. Omit `data_url` entirely. +**Exception**: only add `data_url` for a genuinely custom list endpoint whose `.dspy` file actually exists. For `_text` FK display, create `get_search_{fieldname}.dspy` (Pitfall 20) and reference it in `alters[field].dataurl` — do NOT modify the auto-generated list endpoint. ### Pitfall 16: Generated CRUD directories are NOT committed to git -When `xls2ddl.xls2crud` generates `wwwroot/
/` directories (containing index.ui, get/add/update/delete .dspy), these are auto-generated artifacts. They must NOT be committed to git. Add them to the module's `.gitignore`: -``` -# CRUD definition directories (auto-generated by Sage platform) -wwwroot/llm/ -wwwroot/llm_api_map/ -# ... one entry per generated table -``` -The `json/` CRUD definitions ARE committed (they are the source). The `wwwroot/
/` directories are regenerated from `json/` + `models/` via `xls2ddl.xls2crud`. See `references/xls2crud-generation.md` for the full generation workflow. +`xls2ddl.xls2crud` generates `wwwroot/
/` (index.ui, get/add/update/delete .dspy) — these are auto-generated artifacts and must be gitignored (one `.gitignore` entry per table). The `json/` CRUD definitions ARE committed (they are the source). See `references/xls2crud-generation.md`. -### Pitfall 17: Custom `data_url` in CRUD JSON requires xls2ddl template support -If a CRUD JSON specifies `"data_url": "{{entire_url('../api/custom_list.dspy')}}"` to override the default `get_
.dspy`, the xls2ddl template (`data_browser_tmpl`) must use `{% if data_url %}` (not `{% if get_data_url %}`). As of xls2ddl commit 9f9a60a this is fixed. If you see generated index.ui ignoring `data_url`, update xls2ddl. +### Pitfall 17: Custom `data_url` requires xls2ddl template support +For CRUD JSON `data_url` overrides to take effect, the xls2ddl template (`data_browser_tmpl`) must use `{% if data_url %}` (not `{% if get_data_url %}`). Fixed in xls2ddl commit `9f9a60a`. If the generated index.ui ignores `data_url`, update xls2ddl. ### Pitfall 18: Code-type (uitype: "code") field debugging — never blame bricks first - -**Core principle**: If a code-type field shows `undefined`, raw IDs, or wrong values, the problem is in the **data layer** (API response format, dataurl path, backend query) — NOT in bricks framework. All code-type inputs use the same bricks component; if it were a framework bug, ALL code fields would be affected, not just one specific field. - -**Symptom: `undefined` in filter dropdown options** -- Check: does the `dataurl` endpoint return `[{value, text}]`? This is the ONLY correct format. Any other key names will fail. -- Check: is the `dataurl` path correct? Relative paths like `../api/xxx.dspy` may resolve differently than expected. -- Check: does the endpoint throw silently (returns `[]` on error)? Look at the dspy code for try/except that swallows errors. - -**Symptom: raw IDs in grid cells (not human-readable names)** - -The **dataurl API** (the endpoint specified in `alters[field].dataurl`) must return `[{value, text}]` format using SQL aliases: - +If a code-type field shows `undefined`, raw IDs, or wrong values, the problem is in the **data layer** (API response format, dataurl path, backend query) — NOT bricks (all code fields share the same component; a framework bug would hit all of them). +- Filter dropdown `undefined` → check dataurl returns `[{value, text}]` (ONLY correct format); check the path resolves; check the endpoint isn't swallowing errors into `[]` +- Raw IDs in grid cells → the dataurl API must return `[{value, text}]` via SQL aliases: ```python -# CORRECT - use SQL aliases for value/text -orgs = await sor.sqlExe( - "select id as value, orgname as text from organization order by orgname", {} -) -return orgs # DictObject serializes correctly +orgs = await sor.sqlExe("select id as value, orgname as text from organization order by orgname", {}) +return orgs ``` +Do NOT edit the auto-generated `wwwroot/
/get_
.dspy` (regenerated from templates; it doesn't join reference tables). For grid `_text` resolution use the custom LEFT JOIN list endpoint — see Pitfall 47. -The bricks framework uses `value` for the ID and `text` for display in both filter dropdowns and grid cells. +### Pitfall 19: data_filter dropdown must have empty/default option — FIXED +Fixed in bricks commit `f8f02c6` (`get_filter_fields()`): DataViewer auto-injects `{value: '', text: ''}` as the first option for all code-type filter fields (both `data` and `dataurl` sources), unless data already contains an empty/null/undefined entry. No backend changes needed. Old symptom (if unfixed): user selects a value and cannot reset to "show all". -**DO NOT** edit the auto-generated `wwwroot/
/get_
.dspy` directly — it's regenerated from templates. If the grid still shows raw IDs after fixing the `dataurl` API, the auto-generated list endpoint is not doing code resolution (it doesn't join reference tables). See Pitfall 46 for the custom list endpoint pattern with explicit LEFT JOINs. - -### Pitfall 20: data_filter dropdown must have empty/default option — FIXED - -**Status**: Fixed in bricks commit `f8f02c6` (dataviewer.js `get_filter_fields()`). - -**What changed**: Bricks now auto-injects `{value: '', text: ''}` as first option for all code-type filter fields, unless data already contains an empty/null/undefined value entry. This covers both `data` (static) and `dataurl` (dynamic) sources. - -**Previously**: The `dataurl` endpoint had to manually add empty option. Now the framework handles it. No backend changes needed. - -**Symptom (if unfixed)**: User opens filter form, selects a value, cannot reset to "show all". - -### Pitfall 22: codes fields need dedicated `get_search_{fieldname}.dspy` for filter dropdowns - -**Critical**: The success path MUST prepend the "全部" option to query results. Common mistake: returning `orgs` directly without the empty-value option. - -Correct pattern: +### Pitfall 20: codes fields need dedicated `get_search_{fieldname}.dspy` for filter dropdowns +When a model's `codes` section defines a foreign key (e.g. `providerid` → `organization`), the `browserfields.alters` dataurl should point to a dedicated `get_search_{fieldname}.dspy` (single-purpose, includes the "全部" fallback), not the generic list endpoint: ```python -try: - async with get_sor_context(request._run_ns, 'rbac') as sor: - orgs = await sor.sqlExe( - "select id as value, orgname as text from organization order by orgname", - {} - ) - return json.dumps([{'value': '', 'text': '全部'}] + list(orgs), ensure_ascii=False) -except Exception as e: - debug(f'get_search_providerid error: {e}') - return json.dumps([{'value': '', 'text': '全部'}], ensure_ascii=False) -``` - -When a model's `codes` section defines a foreign key (e.g., `providerid` → `organization`), the CRUD `browserfields.alters` dataurl should point to a **dedicated search script** named `get_search_{fieldname}.dspy`, not the generic list endpoint. - -**Pattern:** -```python -# wwwroot/api/get_search_providerid.dspy result = [{'value': '', 'text': '全部'}] - try: async with get_sor_context(request._run_ns, 'rbac') as sor: - orgs = await sor.sqlExe( - "select id as value, orgname as text from organization order by orgname", {} - ) - # CRITICAL: prepend 全部 to results — do NOT just "return orgs" + orgs = await sor.sqlExe("select id as value, orgname as text from organization order by orgname", {}) return json.dumps([{'value': '', 'text': '全部'}] + list(orgs), ensure_ascii=False) except Exception as e: debug(f'get_search_providerid error: {e}') - return json.dumps(result, ensure_ascii=False) ``` - -**Key rules:** -- Return format: `[{value, text}]` — SQL aliases `id as value, name as text` -- On success: **prepend 全部 to query results** via `[{'value': '', 'text': '全部'}] + list(orgs)` — do NOT just `return orgs` (this drops the 全部 option, a common mistake) -- On error: return fallback `result` with only "全部" option -- Name convention: `get_search_{fieldname}.dspy` in `wwwroot/api/` -- Register in `load_path.py` -- **CRITICAL: Also register in DB `permission` + `rolepermission` tables** — `load_path.py` alone is NOT sufficient for new `api/*.dspy` endpoints. After deploying, run SQL on the target server: - ```sql - INSERT INTO permission (id, path) VALUES (REPLACE(UUID(),'-',''), '/module/api/endpoint.dspy'); - INSERT INTO rolepermission (id, roleid, permid) - SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/endpoint.dspy'; - ``` - The path MUST use `/module/api/xxx.dspy` (not `/module/xxx.dspy`) — files in `wwwroot/api/` are served at `/module/api/`. Without this DB registration, the endpoint returns `403 Forbidden` even if `load_path.py` is correct. The role should be `'logined'` for endpoints that any authenticated user can call. -- No imports allowed (json, get_sor_context, debug are pre-loaded) - -**Why dedicated scripts:** Generic list endpoints may serve multiple purposes with different formats. Search scripts are single-purpose and include the "全部" option as fallback. - -**Architecture note**: Edit-form dropdowns use `alters.dataurl`; filter dropdowns use model `codes` → `get_code.dspy`. Both must be updated when changing a field's data source. See `references/filter-vs-edit-dropdown.md`. - -### Pitfall 24: NOT NULL DEFAULT columns MUST be in `editexclouded` - -### Pitfall 38: `return json.dumps(result)` in hand-written dspy causes double-serialization — tree/list won't update - -**Symptom**: After adding/updating a record via CRUD form (especially tree view), the operation appears to succeed (no error), but the new/changed record does NOT appear in the UI. Refreshing the page shows the data IS in the database — it was inserted but the tree/list didn't refresh. - -**Root cause**: Hand-written create/update/delete `.dspy` files using `return json.dumps(result, ensure_ascii=False)` instead of `return result`. The dspy framework already JSON-serializes the return value. Double-serialization produces a JSON string instead of an object, which the bricks frontend cannot parse (expects `{widgettype: "Message", ...}`, receives `'{"widgettype": "Message", ...}'`). - -**Wrong**: -```python -# WRONG — dspy framework auto-serializes; this double-serializes -return json.dumps(result, ensure_ascii=False) +Key rules: +- Return `[{value, text}]` — SQL aliases `id as value, name as text` +- **CRITICAL: on success prepend 全部** via `[{'value': '', 'text': '全部'}] + list(orgs)` — do NOT just `return orgs` (drops the 全部 option) +- On error: return fallback with only 全部; no imports allowed (json, get_sor_context, debug are pre-loaded) +- Name: `get_search_{fieldname}.dspy` in `wwwroot/api/`; register in `load_path.py` +- **CRITICAL: also register in DB `permission` + `rolepermission` tables** — `load_path.py` alone is NOT sufficient for new `api/*.dspy` endpoints. After deploying, run on the target server: +```sql +INSERT INTO permission (id, path) VALUES (REPLACE(UUID(),'-',''), '/module/api/endpoint.dspy'); +INSERT INTO rolepermission (id, roleid, permid) + SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/endpoint.dspy'; ``` +Path MUST be `/module/api/xxx.dspy` (files in `wwwroot/api/` are served at `/module/api/`). Without DB registration → `403 Forbidden` even if load_path.py is correct. Use role `'logined'` for endpoints any authenticated user can call. +**Architecture note**: edit-form dropdowns use `alters.dataurl`; filter dropdowns use model `codes` → `get_code.dspy`. Both must be updated when changing a field's data source. See `references/filter-vs-edit-dropdown.md`. -**Correct**: -```python -# CORRECT — framework handles serialization -return result -``` - -**Also check**: Remove `import json` at the top of hand-written dspy files — `json` is pre-loaded and importing it triggers dspy compliance warnings. - -**Audit command**: -```bash -grep -rn "json.dumps" wwwroot/api/ --include='*.dspy' -``` - -**Applicability**: Hand-written `.dspy` files in `wwwroot/api/`. Auto-generated CRUD dspy files (in `wwwroot/
/`) use `return r` correctly via the template — only hand-written files have this issue. - -### Pitfall 24 (original): NOT NULL DEFAULT columns MUST be in `editexclouded` - -When a table column has `NOT NULL DEFAULT ` (e.g., `login_fail_count SMALLINT NOT NULL DEFAULT 0`, `created_at TIMESTAMP NOT NULL DEFAULT current_timestamp()`), and the CRUD form should NOT let users edit it, it **MUST** be listed in `editexclouded`. If omitted, the edit form renders an empty input for that field, and on submit the framework sends `NULL` for it — causing `(1048, "Column 'xxx' cannot be null")` errors. - -**Symptom**: Adding a new record via CRUD form fails with `Column 'login_fail_count' cannot be null` even though the DB column has a DEFAULT value. - -**Fix**: Add ALL non-user-editable NOT NULL columns to `editexclouded`: +### Pitfall 21: NOT NULL DEFAULT columns MUST be in `editexclouded` +If a column has `NOT NULL DEFAULT ` (e.g. `login_fail_count SMALLINT NOT NULL DEFAULT 0`, `created_at TIMESTAMP NOT NULL DEFAULT current_timestamp()`) and users should NOT edit it, it MUST be in `editexclouded`. Otherwise the edit form renders an empty input and submits NULL → `(1048, "Column 'xxx' cannot be null")`. ```json "editexclouded": ["id", "created_at", "login_fail_count", "last_login", "last_login_fail"] ``` +Common culprits: `DEFAULT current_timestamp()` timestamps, `DEFAULT 0` counters, `DEFAULT '0'` status columns. -**Common culprits**: timestamp columns with `DEFAULT current_timestamp()`, counter columns with `DEFAULT 0`, status columns with `DEFAULT '0'`. - -### Pitfall 25: `record_toolbar` pattern for state-change action buttons - -CRUD tables support per-row action buttons via `record_toolbar` in `params`. Use this for enable/disable, approve/reject, activate/deactivate, or any state-change operation on individual records. - -**Structure:** +### Pitfall 22: `record_toolbar` pattern for state-change action buttons +Per-row action buttons (enable/disable, approve/reject, activate/deactivate): ```json -"params": { - "record_toolbar": [ - { - "label": "启用", - "actiontype": "dspy", - "url": "/module/table/enable_record.dspy", - "options": { - "icon": "check", - "cwidth": 16, - "cheight": 9 - } - }, - { - "label": "禁用", - "actiontype": "dspy", - "url": "/module/table/disable_record.dspy", - "options": { - "icon": "block", - "cwidth": 16, - "cheight": 9 - } - } - ] -} +"params": {"record_toolbar": [{ + "label": "启用", "actiontype": "dspy", + "url": "/module/table/enable_record.dspy", + "options": {"icon": "check", "cwidth": 16, "cheight": 9} +}]} ``` - -**dspy handler pattern** (enable_user.dspy / disable_user.dspy): -```python -if not params_kw.get('id'): - return {"widgettype":"Error","options":{"title":"Error","message":"no record selected","cwidth":16,"cheight":9,"timeout":3}} - -dbname = get_module_dbname('module_name') -db = DBPools() -async with db.sqlorContext(dbname) as sor: - await sor.U('table_name', {'id': params_kw.id, 'status_field': 'new_value'}) - return {"widgettype":"Message","options":{"title":"Success","message":"record updated","cwidth":16,"cheight":9,"timeout":3}} -``` - -**Key rules:** -- Each button needs its own `.dspy` file in `wwwroot/` -- The dspy receives `params_kw.id` (the selected row's ID) -- Use `sor.U()` with a dict containing `id` + fields to update — do NOT pass a 3rd argument -- Register all dspy paths in `load_path.py` -- Use `cwidth`/`cheight` in options (not fixed px) - -**Global toolbar buttons (`toolbar.tools` + `binds`):** See `references/toolbar-tools-binds.md` for the top-level toolbar pattern — unlike `record_toolbar` (per-row), these sit above the list and require `selected_row: true` + a `binds` entry with `wid: "self"`. Covers urlwidget→PopupWindow, params_mapping, `${id}$` placeholder, DSPY return format, and the 403→load_path→restart flow. +dspy handler pattern: check `params_kw.get('id')` (return `{"widgettype":"Error",...}` if missing); then `db = DBPools(); async with db.sqlorContext(get_module_dbname('module_name')) as sor: await sor.U('table_name', {'id': params_kw.id, 'status_field': 'new_value'})` (dict with `id` + fields — do NOT pass a 3rd argument); return `{"widgettype":"Message",...}`. +Key rules: each button needs its own `.dspy` in `wwwroot/`; dspy receives `params_kw.id`; register all paths in `load_path.py`; use `cwidth`/`cheight` in options (not fixed px). +**Global toolbar buttons** (`toolbar.tools` + `binds`): see `references/toolbar-tools-binds.md` — sit above the list (unlike per-row `record_toolbar`), require `selected_row: true` + a `binds` entry with `wid: "self"`; covers urlwidget→PopupWindow, params_mapping, `${id}$` placeholder, DSPY return format, 403→load_path→restart flow. ### Pitfall 23: dspy files must NOT have import statements — use pre-loaded modules only +Every `.dspy` file is injected into a pre-built async function context. Pre-loaded (NEVER import): +- From ahserver `y_env`: `debug`, `exception`, `error`, `info`, `warning`, `critical`; `get_user`, `get_username`, `get_userorgid`, `get_userinfo`; `entire_url`, `i18n`, `redirect`, `clientinfo`, `terminalType` +- Globals (injected at compile time): `json`, `datetime` (datetime/date/timedelta), `time`, `DictObject`, `DBPools`, `get_sor_context`, `getID`, `curDateString`, `timestampstr`, `FileStorage`, `partial`, `params_kw`, `format_exc` +- **Only allowed import**: `from sqlor.filter import DBFilter` +Common violations (all WRONG): `import json`, `import time`, `import datetime`, `from appPublic.uniqueID import getID`, `from appPublic.log import debug`, `from appPublic.dictObject import DictObject`, `from appPublic.timeUtils import curDateString, timestampstr`, `from sqlor.dbpools import get_sor_context, DBPools`, `from functools import partial`, `from ahserver.filestorage import FileStorage`. +If a module-internal function is needed (e.g. `from llmage.utils import get_llmusage_by_id`), export it via the module's `load_XXX()` in `init.py` (`env.get_llmusage_by_id = get_llmusage_by_id`), then dspy calls it directly. +Audit before every commit touching .dspy: `grep -rn '^import \|^from ' wwwroot/ --include='*.dspy' | grep -v 'sqlor.filter'` — must return empty; delete any matching import line. -**Every `.dspy` file is injected into a pre-built async function context.** The following are already imported and available without any `import` statement: - -**From ahserver `y_env` (processorResource.py):** -- `debug`, `exception`, `error`, `info`, `warning`, `critical` (from appPublic.log) -- `get_user`, `get_username`, `get_userorgid`, `get_userinfo` -- `entire_url`, `i18n`, `redirect`, `clientinfo`, `terminalType` - -**Pre-loaded globals (injected at dspy compile time):** -- `json` (json.dumps, json.loads) -- `datetime` (datetime, date, timedelta) -- `time` (time.time, time.sleep) -- `DictObject` (from appPublic.dictObject) -- `DBPools`, `get_sor_context` (from sqlor.dbpools) -- `getID` (from appPublic.uniqueID) -- `curDateString`, `timestampstr` (from appPublic.timeUtils) -- `FileStorage` (from ahserver.filestorage) -- `partial` (from functools) -- `params_kw` (request params), `format_exc` (traceback) - -**Only allowed import**: `from sqlor.filter import DBFilter` (not pre-loaded). - -**Common violations:** -```python -# WRONG — all of these are pre-loaded, NEVER import them -import json -import time -import datetime -from appPublic.uniqueID import getID -from appPublic.log import debug -from appPublic.dictObject import DictObject -from appPublic.timeUtils import curDateString, timestampstr -from sqlor.dbpools import get_sor_context, DBPools -from functools import partial -from ahserver.filestorage import FileStorage -``` - -**If a module-internal function is needed** (e.g. `from llmage.utils import get_llmusage_by_id`), export it via the module's `load_XXX()` function in `init.py` instead: -```python -# llmage/init.py -from .utils import get_llmusage_by_id - -def load_llmage(): - env = ServerEnv() - env.get_llmusage_by_id = get_llmusage_by_id -``` -Then dspy files call it directly: `record = await get_llmusage_by_id(usage_id)`. - -**Audit command** (run before every commit touching .dspy files): -```bash -grep -rn '^import \|^from ' wwwroot/ --include='*.dspy' | grep -v 'sqlor.filter' -``` -Must return empty. Any match is a violation — delete the import line. - -### Pitfall 19: Never manually add `data_url` to CRUD JSON — framework auto-generates endpoints - -### Pitfall 37: Tabular edit sends `_text` suffix fields — MUST strip before sor.U/sor.C - -**Symptom**: Editing a row with code-type fields (e.g., `user_status`, `orgid`) saves successfully but the changed value does not persist. Or: the update silently fails with no error message. - -**Root cause**: Tabular's edit form collects ALL field data including `_text` suffix display columns (e.g., `user_status_text`, `orgid_text`, `sync_from_text`). These `_text` fields are NOT real DB columns. When `sor.U('table', ns)` receives them in the data dict, some sqlor implementations may fail silently or skip the update. - -**Fix**: In EVERY add and update `.dspy`, add cleanup after `ns = params_kw.copy()`: - +### Pitfall 24: Tabular edit sends `_text` suffix fields — MUST strip before sor.U/sor.C +Tabular's edit form collects ALL field data including `_text` display columns (`user_status_text`, `orgid_text`, `sync_from_text`) that are NOT real DB columns; passing them to `sor.U`/`sor.C` fails silently or skips the update. In EVERY add/update `.dspy` after `ns = params_kw.copy()`: ```python ns = params_kw.copy() -for k,v in ns.items(): +for k, v in ns.items(): if v == 'NaN' or v == 'null': ns[k] = None # remove _text suffix fields sent by Tabular (not real DB columns) @@ -703,804 +342,166 @@ for k in list(ns.keys()): if k.endswith('_text'): ns.pop(k, None) ``` +Affects ALL `add_*.dspy`/`update_*.dspy` generated by xls2crud, plus any custom form handler receiving Tabular/Form code-type fields. Detection: grep for `_text` in params_kw before sor.C/U calls. -**Affected files**: ALL `add_*.dspy` and `update_*.dspy` files generated by xls2crud. +### Pitfall 25: Delegate-pattern endpoints (return result from helper) are valid — do NOT flag as format errors +A `_create.dspy` / `_update.dspy` / `_delete.dspy` that just calls a helper and returns its result (`result = await create_marketing(request, params_kw); return result`) is VALID — the helper (in `init.py`) returns the correct widgettype format. Do NOT flag as `no_widgettype` errors. Only flag endpoints that construct their own return value (e.g. `{'success': True, ...}`) without widgettype. Detect: last line `return result` / `return json.loads(result)` + a `result = await helper(...)` line → delegate pattern, skip format checking. -**Also fix in hand-written dspy files**: Any custom form submit handler that receives data from a Tabular or Form with code-type fields. - -**Detection**: Grep for `_text` in params_kw before sor.C/U calls. If `_text` fields are present and not stripped, the update will fail. - -### Pitfall 47: Delegate-pattern endpoints (return result from helper) are valid — do NOT flag as format errors - -**Symptom**: A `_create.dspy` / `_update.dspy` / `_delete.dspy` file contains only a few lines that call a helper function and return its result: -```python -result = await create_marketing(request, params_kw) -return result -``` - -**Rule**: These are valid. The helper function (defined in `init.py` or elsewhere) is responsible for returning the correct widgettype format. Do NOT flag these as `no_widgettype` errors. Only flag endpoints that construct their own return value (e.g., `return {'success': True, ...}`) without widgettype. - -**How to detect**: If the last line is `return result` or `return json.loads(result)` and the file also has `result = await helper(...)`, it's a delegate pattern — skip format checking. - -### Pitfall 48: Model name variants (dot vs hyphen) must be covered in model_mappings - -**Cross-skill**: This pitfall affects `pricing-data-format` as well. - -**Symptom**: Pricing engine logs `{config_data=..., mismatched}` and raises `没有找到合适的定价` even though the pricing YAML looks correct. - -**Root cause**: LLM API responses use dotted version numbers (e.g., `doubao-seedance-2.0`) while pricing YAML entries use hyphenated versions (e.g., `doubao-seedance-2-0`). The `model_mappings` in the YAML only maps full version suffixes (e.g., `doubao-seedance-2-0-260128` → `doubao-seedance-2-0`) but does NOT map dot-variants. - -**Fix**: Add explicit mappings for all dot-variants: +### Pitfall 26: Model name variants (dot vs hyphen) must be covered in model_mappings +LLM API responses may use dotted versions (`doubao-seedance-2.0`) while pricing YAML entries use hyphens (`doubao-seedance-2-0`); `model_mappings` that only map full version suffixes miss the dot-variants → `{config_data=..., mismatched}` / `没有找到合适的定价`. Fix — map all dot-variants explicitly: ```yaml model_mappings: doubao-seedance-2.0: doubao-seedance-2-0 doubao-seedance-2.0-fast: doubao-seedance-2-0-fast doubao-seedance-2-0-260128: doubao-seedance-2-0 ``` +Detection: compare the incoming `model` field against pricing filter values; if they differ only by `.` vs `-` in version numbers, add the mapping. Cross-skill: also affects `pricing-data-format`. -**Detection**: Compare the `model` field in incoming usage data against the `model` filter values in pricing entries. If they differ only by `.` vs `-` in version numbers, add the mapping. - -**Proper fix workflow (do NOT edit generated files):** - -Since generated CRUD dspy files must never be edited directly (Pitfall 28), the correct approach is: - -1. Create custom dspy files in `wwwroot/api/` (e.g., `add_user.dspy`, `update_user.dspy`) -2. Copy the generated dspy logic, add `_text` cleanup + any other fixes -3. Point the CRUD JSON to the custom files: -```json -{ - "tblname": "users", - "params": { - "new_data_url": "{{entire_url('/module/api/add_user.dspy')}}", - "update_data_url": "{{entire_url('/module/api/update_user.dspy')}}", - ... - } -} -``` -4. Register the new paths in `sage/load_path.py` -5. Regenerate: `xls2crud -m models -o wwwroot json/
.json` - -The generated `index.ui` will pick up the custom URLs automatically. The generated dspy files remain untouched and can be regenerated at any time without losing fixes. - -### Pitfall 36: Tabular sends request params via `data_params`, NOT `params` - -**Symptom**: Tabular loads data but request parameters (e.g., `discountid`, parent record ID) are not sent to the data_url endpoint. The dspy receives no params and returns empty data. - -**Root cause**: bricks `DataViewer` (line 8-14) reads `this.opts.data_params` as the default request parameters — NOT `this.opts.params`: -```javascript -this.loader = new bricks.PageDataLoader({ - url:this.opts.data_url || this.opts.url, - params:this.opts.data_params, // ← data_params, not params - ... -}); -``` - -**Wrong**: +### Pitfall 27: Tabular sends request params via `data_params`, NOT `params` +bricks DataViewer reads `this.opts.data_params` as the default request parameters — NOT `this.opts.params` (which is why a Tabular gets no `discountid` and returns empty data). **Wrong**: `"params": {"discountid": "{{params_kw.discountid}}"}`. **Correct**: ```json "data_url": "...dspy", -"params": { - "discountid": "{{params_kw.discountid}}" -} +"data_params": {"discountid": "{{params_kw.discountid}}"} ``` +Applies to ALL hand-written Tabular widgets, including subtable pages; CRUD auto-generated files handle this correctly via the template. -**Correct**: +### Pitfall 28: data_filter conflicts with logined_userorgid — FIXED in xls2ddl +Fixed in xls2ddl commit `ebd4b4a`: the generated `get_
.dspy` now injects `logined_userorgid`/`logined_userid` conditions into `filterjson` before DBFilter processes it (appending `{'field': '', 'op': '=', 'var': '__logined_orgid__'}` + `ns['__logined_orgid__'] = userorgid`; same for userid). Ensures both work together. Old workaround (manual ownerid filter in dspy / const in data_filter) no longer needed. After updating xls2ddl, regenerate affected modules. + +### Pitfall 29: Subtables auto-population — use `field` + `mapping` to pass parent values +For a subtable's add form to receive the parent record's ID (e.g. supplier contract needs `supplier_id`): (1) add the field to the subtable's `editexclouded` so users don't edit it; (2) carry the value via `new_data_url`: ```json -"data_url": "...dspy", -"data_params": { - "discountid": "{{params_kw.discountid}}" -} +"editexclouded": ["id", "resellerid", "supplier_id"], +"editable": {"new_data_url": "{{entire_url('../api/create.dspy')}}?supplier_id={{params_kw.get('supplier_id','')}}"} ``` +The xls2crud `params_mapping.mapping` sends `parent.id → subtable.field` to the subtable page as URL params; the add form's `new_data_url` carries it to the dspy. -This applies to ALL hand-written Tabular widgets, including subtable pages. The CRUD auto-generated files handle this correctly via the template — only hand-written Tabular UIs need this fix. - -### Pitfall 19 (original): Never manually add `data_url` to CRUD JSON — framework auto-generates endpoints - -The CRUD framework automatically generates `get_{table}.dspy` endpoints from JSON definitions in the `json/` directory. Do NOT add `"data_url": "{{entire_url('../api/get_{table}.dspy')}}"` to the CRUD JSON — this overrides the auto-generated path and points to a non-existent file, causing 500 errors. - -**Wrong:** -```json -{ - "tblname": "llm", - "params": { - "data_url": "{{entire_url('../api/get_llm.dspy')}}", // WRONG — file doesn't exist - ... - } -} -``` - -**Correct:** Omit `data_url` entirely. The framework uses the auto-generated `wwwroot/{table}/get_{table}.dspy`. - -**Exception:** Only add `data_url` when you have a genuinely custom list endpoint with special logic that the framework cannot handle. Even then, ensure the target `.dspy` file actually exists. - -**Related:** If you need to add `_text` fields for foreign key display, create a `get_search_{fieldname}.dspy` endpoint (see Pitfall 22) and reference it in `alters[field].dataurl`. Do NOT try to modify the auto-generated list endpoint. - -### Pitfall 21: data_filter conflicts with logined_userorgid — FIXED in xls2ddl - -**Status**: Fixed in xls2ddl commit `ebd4b4a` (tmplspy `get_data_tmpl`). - -**What changed**: The generated `get_
.dspy` now injects `logined_userorgid`/`logined_userid` conditions into the `filterjson` object before DBFilter processes it. This ensures both work together correctly. - -**Fix mechanism** (in template): +### Pitfall 30: xls2ddl `json.dumps(true)` generates Python-invalid `true` — MUST wrap in `json.loads()` +Generated dspy fails `NameError: name 'true' is not defined` (JSON booleans like `"not_null": true` are valid JSON but not Python). Fix in xls2ddl `tmpls.py` — **CRITICAL: the `json.loads()` call MUST have single quotes around the Jinja2 expression** (without quotes it receives a dict → `TypeError: the JSON object must be str, bytes or bytearray, not dict`): ```python -# After default_filterjson fallback, before DBFilter: -{% if logined_userorgid or logined_userid %} -if filterjson: - if not isinstance(filterjson, dict) or 'AND' not in filterjson: - filterjson = {'AND': [filterjson] if filterjson else []} -{% if logined_userorgid %} - filterjson['AND'].append({'field': '{{logined_userorgid}}', 'op': '=', 'var': '__logined_orgid__'}) - ns['__logined_orgid__'] = userorgid -{% endif %} -{% if logined_userid %} - filterjson['AND'].append({'field': '{{logined_userid}}', 'op': '=', 'var': '__logined_uid__'}) - ns['__logined_uid__'] = userid -{% endif %} -{% endif %} -``` - -**Deployment**: After updating xls2ddl, regenerate affected modules: -```bash -cd && PYTHONPATH= python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json -``` - -**Previously**: Workaround was to manually add ownerid filter in dspy or use const in data_filter. No longer needed. - -### Pitfall 49: Subtables auto-population — use `field` + `mapping` to pass parent values - -When a subtable's add form needs the parent record's ID auto-populated (e.g., supplier contract needs `supplier_id`), the subtable definition's `field` already handles filtering. For the add form to receive the value: - -1. **Add the field to `editexclouded`** in the subtable's CRUD JSON — so users don't see/manually edit it -2. **Modify `new_data_url`** to carry the value: `new_data_url: "...dspy?supplier_id={{params_kw.get('supplier_id','')}}"` - -The xls2crud `params_mapping.mapping` sends `parent.id → subtable.field` to the subtable page as URL params. The add form's `new_data_url` then carries it to the dspy. - -Example: -```json -// Parent CRUD JSON (suppliers_list.json) -"subtables": [{ - "field": "supplier_id", // parent's id → subtable's supplier_id - "title": "供应商合同", - "url": "{{entire_url('../supply_contracts_list')}}", - "subtable": "supply_contracts" -}] - -// Subtable CRUD JSON (supply_contracts_list.json) -"editexclouded": ["id", "resellerid", "supplier_id", ...], -"editable": { - "new_data_url": "{{entire_url('../api/create.dspy')}}?supplier_id={{params_kw.get('supplier_id','')}}" -} -``` - -### Pitfall 40: xls2ddl `json.dumps(true)` generates Python-invalid `true` — MUST wrap in `json.loads()` - -**Symptom**: Generated `.dspy` files fail with `NameError: name 'true' is not defined. Did you mean: 'True'?`. The auto-generated `update_
.dspy` contains `"not_null": true` (JSON boolean) instead of `"not_null": True` (Python). - -**Root cause**: xls2ddl templates use `{{json.dumps(fields, ensure_ascii=False)}}` to inline Python data structures. JSON's `true`/`false` are valid JSON but not valid Python literals. When the generated code runs in Python context, `NameError` occurs. - -**Fix in xls2ddl `tmpls.py`** — wrap Python-context `json.dumps()` in `json.loads()` to convert JSON booleans back to Python. **CRITICAL: The `json.loads()` call MUST have single quotes around the Jinja2 expression.** Jinja2's `{{json.dumps(...)}}` embeds the result as Python literal (not a string), so without quotes `json.loads()` receives a dict and crashes with `TypeError: the JSON object must be str, bytes or bytearray, not dict`: - -```python -# BEFORE (broken — JSON booleans in Python code): -tblfields = {{json.dumps(fields, ensure_ascii=False)}} -filterjson = {{json.dumps(data_filter, ensure_ascii=False)}} - -# ALSO BROKEN — json.loads receives dict, not string: -tblfields = json.loads({{json.dumps(fields, ensure_ascii=False)}}) - -# CORRECT — quotes make it a string, json.loads parses true→True: +# WRONG: tblfields = {{json.dumps(fields, ensure_ascii=False)}} +# WRONG: tblfields = json.loads({{json.dumps(fields, ensure_ascii=False)}}) # dict, not string +# CORRECT: tblfields = json.loads('{{json.dumps(fields, ensure_ascii=False)}}') filterjson = json.loads('{{json.dumps(data_filter, ensure_ascii=False)}}') ns['sort'] = json.loads('{{json.dumps(sortby)}}') ``` +Apply ONLY to Python-context lines (tblfields, filterjson, sort arrays). JSON-context lines (browserfields, toolbar, binds) use `true`/`false` correctly and must NOT be wrapped. Commit: xls2ddl `af6006f`. -**Where to apply**: Only Python-context lines (tblfields, filterjson, sort arrays). JSON-context lines (browserfields, toolbar, binds) use `true`/`false` correctly and must NOT be wrapped. +### Pitfall 31: Post-insert re-query uses `primary`, not `pkey` — xls2ddl template bug +`OperationalError: (1054, "Unknown column 'None' in 'WHERE'")` with `SELECT * FROM
WHERE None = %s` after adding a record (xls2ddl `fc91486` added a post-insert re-query `... WHERE {{summary[0].pkey}} = ${id}$`; model JSONs use `primary` (array), not `pkey` → Jinja2 renders `None`). Fix: `{{summary[0].pkey}}` → `{{summary[0].primary[0]}}` in xls2ddl `tmpls.py`, then regenerate all modules. -**Commit reference**: xls2ddl `af6006f`. - -**Symptom**: After adding a record via CRUD-generated `new_*.dspy` / `add_*.dspy`, the operation fails with: -``` -OperationalError: (1054, "Unknown column 'None' in 'WHERE'") -markedSQL='SELECT * FROM
WHERE None = %s' -``` -The error occurs in `sor.sqlExe()` inside the generated dspy, on the post-insert re-query line. Affects ALL tables across ALL modules. - -**Root cause**: In xls2ddl `tmpls.py` `data_new_tmpl`, commit `fc91486` added a post-insert re-query: -```python -_new_rows = await sor.sqlExe("SELECT * FROM {{summary[0].name}} WHERE {{summary[0].pkey}} = ${id}$", {'id': id}) -``` -But model JSONs use `primary` (array), not `pkey`. Jinja2 resolves to Python `None`, rendering `${None}$` → `WHERE None = %s`. - -**Fix**: `{{summary[0].pkey}}` → `{{summary[0].primary[0]}}` in xls2ddl `tmpls.py`. Then regenerate all modules. - -### Pitfall 36: logined_userorgid generates `WHERE None = %s` — xls2ddl template bug - -**Symptom**: `OperationalError: (1054, "Unknown column 'None' in 'WHERE'")` with `SELECT * FROM table WHERE None = %s`. The auto-generated `get_*.dspy` substitutes Python `None` for the `logined_userorgid` field name when the user's orgid is not set. - -**Fix**: Update xls2ddl to commit `ebd4b4a` or later (fixes `get_data_tmpl` to properly inject `logined_userorgid`/`logined_userid` conditions into `filterjson`). Then regenerate all affected CRUD files: +### Pitfall 32: logined_userorgid generates `WHERE None = %s` — FIXED in xls2ddl +Same `(1054, "Unknown column 'None' in 'WHERE'")` symptom when `logined_userorgid` is configured but the user's orgid is not set. Fix: update xls2ddl to commit `ebd4b4a` or later, then regenerate ALL affected modules: ```bash cd ~/repos/xls2ddl && git pull cd ~/repos/ && PYTHONPATH=~/repos/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/*.json ``` -**Note**: Affects ALL modules using `logined_userorgid` or `logined_userid` in their CRUD JSON. Must regenerate every affected module. - -**Symptom**: Data API correctly returns `{fieldname}_text` columns (e.g., `llmid_text`, `userid_text`), CRUD alters has `valueField`/`textField` configured, but the list grid still shows raw IDs instead of human-readable names. - -**Root cause**: The `alters` entry is missing `"uitype": "code"`. Without it, the Tabular/DataViewer treats the field as plain text and ignores the textField mapping entirely. - -**Wrong**: +### Pitfall 33: `valueField`/`textField` in alters MUST also have `uitype: "code"` +Without `uitype: "code"` the framework treats the field as plain text and **silently discards the textField mapping** — grid shows raw IDs even though the API returns `{fieldname}_text` columns: ```json -"llmid": { - "valueField": "llmid", - "textField": "llmid_text" -} +// WRONG: {"llmid": {"valueField": "llmid", "textField": "llmid_text"}} +// CORRECT: +{"llmid": {"uitype": "code", "valueField": "llmid", "textField": "llmid_text"}} ``` +Applies to both hand-written `.ui` files and CRUD JSON `browserfields.alters`. See Pitfall 34 for the full valueField/textField pattern. -**Correct**: +### Pitfall 34: `valueField`/`textField` in alters apply to BOTH filter form AND edit form +`browserfields.alters` is a **single configuration** shared by the list grid, the filter/search form, and the add/edit form. Use valueField/textField when: the `get_search_*.dspy` returns `{fieldname, fieldname_text}` keys (not `{value, text}`), the list endpoint returns `{fieldname}_text` columns, and you need the stored value key to be the actual field name: ```json -"llmid": { +"providerid": { "uitype": "code", - "valueField": "llmid", - "textField": "llmid_text" + "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}", + "valueField": "providerid", + "textField": "providerid_text" } ``` +The dspy MUST return those exact keys: `select id as providerid, orgname as providerid_text from organization order by orgname`, and the "全部" fallback must use the same keys (`{'providerid': '', 'providerid_text': '全部'}`). +Key rules: valueField/textField must exactly match the endpoint's returned keys; the same pair is used in filter form AND edit form (they cannot differ); if the dspy returns `{value, text}`, do NOT set valueField/textField (framework defaults). Symptom if wrong: filter dropdown shows `undefined`/raw IDs; edit form sends wrong values; filter selection doesn't match stored data. -**Rule**: Whenever you use `valueField`/`textField` in `alters`, you MUST also include `"uitype": "code"`. The framework only activates the textField lookup for code-type fields. Without `uitype`, it renders the raw field value (the ID) and silently discards the text mapping. - -**Related**: This applies to both hand-written `.ui` files and CRUD JSON `browserfields.alters`. See Pitfall 26 for the full valueField/textField pattern. - -### Pitfall 26: `valueField`/`textField` in alters apply to BOTH filter form AND edit form - -CRUD JSON `browserfields.alters` is a **single configuration** shared by the list grid, the filter/search form, and the add/edit form. When you set `valueField`/`textField` on a code-type field, it affects ALL three contexts. - -**When to use valueField/textField:** -- The `get_search_*.dspy` returns `{fieldname, fieldname_text}` keys (not `{value, text}`) -- The main list endpoint also returns `{fieldname}_text` columns for grid display -- You need the stored value key to be the actual field name (e.g., `providerid`) rather than generic `value` - -**Example — code field with custom valueField/textField:** -```json -"alters": { - "providerid": { - "uitype": "code", - "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}", - "valueField": "providerid", - "textField": "providerid_text" - } -} -``` - -The `get_search_providerid.dspy` MUST return data with those exact keys: -```python -# CORRECT — keys match valueField/textField -rows = await sor.sqlExe( - "select id as providerid, orgname as providerid_text from organization order by orgname", {} -) -return json.dumps([{'providerid': '', 'providerid_text': '全部'}] + list(rows), ensure_ascii=False) -``` - -**Key rules:** -- `valueField`/`textField` must exactly match the keys returned by the `dataurl` endpoint -- The same `valueField`/`textField` is used in the filter form AND the edit form — they cannot differ -- If the dspy returns `{value, text}` (the simple pattern), do NOT set valueField/textField — the framework defaults to those -- The "全部" fallback option must also use the same keys (not `{value: '', text: '全部'}` when valueField is `providerid`) - -**Symptom if wrong:** Filter dropdown shows `undefined` or raw IDs; edit form sends wrong values; filter selection doesn't match stored data. - -### Pitfall 27: MUST regenerate index.ui on each server after modifying CRUD JSON alters - -The `wwwroot/
/` directory (containing `index.ui`) is auto-generated by `xls2crud` and is **gitignored** (see Pitfall 16). When you modify CRUD JSON `alters` (adding/changing `valueField`, `textField`, `dataurl`, `uitype`, etc.), the change only takes effect after re-running `xls2crud` on **each deployment environment**. - -**Common mistake:** Update JSON, commit and push, then `git pull` on the server — but `index.ui` is NOT in git, so the server still has the stale auto-generated UI. - -**Fix — run on each server after pull:** +### Pitfall 35: MUST regenerate index.ui on each server after modifying CRUD JSON alters +`wwwroot/
/` (index.ui) is auto-generated by `xls2crud` and **gitignored** (Pitfall 16) — `git pull` on a server leaves stale UI. After ANY change to `browserfields.alters` (dataurl/valueField/textField/uitype/data), `browserfields.exclouded`, `editexclouded`, `data_filter`, `filter_labels`, `filter_title`, `subtables`, `record_toolbar`, `editor.binds`, or model definitions in `models/`, re-run on EACH environment: ```bash cd /path/to/module PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` -**Triggers that require regeneration:** -- Any change to `browserfields.alters` (dataurl, valueField, textField, uitype, data) -- Any change to `browserfields.exclouded` or `editexclouded` -- Any change to `data_filter`, `filter_labels`, `filter_title` -- Any change to `subtables` -- Any change to `record_toolbar` -- Any change to `editor.binds` -- Any change to model definitions in `models/` directory - -### Pitfall 30: CRUD page not triggering data request — diagnose CRUD spec FIRST, not permissions - -**User correction**: When a CRUD list page loads but does NOT trigger the `get_
.dspy` data request, the root cause is in the CRUD specification or generated template — NOT in RBAC/permissions. Do NOT waste time checking `load_path.py` PATHS_ANY vs PATHS_LOGINED first. - -**Correct diagnostic order:** -1. **Check generated `index.ui`** — does it have a `data_url` field pointing to the correct endpoint? -2. **Check CRUD JSON** — does it have the required structure (tblname, params, editable)? -3. **Check `get_
.dspy`** — does the file exist in the CRUD directory? -4. **Compare with a working module** — diff against llmage/llm or pricing which are known to work -5. **Only then** check RBAC/load_path.py if all above are correct - -**Common causes (in order of frequency):** -- CRUD directory not regenerated after JSON changes (stale index.ui) -- `data_url` missing or incorrect in generated index.ui -- CRUD JSON missing required keys (editable, browserfields) -- Template version outdated (xls2ddl needs update) - -**Anti-pattern (DO NOT):** -- Immediately assume it's a permissions issue -- Move paths between PATHS_ANY and PATHS_LOGINED as first diagnostic step -- Blame the bricks framework before checking the CRUD config - -### Pitfall 31: `data_url` vs `get_data_url` in generated index.ui — two different mechanisms - -The xls2crud template (`data_browser_tmpl` in `tmpls.py`) generates TWO data-loading URLs with different purposes: - -1. **`data_url`** (outside `editable` block) — used by the Tabular/DataViewer component for **initial page load**. Points to `get_
.dspy` by default: - ``` - data_url: "{{entire_url('./get_suppliers.dspy')}}" - ``` - - If CRUD JSON has a `data_url` key, that value is used instead (custom endpoint) - - If CRUD JSON omits `data_url`, the template defaults to `./get_
.dspy` - -2. **`get_data_url`** (inside `editable` block, optional) — used to **override** the data URL with extra parameters (e.g., `?pagerows=50`). Only generated when the CRUD JSON explicitly defines it: - ```json - "editable": { - "get_data_url": "{{entire_url('get_llmusage.dspy')}}?pagerows=50", - "new_data_url": "...", - "update_data_url": "...", - "delete_data_url": "..." - } - ``` - -**Key rules:** -- Most CRUD pages only need `data_url` (auto-generated) — no `get_data_url` needed -- `get_data_url` is for special cases (pagination overrides, custom query params) -- The Tabular component uses `data_url` on initial render, then `get_data_url` for subsequent refreshes if defined -- If neither exists in the generated index.ui, the table renders empty with no network request - -### Pitfall 29: `data_filter` MUST use DBFilter tree structure, NOT `{"fields": [...]}` - -A common mistake is using a flat `fields` array instead of the required `AND`/`OR` tree: +### Pitfall 36: CRUD page not triggering data request — diagnose CRUD spec FIRST, not permissions +When a CRUD list page loads but does NOT trigger the `get_
.dspy` request, the root cause is the CRUD spec or generated template — NOT RBAC/permissions. Diagnostic order: (1) generated `index.ui` has a `data_url` pointing to the right endpoint; (2) CRUD JSON has required structure (tblname, params, editable); (3) `get_
.dspy` exists in the CRUD directory; (4) diff against a working module (e.g. llmage/llm, pricing); (5) only then check RBAC/load_path.py. +Common causes (frequency order): stale index.ui (not regenerated), data_url missing/incorrect in generated index.ui, CRUD JSON missing required keys, outdated xls2ddl template. **Anti-pattern**: assume permissions first, shuffle PATHS_ANY/PATHS_LOGINED, or blame bricks before checking the CRUD config. +### Pitfall 37: `data_url` vs `get_data_url` in generated index.ui — two different mechanisms +1. **`data_url`** (outside `editable` block) — initial page load by Tabular/DataViewer. Defaults to `./get_
.dspy`; overridden if CRUD JSON has a `data_url` key. +2. **`get_data_url`** (inside `editable` block, optional) — overrides the data URL with extra parameters (e.g. `?pagerows=50`). Only generated when the CRUD JSON explicitly defines it: ```json -// WRONG — this format does NOT work, search will silently fail -"data_filter": { - "fields": [ - {"field": "supplier_org_id", "title": "供应商", "uitype": "code"}, - {"field": "resource_type", "title": "资源类型", "uitype": "code"} - ] -} - -// CORRECT — DBFilter tree with op/var -"data_filter": { - "AND": [ - {"field": "supplier_org_id", "op": "=", "var": "supplier_org_id"}, - {"field": "resource_type", "op": "=", "var": "resource_type"} - ] -} +"editable": {"get_data_url": "{{entire_url('get_llmusage.dspy')}}?pagerows=50", "new_data_url": "...", ...} ``` +Most CRUD pages only need the auto `data_url`. Tabular uses `data_url` on initial render, then `get_data_url` for subsequent refreshes if defined. If neither exists in generated index.ui, the table renders empty with no network request. -**Symptom**: Search button renders but filtering does nothing, or search popup doesn't appear at all. -**Fix**: Replace `"fields": [...]` with `"AND": [{"field": "...", "op": "...", "var": "..."}]` and add a `"filter_labels"` object for Chinese labels. - -### Pitfall 28: NEVER directly edit auto-generated .ui files — always modify source configs - -**Critical architectural rule**: Files in `wwwroot/
/` (especially `index.ui`) are auto-generated by `xls2crud` from source configurations. **Never edit them directly**, even in production environments. - -**Wrong approach** (causes user frustration): -```bash -# WRONG — directly editing production file -sudo sed -i 's/old_pattern/new_pattern/g' /d/apitest/sage/wwwroot/discount/discount_setting/index.ui -``` - -**Correct approach**: -1. Find the source configuration: - - For CRUD-generated UIs: `json/
.json` (CRUD config) - - For hand-written UIs: `wwwroot/.ui` (directly editable) -2. Modify the source config -3. Regenerate (if CRUD): - ```bash - cd /path/to/module - PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json - ``` -4. Deploy the regenerated files - -**How to determine if a file is auto-generated:** -- Check if `json/
.json` exists → yes = auto-generated -- Check `wwwroot/
/` directory structure (has `index.ui`, `get_
.dspy`, etc.) → yes = auto-generated -- Files outside `wwwroot/
/` directories (e.g., `wwwroot/custom_feature.ui`) → hand-written, can edit directly - -### Pitfall 34: CRUD list performance — FOUR critical optimizations required - -**Symptom**: CRUD list page loads extremely slowly, queries take 3+ seconds, users complain about unacceptable performance even after initial fixes. - -**CRITICAL: Correct file location for custom list endpoints** - -The auto-generated `wwwroot/
/get_
.dspy` is **gitignored** and regenerated by `xls2crud`. DO NOT edit it — your changes will be lost on next regeneration. - -**Correct approach**: Write a custom list endpoint in `wwwroot/api/
_list.dspy` and point the CRUD JSON's `editable.get_data_url` to it: - +### Pitfall 38: `data_filter` MUST use DBFilter tree structure, NOT `{"fields": [...]}` +A flat `"fields": [{"field": ..., "title": ..., "uitype": "code"}]` silently fails (search does nothing / popup doesn't appear). Correct — DBFilter tree with op/var: ```json -// json/
.json -"params": { - "editable": { - "get_data_url": "{{entire_url('../api/
_list.dspy')}}?pagerows=50", - "new_data_url": "...", - "update_data_url": "...", - "delete_data_url": "..." - } -} +"data_filter": {"AND": [ + {"field": "supplier_org_id", "op": "=", "var": "supplier_org_id"}, + {"field": "resource_type", "op": "=", "var": "resource_type"} +]} ``` +Add a `"filter_labels"` object for Chinese labels. -This custom endpoint is committed to git and persists across regenerations. +### Pitfall 39: NEVER directly edit auto-generated .ui files — always modify source configs +Files in `wwwroot/
/` (especially index.ui) are auto-generated by `xls2crud` — never edit directly, even in production (e.g. no `sed -i` on production index.ui). Correct approach: (1) find the source config — `json/
.json` for CRUD-generated UIs, `wwwroot/.ui` for hand-written UIs; (2) modify the source; (3) regenerate (CRUD only, see Pitfall 35 command); (4) deploy. +How to tell if auto-generated: `json/
.json` exists → yes; directory contains `index.ui` + `get_
.dspy` → yes; files outside `wwwroot/
/` (e.g. `wwwroot/custom_feature.ui`) → hand-written, editable directly. -**Root causes (in order of impact):** +### Pitfall 40: CRUD list performance — FOUR critical optimizations required +Symptom: list page 3+ seconds. **File location**: never edit the gitignored, regenerated `wwwroot/
/get_
.dspy` — write `wwwroot/api/
_list.dspy` and point `editable.get_data_url` at it (committed, survives regeneration). Apply optimizations in order (each 2-5x; together 20-40x): +1. **`sqlPaging` wraps queries in subqueries (slowest)** → separate count + data queries: `select count(*) as cnt ...` then `select col1, col2 ... limit {rows_per_page} offset {(page-1)*rows_per_page}` with `page = int(ns.get('page', 1))`, `rows_per_page = int(ns.get('rows', ns.get('pagerows', 50)))` +2. **`default_filterjson` generates LIKE for ALL fields incl. large TEXT (full table scan)** → exclude TEXT columns: `filter_fields = [f['name'] for f in ori_fields if f['name'] not in ('usages', 'ioinfo')]` +3. **`SELECT *` pulls large TEXT/BLOB (off-page storage I/O)** → explicit column list +4. **DBFilter + ArgsConvert framework overhead** → bypass entirely with raw SQL + manual WHERE on 5-7 common filter fields (`conditions = ['1=1']`, append `'field=${var}$'` + `ns['var'] = value` only when value present), using `DBPools()` + `get_module_dbname('module_name')`; return `{'success': True, 'total': total, 'rows': rows, 'page': page, 'page_size': rows_per_page}`; wrap in try/except with `debug(format_exc())` and `{'success': False, ...}` fallback. +Real-world (llmage/llmusage, 17 columns): sqlPaging + all-fields filter + SELECT * → 8-12s; after #1: 3-4s; after #2: 1-2s; after #3: 0.8-1.5s; after #4: 0.2-0.5s. -**1. `sqlPaging` wraps queries in subqueries (slowest)** +### Pitfall 41: `"editable": "default"` string causes xls2ui serialization error +`"editable": "default"` (string instead of object) → `build.sh`/`xls2ui` fails with `TypeError: Object of type builtin_function_or_method is not JSON serializable` at `json.dumps(binds, ...)`. Fix: replace with the full object containing `get_data_url`, `new_data_url`, `update_data_url`, `delete_data_url` (the `"default"` shorthand is unsupported). This is a case of Pitfall 7 focused on *format* — must be an object, not a string. -The `sor.sqlPaging(sql, ns)` function wraps the data query in a subquery for count, which is extremely slow for large tables: -```python -# SLOW — sqlPaging generates: SELECT * FROM (SELECT ... WHERE ...) AS t -r = await sor.sqlPaging(sql, ns) -``` - -**Fast pattern — separate count and data queries:** -```python -# Separate count query -count_sql = f'select count(*) as cnt from tablename {where_clause}' -count_recs = await sor.sqlExe(count_sql, ns) -total = count_recs[0].cnt if count_recs else 0 - -# Separate data query with LIMIT/OFFSET -page = int(ns.get('page', 1)) -rows_per_page = int(ns.get('rows', ns.get('pagerows', 50))) -offset = (page - 1) * rows_per_page - -data_sql = f'''select col1, col2, ... from tablename {where_clause} -order by {ns.get('sort', 'use_time desc')} -limit {rows_per_page} offset {offset}''' -rows = await sor.sqlExe(data_sql, ns) - -return {'total': total, 'rows': rows if rows else []} -``` - -**2. `default_filterjson` generates LIKE filters for TEXT fields (full table scan)** - -When no filter is provided, `default_filterjson(fields, ns)` generates LIKE conditions for ALL fields, including large TEXT columns: - -```python -# SLOW — generates LIKE filters for TEXT fields -filterjson = default_filterjson(fields, ns) -``` - -**Fast pattern — exclude TEXT fields from filter generation:** -```python -filter_fields = [f['name'] for f in ori_fields if f['name'] not in ('usages', 'ioinfo')] -filterjson = default_filterjson(filter_fields, ns) -``` - -**3. SELECT * includes large TEXT/BLOB columns (I/O overhead)** - -```python -# SLOW — fetches 100+ KB per row from off-page storage -sql = "SELECT * FROM tablename WHERE ..." - -# FAST — explicit column list -sql = '''select id, col1, col2, ... from tablename where 1=1''' -``` - -**4. DBFilter + ArgsConvert framework overhead (bypass entirely for max performance)** - -When the above three optimizations still leave the query slow (2+ seconds), completely bypass the DBFilter/ArgsConvert framework and write raw SQL: - -```python -# In wwwroot/api/
_list.dspy — no imports needed -result = {'success': False, 'rows': [], 'total': 0, 'page': 1, 'page_size': 50} - -try: - page = int(params_kw.get('page', 1)) - rows_per_page = int(params_kw.get('rows', params_kw.get('pagerows', 50))) - offset = (page - 1) * rows_per_page - sort_field = params_kw.get('sort', 'use_time desc') - - # Manually build WHERE conditions (only common filter fields) - conditions = ['1=1'] - ns = {} - - llmid = params_kw.get('llmid') - if llmid: - conditions.append('llmid=${llmid}$') - ns['llmid'] = llmid - - status = params_kw.get('status') - if status: - conditions.append('status=${status}$') - ns['status'] = status - - # ... add other common filter fields as needed ... - - where = 'WHERE ' + ' AND '.join(conditions) - - # List fields (exclude large TEXT columns) - select_fields = 'id, col1, col2, ...' - - db = DBPools() - dbname = get_module_dbname('module_name') - - async with db.sqlorContext(dbname) as sor: - count_recs = await sor.sqlExe(f'SELECT count(*) as cnt FROM tablename {where}', ns) - total = count_recs[0].cnt if count_recs else 0 - - rows = await sor.sqlExe( - f'SELECT {select_fields} FROM tablename {where} ORDER BY {sort_field} LIMIT {rows_per_page} OFFSET {offset}', - ns - ) - - result['success'] = True - result['total'] = total - result['rows'] = rows if rows else [] - result['page'] = page - result['page_size'] = rows_per_page - -except Exception as e: - debug(f'
_list error: {format_exc()}') - result['error'] = str(e) - -return json.dumps(result, ensure_ascii=False, default=str) -``` - -**Why bypassing DBFilter matters:** -- DBFilter generates filter conditions for ALL fields in `ori_fields`, even when the user didn't provide values -- ArgsConvert template processing adds overhead for every request -- Manual WHERE clause building with only 5-7 common filter fields is 3-5x faster than full framework processing - -**Performance comparison (real-world example — llmage/llmusage, 17 columns):** -- Original: `sqlPaging` + `default_filterjson` on all fields + `SELECT *` → 8-12 seconds -- After fix #1 (separate queries): 3-4 seconds -- After fix #2 (exclude TEXT from filter): 1-2 seconds -- After fix #3 (explicit columns): 0.8-1.5 seconds -- After fix #4 (bypass DBFilter entirely): 0.2-0.5 seconds -- Total: 20-40x improvement - -**How to identify the problem:** -1. Check if `get_
.dspy` uses `sqlPaging` → replace with separate queries -2. Check if `default_filterjson` includes TEXT/BLOB fields → exclude them -3. Check if query uses `SELECT *` → replace with explicit column list -4. If still slow (>1s), bypass DBFilter entirely in custom `api/
_list.dspy` - -**Rule**: When a CRUD list query is slow, apply optimizations in order: (1) separate count/data queries, (2) exclude TEXT fields from filter, (3) explicit column list, (4) bypass DBFilter entirely. Each provides 2-5x improvement; together they provide 20-40x improvement. - -### Pitfall 33: `"editable": "default"` string causes xls2ui serialization error - -**Symptom**: Running `build.sh` (which calls `xls2ui`) fails with: -``` -TypeError: Object of type builtin_function_or_method is not JSON serializable -``` -at `json.dumps(binds, indent=4, ensure_ascii=False)`. - -**Root cause**: The CRUD JSON has `"editable": "default"` as a **string** instead of an object. The framework internally tries to generate binds configuration from this shorthand, and some fields end up being set to function references instead of strings, causing JSON serialization to fail. - -**Wrong**: -```json -{ - "tblname": "pricing_program", - "params": { - "editable": "default", // WRONG — causes serialization error - "sortby": "name" - } -} -``` - -**Correct**: -```json -{ - "tblname": "pricing_program", - "params": { - "editable": { - "get_data_url": "{{entire_url('get_pricing_program.dspy')}}", - "new_data_url": "{{entire_url('../api/pricing_program_create.dspy')}}", - "update_data_url": "{{entire_url('../api/pricing_program_update.dspy')}}", - "delete_data_url": "{{entire_url('../api/pricing_program_delete.dspy')}}" - }, - "sortby": "name" - } -} -``` - -**Fix**: Replace `"editable": "default"` with the full object containing all four URL keys (`get_data_url`, `new_data_url`, `update_data_url`, `delete_data_url`). The `"default"` shorthand is not supported by the framework. - -**Related**: This is a specific case of Pitfall 7 and Pitfall 15 (editable section required). The difference is that those pitfalls focus on the *presence* of editable; this pitfall focuses on the *format* — it must be an object, not a string. - -### Pitfall 32: Field title renaming in model JSON affects ALL CRUD views - -When the user asks to rename field display titles (e.g., "用户id" → "username", "模型机构" → "orgname"), modify the `title` field in `models/
.json`, NOT in the CRUD JSON or generated .ui files. - -**Why**: The model's `fields[].title` is the single source of truth for column headers. CRUD auto-generation reads from the model and propagates titles to: -- List grid column headers -- Filter form labels -- Edit form field labels -- Add form field labels - -**Correct approach**: -```json -// models/llmusage.json -{ - "name": "userid", - "title": "username", // Changed from "用户id" - "type": "str", - "length": 32 -} -``` - -**Wrong approaches**: -- Editing `json/
.json` browserfields — titles there are for overrides, not primary labels -- Editing generated `wwwroot/
/index.ui` — this is a build artifact -- Editing `wwwroot/api/get_
.dspy` — titles are not in the query layer - -**After renaming**: Regenerate CRUD files if needed: -```bash -cd /path/to/module -PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json -``` - -**Real-world example** (llmage/llmusage, 2026-06-25): -- userorgid: "用户机构" → "orgname" -- ownerid: "模型机构" → "orgname" -- userid: "用户id" → "username" -- llmid: "模型id" → "model" -- Commit: `13c123c` +### Pitfall 42: Field title renaming — modify `models/
.json`, NOT CRUD/UI files +To rename display titles (e.g. "用户id" → "username"), change `fields[].title` in `models/
.json`. The model `title` is the single source of truth, propagated to list grid headers, filter form labels, edit form labels, and add form labels. Do NOT edit titles in `json/
.json` browserfields (overrides only), generated `wwwroot/
/index.ui` (build artifact), or `wwwroot/api/get_
.dspy` (titles aren't in the query layer). After renaming, regenerate CRUD files. Real example (llmage/llmusage 2026-06-25, commit `13c123c`): userorgid "用户机构"→"orgname", ownerid "模型机构"→"orgname", userid "用户id"→"username", llmid "模型id"→"model". ### Pitfall 43: `logined_userorgid` and `logined_userid` go in `params`, NOT `browserfields` - -**Symptom**: After adding `logined_userorgid` to a CRUD JSON, `xls2ui` crashes with `JSONDecodeError`, or the field appears in the wrong location in generated files. - -**Wrong** (inside `browserfields`): +Putting them inside `browserfields` crashes `xls2ui` with `JSONDecodeError` or places them wrongly in generated files. They are first-class `params` keys, siblings of `sortby`, `data_filter`, `editable`: ```json -{ - "tblname": "payment_log", - "params": { - "browserfields": { - "exclouded": ["id"], - "logined_userorgid": "customerid" // WRONG — should be at params level - } - } +"params": { + "logined_userorgid": "customerid", + "browserfields": {"exclouded": ["id"]} } ``` -**Correct** (at `params` level, sibling to `browserfields`): -```json -{ - "tblname": "payment_log", - "params": { - "logined_userorgid": "customerid", - "browserfields": { - "exclouded": ["id"] - } - } -} -``` +### Pitfall 44: `editable.new_data_url` was NEVER read by xls2ddl template — fixed in fb613d0 +CRUD JSON with `params.editable.new_data_url` set, but the generated index.ui still uses the default `add_
.dspy` (custom create dspy never called). Root cause: the template checked `{% if new_data_url %}` at top level, but after `desc.update(crud_data.params.copy())` the key is nested in `desc.editable`; Jinja2 resolves it to `None` → `{% else %}` always runs. Fix: xls2ddl commit `fb613d0`+ — template checks `{% if (editable and editable.new_data_url) or new_data_url %}` (same for `delete_data_url`, `update_data_url`). Was masked while pre-gitignore generated dspy files existed on servers. Regenerate after updating xls2ddl. -`logined_userorgid` and `logined_userid` are first-class `params` keys alongside `sortby`, `data_filter`, `editable` — NOT `browserfields` children. +### Pitfall 45: filter_fields generated without inline data from browserfields.alters +Search popup crashes with `TypeError: Cannot read properties of undefined (reading 'length')` at `bricks.UiCode.build_options` when a data_filter field has inline `data` only in `browserfields.alters` (e.g. `{"uitype":"code","data":[...]}`) but no corresponding model `codes` entry — filter_fields gets a code header with `data: undefined`. Fix: xls2ddl commit `faa571f`+ (merges alters `data`/`dataurl`/`valueField`/`textField` into filter_fields, applies alters `uitype` override). Regenerate affected CRUD pages. -### Pitfall 45: `editable.new_data_url` in `params.editable` was NEVER read by xls2ddl template — fixed in fb613d0 - -**Symptom**: CRUD JSON has `"params": {"editable": {"new_data_url": "{{entire_url('../api/custom_create.dspy')}}"}}` but the generated `index.ui` still uses the default `add_
.dspy` URL. The custom create dspy is never called. - -**Root cause**: The xls2ddl template (`data_browser_tmpl` in `tmpls.py`) checks `{% if new_data_url %}` at the TOP LEVEL of the template context. But after `desc.update(crud_data.params.copy())`, `new_data_url` is nested inside `desc.editable` (a dict), NOT at `desc.new_data_url`. Jinja2 resolves `new_data_url` → `desc.new_data_url` → `None` (via `DictObject.__getattr__` which returns None for missing keys), so the `{% else %}` branch always runs. - -**Fix**: Update xls2ddl to commit `fb613d0` or later — template now checks `{% if (editable and editable.new_data_url) or new_data_url %}`, falling back through nested editable, then top-level, then default. Same for `delete_data_url` and `update_data_url`. - -**Why this was masked**: In practice, the auto-generated `add_
.dspy` files existed on servers from BEFORE they were gitignored. So even though the template used the default URL, the file was present and the add flow worked. After gitignoring the generated directories, the default dspy files were no longer deployed → the old bug surfaced. - -**Regeneration required**: -```bash -cd ~/repos/xls2ddl && git pull -cd ~/repos/ -~/repos/xls2ddl/py3/bin/python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json -``` - -### Pitfall 44: filter_fields generated without inline data from browserfields.alters - -**Symptom**: CRUD list page loads but the search/filter form's code-type dropdowns (e.g., `is_external`, `status`) crash with `TypeError: Cannot read properties of undefined (reading 'length')` at `bricks.UiCode.build_options`. The filter popup doesn't appear at all. - -**Root cause**: xls2ddl's `build_filter_field_list()` only merges `data` from model-level `codes` definitions, NOT from `browserfields.alters` inline `data` arrays. Fields like `is_external` with `{"uitype":"code","data":[{"value":"1","text":"外部供应商"}]}` in alters — but no corresponding model `codes` entry — get a `uitype:"code"` header in filter_fields with `data: undefined`. - -**Fix**: Update xls2ddl to commit `faa571f` or later (merges alters `data`/`dataurl`/`valueField`/`textField` into filter_fields, and applies alters `uitype` override). - -**Regeneration required**: -```bash -cd ~/repos/xls2ddl && git pull -cd ~/repos/ -~/repos/xls2ddl/py3/bin/python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json -``` - -**Affected**: All CRUD pages where `data_filter` includes fields that have inline `data` only in `browserfields.alters` (no model `codes` entry). - -### Pitfall 42: CRUD JSON files must be PURE JSON — NO Jinja2 control-flow blocks - -**Hard rule**: CRUD JSON files in `json/` are consumed by `xls2ddl.xls2crud`, which parses them as pure JSON. Jinja2 control-flow blocks (`{% if %}`, `{% for %}`, `{% endif %}`) cause `JSONDecodeError` at parse time. - -**Wrong** (causes `json.decoder.JSONDecodeError` at xls2ui runtime): -```json -{ - "binds": [{ - "popup_options": { -{% if params_kw._is_mobile %} - "width": "100%", -{% else %} - "width": "40%", -{% endif %} - "archor": "cc" - } - }] -} -``` - -**Correct** — use fixed values: -```json -{ - "binds": [{ - "popup_options": { - "width": "40%", - "height": "80%", - "archor": "cc" - } - }] -} -``` - -`{{entire_url('...')}}` placeholders inside JSON string values are acceptable — they're handled by the framework during URL generation. But `{% %}` Jinja2 control flow is never allowed in CRUD JSON files. This is distinct from `.ui` template files which DO support Jinja2. - -### Pitfall 50: Custom list endpoint for code resolution — LEFT JOIN reference tables for `_text` grid display - -**Symptom**: Form/filter dropdowns show orgname correctly (via `uitype: "code"` + `dataurl`), but the list grid still shows raw IDs. The `dataurl` endpoint returns correct `[{value, text}]` / `[{fieldname, fieldname_text}]`. - -**Root cause**: The auto-generated `get_
.dspy` queries only the base table (`SELECT * FROM table WHERE ...`). It does NOT join reference tables to produce `_text` columns for grid cells. `uitype: "code"` + `dataurl` only controls dropdown rendering in forms/filters — it does NOT extend to grid cell display. **Inline `data` arrays for status/type fields also do NOT auto-resolve for list grid cells.** - -**Why Pitfall 18 and Pitfall 19 don't cover this**: Pitfall 18 says "DO NOT...the CRUD auto-generated list handles everything" — it doesn't. The auto-generated list never LEFT JOINs lookup tables. Pitfall 19 says "Never manually add data_url" — but custom list endpoints are the only way to get code resolution in grid cells. - -**Fix — TWO approaches (choose one):** - -**Approach A — COALESCE (recommended, more reliable):** Replace raw IDs with display names directly in SQL output. Guarantees grid renders names regardless of bricks `_text` detection behavior. +### Pitfall 46: CRUD JSON files must be PURE JSON — NO Jinja2 control-flow blocks +`json/` files are parsed as pure JSON by `xls2ddl.xls2crud`; `{% if %}`, `{% for %}`, `{% endif %}` cause `JSONDecodeError` at parse time. Use fixed values instead (e.g. hardcode `"width": "40%"` instead of `{% if params_kw._is_mobile %}`). `{{entire_url('...')}}` placeholders inside JSON string values ARE acceptable — handled during URL generation. This is distinct from `.ui` template files which DO support Jinja2. See `references/crud-json-rules.md`. +### Pitfall 47: Custom list endpoint for code resolution — LEFT JOIN reference tables for `_text` grid display +Symptom: form/filter dropdowns show names (via `uitype: "code"` + dataurl) but the list grid still shows raw IDs. Root cause: the auto-generated `get_
.dspy` queries only the base table — no reference-table joins. `uitype:"code"` + dataurl only controls form/filter dropdowns, NOT grid cells; **inline `data` arrays for status/type fields also do NOT auto-resolve for grid cells**. Pitfalls 14/15 don't cover this — the auto-generated list never LEFT JOINs, and a custom list endpoint is the only way to get grid code resolution. +**Approach A — COALESCE (recommended, the only reliable one)**: replace raw IDs with display names directly in SQL output — guarantees grid rendering regardless of bricks `_text` detection: ```python sql = '''select a.id, a.domain, COALESCE(b.orgname, a.resellerid) as resellerid, COALESCE(c.orgname, a.orgid) as orgid, - CASE a.status WHEN 'active' THEN '启用' WHEN 'inactive' THEN '停用' ELSE a.status END as status, - a.created_at, a.updated_at + CASE a.status WHEN 'active' THEN '启用' WHEN 'inactive' THEN '停用' ELSE a.status END as status from (select * from tenant_domain where 1=1 [[filterstr]]) a left join (select id, orgname from organization) b on a.resellerid = b.id left join (select id, orgname from organization) c on a.orgid = c.id''' ``` - -Pros: Guaranteed. Filter WHERE applies on inner query with raw IDs before COALESCE. Cons: Edit form must use separate `_get.dspy` returning raw IDs. - -**Approach B — `_text` suffix:** Return extra `resellerid_text`, `orgid_text` columns via LEFT JOIN. Bricks Tabular MAY auto-detect `_text` columns — but detection is unreliable (in our 2026-07-10 testing, did NOT render). If it doesn't work, fall back to Approach A. - -**Placement — TWO valid locations:** -1. `wwwroot/{alias}/get_{alias}.dspy` — CRUD framework auto-discovers. No `get_data_url` needed. BUT this directory may be gitignored (use `git add -f`). -2. `wwwroot/api/
_list.dspy` + `editable.get_data_url` — requires xls2ddl template support (Pitfall 45). - -**RBAC registration (required):** -```sql -INSERT INTO permission (id,path) VALUES (REPLACE(UUID(),'-',''),'/module/api/.dspy'); -INSERT INTO rolepermission (id,roleid,permid) - SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/.dspy'; -``` - -**Key rules:** -- LEFT JOIN alias MUST match the field name (e.g., `id as resellerid` → `on a.resellerid = b.id`) -- DO NOT edit `wwwroot/
/get_
.dspy` — it's gitignored and regenerated -- **Inline `data` arrays in CRUD JSON alters do NOT resolve for list grid cells** — handle status/type fields in SQL with CASE WHEN -- New dspy endpoints need BOTH `permission` table entry AND `rolepermission` entry (role `logined`) -- The auto-discovery path `wwwroot/{alias}/get_{alias}.dspy` takes precedence over `get_data_url` - +Filter WHERE applies on the inner query with raw IDs before COALESCE. Con: the edit form must use a separate `_get.dspy` returning raw IDs. +**Approach B — `_text` suffix**: return `resellerid_text`/`orgid_text` columns via LEFT JOIN; bricks Tabular MAY auto-detect `_text` columns but detection is unreliable (2026-07-10 testing: did NOT render) — fall back to Approach A. +**Placement (two valid locations)**: (1) `wwwroot/{alias}/get_{alias}.dspy` — CRUD framework auto-discovers; no `get_data_url` needed; directory may be gitignored (use `git add -f`); takes precedence over `get_data_url`; (2) `wwwroot/api/
_list.dspy` + `editable.get_data_url` — requires xls2ddl ≥ fb613d0 (Pitfall 44). +**Key rules**: LEFT JOIN alias MUST match the field name; never edit the gitignored `wwwroot/
/get_
.dspy`; handle status/type fields in SQL with CASE WHEN; new dspy endpoints need BOTH `permission` + `rolepermission` (`role 'logined'`) SQL rows (see Pitfall 20). **CRITICAL deployment pitfalls (2026-07-10):** - -1. **Server restart required for CRUD directory dspys**: The CRUD framework caches custom `get_{alias}.dspy` files at server STARTUP. Hot-reload applies to `wwwroot/api/` but NOT to auto-generated CRUD directories. After deploying a new `get_{alias}.dspy`, restart Sage (`./stop.sh && ./start.sh`). - -2. **`default_filterjson` trap — ns field pollution**: Any field set in `ns` before `default_filterjson(fields, ns)` becomes an implicit filter. NEVER set business fields in `ns` — only set framework vars (`__logined_orgid__`, `userorgid`). Example: `ns['resellerid'] = userorgid` leaked into default_filterjson and filtered the list to 1 row. - -3. **Approach A (COALESCE) is the only reliable option**: In practice, bricks Tabular did NOT render Approach B's `_text` columns. COALESCE replaces values directly in the output column — guaranteed to work. - -4. **Git-ignored directory**: `wwwroot/{alias}/` is in `.gitignore`. Custom dspys require `git add -f` to track. \ No newline at end of file +1. **Server restart required for CRUD directory dspys**: custom `get_{alias}.dspy` files in CRUD directories are cached at server STARTUP — hot-reload applies to `wwwroot/api/` but NOT to auto-generated CRUD directories. After deploying, restart Sage (`./stop.sh && ./start.sh`). +2. **`default_filterjson` trap — ns field pollution**: any field set in `ns` before `default_filterjson(fields, ns)` becomes an implicit filter. NEVER set business fields in `ns` — only framework vars (`__logined_orgid__`, `userorgid`). Example: `ns['resellerid'] = userorgid` leaked into default_filterjson and filtered the list to 1 row. +3. **Approach A (COALESCE) is the only reliable option** — Approach B's `_text` columns did not render in practice. diff --git a/skills_library/all/database-table-definition-spec/SKILL.md b/skills_library/all/database-table-definition-spec/SKILL.md index 4cbe9d5..a2ba163 100644 --- a/skills_library/all/database-table-definition-spec/SKILL.md +++ b/skills_library/all/database-table-definition-spec/SKILL.md @@ -185,8 +185,6 @@ d['codes'] = [c for c in d['codes'] if not (c.get('field') in seen or seen.add(c ``` **Why this happens**: Adding codes programmatically (e.g., Python dict.append) without checking for existing entries. Always verify the codes array has unique `field` values before saving. -1. **Dictionary codes** (table=`appcodes_kv`): Use `parentid=` cond, `valuefield: "k"`, `textfield: "v"`. Data comes from `init/data.json` Format B. -2. **Foreign key codes** (table=other table): Use `valuefield: "id"`, `textfield: "display_field"`, no cond needed. References another module's table for dropdown population. ## Creating Models Directory (New Modules) diff --git a/skills_library/all/dspy-file-implementation-spec/SKILL.md b/skills_library/all/dspy-file-implementation-spec/SKILL.md index 6ac55ac..a13a0bd 100644 --- a/skills_library/all/dspy-file-implementation-spec/SKILL.md +++ b/skills_library/all/dspy-file-implementation-spec/SKILL.md @@ -8,546 +8,104 @@ tags: [ahserver, dspy, backend, web-development, python] # .dspy File Implementation Specification ## Overview -.dspy files are controlled Python scripts executed by the ahserver web framework to provide dynamic API endpoints. They must follow strict conventions to ensure security, performance, and compatibility with the framework's architecture. +.dspy files are controlled Python scripts executed by the ahserver web framework to provide dynamic API endpoints. They must follow strict conventions for security, performance, and framework compatibility. -## Core Principles +## Core Rules -1. **No Import Statements** -**Never use import statements** in .dspy files. The ahserver framework: -- Automatically provides access to functions exported by your application module through `load_{modulename}()` -- Has already pre-loaded common Python modules (datetime, json, os, sys, etc.) into the global context +### 1. No Import Statements +Never use `import` in .dspy files. The framework auto-provides module functions via `load_{modulename}()` and pre-loads common modules (`datetime` as the full module, `json`, `os`, `sys`) into the global context. +- ⚠️ `from datetime import date` WILL fail with an import error — use `datetime.date.today()`. +- ✅ `today = datetime.date.today().isoformat()`; `json.dumps({'k':'v'})`; module functions like `get_all_records()` used bare. -**❌ Incorrect:** -```python -import json -import datetime -from datetime import date, timedelta -from myapp.init import get_all_records -``` - -**✅ Correct — use pre-loaded modules directly:** -```python -# datetime is pre-loaded as the full module — access via datetime.date, datetime.datetime, datetime.timedelta -today = datetime.date.today().isoformat() -now = datetime.datetime.now() -five_min_ago = (now - datetime.timedelta(minutes=5)).strftime('%Y-%m-%d %H:%M:%S') - -# json is pre-loaded -result = json.dumps({'key': 'value'}) - -# Directly use functions provided by load_app_module() -records = get_all_records() -``` - -**⚠️ Pitfall**: `from datetime import date` looks innocent but WILL cause the .dspy file to fail with an import error. Use `datetime.date.today()` instead. - -### 2. Use Return, Not Print -**Always use `return` to send data back to the client**, never use `print()`. The ahserver framework handles JSON serialization automatically. - -**❌ Incorrect:** -```python -result = {"data": records} -print(json.dumps(result)) -``` - -**✅ Correct:** -```python -return records -``` +### 2. Return, Not Print +Always `return` data to the client; ahserver handles JSON serialization. `print()` writes to stdout that ahserver ignores → `return data type error, `. +- ✅ `return records` (never `print(json.dumps(result))`). ### 3. ID Generation: `uuid()` in .dspy/.ui, `getID()` in .py +Both `uuid()` and `getID()` work in .dspy context (verified: llmage dspy files use `getID()` without import). `uuid()` returns shorter IDs, `getID()` returns 22-char IDs. In `.py` files (init.py, utils.py) you must `from appPublic.uniqueID import getID`. -**CRITICAL**: Both `uuid()` and `getID()` are available in `.dspy` context: +### 4. Error Handling +- Array-returning endpoints (code components): `try: ... return result except Exception: return []`. +- Object-returning endpoints: `try: ... return record except Exception: return {"error": str(e)}`. -```python -# Both work in .dspy context — use uuid() for new IDs (shorter, simpler) -new_id = uuid() +### 5. Input Validation & Safety +- Validate/sanitize `params_kw` inputs, e.g. `if not record_id or not str(record_id).isdigit(): return {"error": "Invalid ID parameter"}`. +- Never return sensitive fields (passwords, API keys) unless required and authorized. +- For expensive operations in production, add rate limiting. -# getID() is also pre-loaded in .dspy context (verified: llmage dspy files use it without import) -new_id = getID() -``` - -In `.py` files (e.g., `init.py`, `utils.py`), you must import: `from appPublic.uniqueID import getID`. - -### 4. Proper Error Handling -Handle exceptions gracefully and return appropriate data structures based on component requirements. - -**For array-returning endpoints (e.g., code components):** -```python -try: - records = get_all_records() - result = [] - for record in records: - result.append({ - "value": str(record.get('id')), - "text": record.get('name', f"Record {record.get('id')}") - }) - return result -except Exception as e: - return [] # Return empty array on error -``` - -**For object-returning endpoints:** -```python -try: - record = get_record_by_id(id) - return record -except Exception as e: - return {"error": str(e)} -``` - -## Common Use Cases - -### 1. Code Component Data Endpoints -Code components require specific `{value, text}` array format: - -**File:** `/wwwroot/entity_name/list/index.dspy` -```python -# Get entity list for code dropdown -# This .dspy file uses functions released by load_app_module() - -try: - # Use the function provided by your module - records = get_all_records() - - # Format for code component (value, text pairs) - result = [] - for record in records: - result.append({ - "value": str(record.get('id')), - "text": record.get('name', f"Record {record.get('id')}") - }) - - # Return array directly for code component - return result -except Exception as e: - # On error or no data, return empty array - return [] -``` - -### 2. Single Record Endpoints -For retrieving individual records: - -**File:** `/wwwroot/entity_name/get/index.dspy` -```python -# Get single entity record -# Access query parameters via params_kw dictionary - -try: - record_id = params_kw.get('id') - if not record_id: - return {"error": "ID parameter required"} - - record = get_record_by_id(record_id) - return record -except Exception as e: - return {"error": str(e)} -``` - -### 3. Action Endpoints -For performing actions like testing connections: - -**File:** `/wwwroot/entity_name/test/index.dspy` -```python -# Test entity connection or perform action - -try: - entity_id = params_kw.get('id') - if not entity_id: - return {"status": "error", "message": "ID parameter required"} - - result = test_entity_connection(entity_id) - return {"status": "success", "message": result} -except Exception as e: - return {"status": "error", "message": str(e)} -``` - -### 4. Login Endpoint Pattern -Login endpoints require special handling for password encoding and session creation: - -**File:** `/wwwroot/login.dspy` -```python -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -"""Login handler - uses server-env functions, no imports needed""" -username = params_kw.get('username', '') -password = params_kw.get('password', '') - -if not username: - return json.dumps({'status': 'error', 'message': 'Username required'}, ensure_ascii=False) -if not password: - return json.dumps({'status': 'error', 'message': 'Password required'}, ensure_ascii=False) - -# Encode password for comparison with stored hash -passwd = password_encode(password) - -# Use server-env registered check_user_password -rzt = await check_user_password(request, username, passwd) - -if rzt: - # Get user info from database - dbname = get_module_dbname('rbac') - async with DBPools().sqlorContext(dbname) as sor: - users = await sor.sqlExe( - "SELECT id, username, name, orgid FROM users WHERE username=${username}$", - {'username': username} - ) - - if users: - user = users[0] - # Create session using remember_user (available in .dspy context) - await remember_user(user.id, user.username, getattr(user, 'orgid', '') or '') - return json.dumps({ - 'status': 'ok', - 'message': 'Login successful', - 'redirect': '/main/base.ui', - 'userid': user.id, - 'username': user.username - }, ensure_ascii=False) - -# Failed login -return json.dumps({'status': 'error', 'message': 'Invalid credentials'}, ensure_ascii=False) -``` - -**Key points for login .dspy:** -- Use `password_encode()` to hash the submitted password before comparison -- Use `check_user_password(request, username, encoded_password)` for RBAC authentication -- Use `remember_user(userid, username, userorgid)` to create session (NOT `user_login()` - that requires explicit import which fails in .dspy) -- Return a string via `json.dumps()`, never return `None` - -## Security Considerations - -### 1. Input Validation -Always validate and sanitize input parameters from `params_kw`: - -```python -# Validate ID parameter -record_id = params_kw.get('id') -if not record_id or not str(record_id).isdigit(): - return {"error": "Invalid ID parameter"} -``` - -### 2. Avoid Sensitive Data -Never return sensitive fields like passwords, API keys, or internal system data unless explicitly required and properly authorized. - -### 3. Rate Limiting -For production applications, implement rate limiting for expensive operations: - -```python -# Check rate limit (pseudo-code) -if is_rate_limited(request_ip): - return {"error": "Rate limit exceeded"} -``` - -## Performance Guidelines - -### 1. Efficient Data Retrieval -Use appropriate database queries with proper filtering and pagination: - -```python -# Use efficient queries with limits -records = get_records_with_limit(offset=0, limit=100) -``` - -### 2. Caching -Implement caching for frequently accessed, rarely changing data: - -```python -# Use application-level cache -cache_key = f"records_list_{timestamp}" -if cache_key in app_cache: - return app_cache[cache_key] - -records = get_all_records() -app_cache[cache_key] = records -return records -``` - -## DSPY Code Review Checklist - -A structured checklist for reviewing `.dspy` files — see `references/dspy-code-review-checklist.md` for detailed walkthroughs of each check with real-world bug examples (Decimal serialization crashes, missing `int()` on SUM aggregates, DRY violations, sibling-file inconsistency detection). - -### Syntax & Security -- [ ] **No imports** — module DSPY files must have zero import statements. All needed names (`json`, `datetime`, `get_sor_context`, `DBPools`, `params_kw`, `request`, `uuid`, `time`, `os`, `DictObject`, `FileStorage`, logging functions) are pre-loaded. -- [ ] **No forbidden patterns** — no `eval()`, `exec()`, `__import__()`, `os.system()`, `subprocess`, `pickle.loads()`. -- [ ] **Valid Python AST** — file passes `ast.parse()`. Quick check: `python3 -c "import ast; ast.parse(open('file.dspy').read()); print('OK')"`. **⚠️ .dspy files contain top-level `await`/`async with` which bare ast.parse rejects ("await outside async function")** — wrap first: `wrapped = 'async def __c__(params_kw, request, uid, org_id, json, DBPools, get_user, get_userorgid, get_module_dbname, getID, debug, sor, params_kw=None):\n' + '\n'.join(' ' + line if line.strip() else line for line in src.split('\n')); ast.parse(wrapped)` (add injected names the file uses to the wrapper signature). This wrapped check is **mandatory after patching triple-quoted prompt constants** — a stray `"""` silently closes the string and dumps the following prose as code; only ast.parse exposes it (caught live 2026-08 in cockpit_chat.dspy). -- [ ] **All branches return** — every code path ends with an explicit `return`. Missing return → `return data type error, `. - -### SQL & Database -- [ ] **Parameterized queries** — uses `${param}$` syntax, never f-string interpolation or `%s` formatting in SQL strings. -- [ ] **Decimal / SUM aggregate safety** — `SUM()` in MySQL returns `Decimal`. Must wrap with `int()` or pass `default=str` in `json.dumps()`. Check: `r.total_size or 0` should be `int(r.total_size or 0)`. This is the same class of bug as doc_count/chunk_count lacking `int()`. -- [ ] **Cross-module access** — uses `get_sor_context(env, 'module')`, not `DBPools().sqlorContext(dbname)` for modules outside the current one. -- [ ] **sqlExe return type awareness** — without `page`/`rows` in ns → list of row objects (use `r.field` attrs); with `page`/`rows` → `{'total': N, 'rows': [...]}` dict. -- [ ] **Error handling** — at least a try/except around DB ops with a fallback return. - -### Code Quality (KISS/DRY) -- [ ] **Sibling file consistency** — compare against other `.dspy` files in the same directory. Inconsistent return format (raw dict vs `json.dumps()`), divergent helper signatures, or different API patterns are red flags. -- [ ] **DRY — no duplicated helpers** — check for size formatters (`fmt_size`, `fmt`), date formatters, or SQL builders duplicated across files in the project. Three identical copies of the same function is a signal to extract. -- [ ] **No hardcoded config values** — storage limits, API URLs, timeouts should come from config, not be embedded in code. -- [ ] **f-string safety** — avoid f-strings in dict returns; `exec()` wrapping can misparse `}` braces. Use concatenation `'prefix: ' + str(var)` instead. -- [ ] **No `print()`** — use `return` for output. `print()` writes to stdout that ahserver ignores, producing `NoneType` error. - -### Return Format -- [ ] **Consistent return style** — all DSPY files in a directory should use the same pattern: either raw dict `return {...}` or `json.dumps({...})`. -- [ ] **DataViewer CRUD endpoints** — must return `Message` widget JSON, not raw data. -- [ ] **Code component endpoints** — must return `[{value, text}]` array. -- [ ] **JSON validity** — if the DSPY returns a hardcoded JSON-like dict, validate the resulting JSON serializes correctly (watch for `Decimal`, `datetime`, `bytes` types that `json.dumps` can't handle without `default=str`). - -## Testing and Validation - -### 1. Manual Testing -Test .dspy endpoints directly by accessing their URLs in a browser: - -``` -http://localhost:8000/app-name/entity_name/list/ -``` - -### 2. Data Format Validation -Verify that returned data matches the expected format for the consuming component: - -- **Code components**: Array of `{value, text}` objects -- **DataViewer**: Array of full record objects -- **Forms**: Single record object or success/error object - -### 3. Error Scenario Testing -Test error scenarios like missing parameters, invalid IDs, and database failures. - -## Integration with Bricks Framework - -### 1. UI File References -Reference .dspy endpoints in .ui files using standard URL format: - -```json -{ - "uitype": "code", - "data_url": "/app-name/entity_name/list/" -} -``` - -### 2. Parameter Passing -Pass parameters to .dspy endpoints using query strings: - -```json -{ - "data_url": "/app-name/entity_name/get/?id={{selectedRow.id}}" -} -``` - -## CRUD List API Pattern (sqlor-based) - -For DataGrid/CRUD widget data endpoints, use this standardized pattern: - -```python -# CRUD list API for DataViewer — no imports needed, json/DBPools are pre-loaded - -result = {'success': False, 'rows': [], 'total': 0} - -try: - dbname = get_module_dbname('module_name') - async with DBPools().sqlorContext(dbname) as sor: - # Build WHERE clause dynamically - where_clauses = [] - where_ns = {} - - customer_id = params_kw.get('customer_id', '') - status = params_kw.get('status', '') - - if customer_id: - where_clauses.append("customer_id=${customer_id}$") - where_ns['customer_id'] = customer_id - if status: - where_clauses.append("status=${status}$") - where_ns['status'] = status - - where_sql = " AND ".join(where_clauses) - where_prefix = " WHERE " if where_clauses else "" - - # Count query (no pagination needed) - count_sql = "SELECT count(*) rcnt FROM table_name" + where_prefix + where_sql - count_rows = await sor.sqlExe(count_sql, where_ns) - total = 0 - if count_rows and len(count_rows) > 0: - r = count_rows[0] - if hasattr(r, 'keys'): - total = r.get('rcnt', 0) - elif isinstance(r, dict): - total = r.get('rcnt', 0) - elif hasattr(r, 'rcnt'): - total = r.rcnt - - if total > 0: - # Pagination query - ns = {'page': int(params_kw.get('page', 1)), 'rows': int(params_kw.get('rows', 20)), 'sort': params_kw.get('sort', 'id')} - sql = "SELECT col1, col2, col3 FROM table_name" + where_prefix + where_sql - - # Merge ns and where_ns (avoid {**ns, **sql_ns} which fails) - query_ns = dict(list(ns.items()) + list(where_ns.items())) - rows = await sor.sqlExe(sql, query_ns) - - # sqlExe with page/rows returns {'total': N, 'rows': [...]} - if isinstance(rows, dict): - result['rows'] = rows.get('rows', []) - result['total'] = rows.get('total', total) - elif rows: - result['rows'] = [dict(r) if hasattr(r, 'keys') else r for r in rows] - result['total'] = total - - result['success'] = True -except Exception as e: - result['error'] = str(e) - -return json.dumps(result, ensure_ascii=False, default=str) -``` - -**Key points:** -- Return format: `{'success': bool, 'rows': [...], 'total': int}` -- Use `params_kw.get()` for pagination parameters -- Use `${param}$` syntax for LIMIT/OFFSET in sqlExe -- Convert rows to dicts: `[dict(r) for r in data]` -- Use `default=str` in json.dumps for datetime handling -- **CRITICAL**: All SELECT columns must match the actual database schema exactly. Always verify with `DESCRIBE table_name` before writing queries. - -## Cross-Module Database Access Pattern - -When a .dspy file in one module needs to access tables belonging to another module: - -### REQUIRED: `get_sor_context(request._run_ns, 'module')` — the ONLY correct pattern - -```python -# In .dspy files — request is auto-injected -env = request._run_ns -async with get_sor_context(env, "module_name") as sor: - records = await sor.R('table_name', {'filter': 'value'}) -``` - -This is the **only** cross-db access pattern. It works because it delegates to the `module_dbname` config: in the Sage system, a module named "tenant" resolves to the `sage` database; in the pipeline-app, the same module resolves to the `pipeline` database. The module's owner configures this mapping per deployment. - -### ❌ NEVER use hardcoded database names - -```python -# WRONG — hardcoded db name breaks cross-deployment portability -async with db.sqlorContext("pipeline") as sor: - ... -``` - -This is the single most common cross-module DSPY error. It works in one environment but fails in another (e.g., Sage queries "pipeline" DB which doesn't exist in its DBPools config). Always use `get_sor_context(env, "module_name")` instead. - -### ❌ NEVER use `DBPools()` + `sqlorContext()` for cross-module access - -The `DBPools()` pattern is for accessing the **current** module's database. For cross-module access, use only `get_sor_context`. - -**Key points:** -- **Never use `ServerEnv()` in .dspy files** — all server-env functions (`get_module_dbname`, `DBPools`, `getConfig`, `password_encode`, etc.) are already injected into the .dspy execution context via globals -- **Never hardcode database names** in .dspy files — use `get_sor_context(env, "module_name")` to resolve via config -- `get_sor_context(request._run_ns, 'module')` is the **required** pattern for cross-module DB access -- If a cross-module function is registered via `load_{modulename}()` (like `create_user_apikey` from dapi), use it directly: `create_user_apikey(sor, dappid, user_id, user_orgid)` - -## Batch Operations with $or Queries - -For batch lookups by ID list, use `$or` conditions in the sor.R filter: - -```python -# user_ids is a list of IDs to look up -or_conditions = [{'id': uid} for uid in user_ids] -query_ns = {'$or': or_conditions} -users = await sor.R('users', query_ns) -``` - -**Key points:** -- The `$or` operator is supported by sqlor's filter system -- For large lists (>100 items), consider chunking to avoid query complexity limits -- Always validate the ID list is non-empty before querying - -## Safe Attribute Access on SQLor Row Objects - -SQLor returns row objects that may or may not support dict-style access. Use `getattr()` for safe attribute access: - -```python -user = users[0] -user_id = getattr(user, 'id', '') -username = getattr(user, 'username', '') -user_orgid = getattr(user, 'orgid', '') or '' # Handle None -> '' -``` - -**Key points:** -- `getattr(obj, 'attr', default)` is safer than `obj.attr` (avoids AttributeError) -- Use `or ''` pattern for fields that may be None but need to be a string -- For dict-like access: `getattr(user, 'orgid', '') or ''` handles both missing attribute and None value - -## Server-Env Functions Available in .dspy Context - -The ahserver framework injects many functions into the .dspy execution context. **No import needed** - just use them directly: +## Pre-Loaded Server-Env Functions (.dspy context — NO import needed) | Function | Description | |----------|-------------| -| `password_encode(s)` | Hash a password using the app's configured key | -| `password_decode(s)` | Decode a hashed password | +| `password_encode(s)` / `password_decode(s)` | Hash / decode a password using the app's configured key | | `remember_user(userid, username, userorgid)` | Set session user (login) | | `forget_user()` | Clear session user (logout) | -| `get_user()` | Get current logged-in user ID | -| `get_username()` | Get current user's display name | -| `get_userorgid()` | Get current user's org ID | -| `get_userinfo()` | Get full user info object | -| `get_session()` | Get session object | -| `session_getvalue(key)` | Read session value | -| `session_setvalue(key, value)` | Write session value | +| `get_user()` / `get_username()` / `get_userorgid()` / `get_userinfo()` | Current user id / display name / org id / full user object | +| `get_session()` / `session_getvalue(key)` / `session_setvalue(key, value)` | Session access | | `get_module_dbname(modulename)` | Get DB name for a module | | `get_sor_context(env, modulename)` | Async context manager for cross-module DB access | -| `DBPools()` | Get database connection pool | -| `params_kw` | Dictionary of request parameters — query string + POST body (including `application/json`), merged into one dict. Nested JSON objects preserved as dict/list. **This is the ONLY way to access request data — there is NO `http_request` variable.** | +| `DBPools()` | Database connection pool (current module's DB) | +| `params_kw` | Dict of request params — query string + POST body (incl. `application/json`) merged; nested JSON preserved as dict/list. **The ONLY way to access request data — there is NO `http_request` variable.** | | `request` | The ahserver Request object (auto-injected) | -| `json` | json module (json.dumps, json.loads) | -| `datetime` | datetime module (datetime.date, datetime.datetime, datetime.timedelta) | -| `uuid` / `getID` | ID generation — both work. `uuid()` returns shorter IDs, `getID()` returns 22-char IDs | -| `time` | time module | -| `os` | os module (MAY be available — verify if needed; observed as imported in recover_usages.dspy for `os.path.isfile`) | -| `DictObject` | From appPublic.dictObject — available directly (no import) | -| `partial` | functools.partial — available directly (no import) | -| `FileStorage` | From ahserver.filestorage — available directly (no import) | -| `curDateString` / `timestampstr` | From appPublic.timeUtils — date/time string helpers | +| `json`, `datetime`, `time`, `os` | Pre-loaded modules (`os` MAY be available — verify if needed) | +| `uuid` / `getID` | ID generation (both work) | +| `DictObject` | appPublic.dictObject — available directly | +| `partial` | functools.partial — available directly | +| `FileStorage` | ahserver.filestorage — available directly | +| `curDateString` / `timestampstr` | appPublic.timeUtils date/time string helpers | | `get_config_value(key)` | Get config value | | `exception`, `error`, `debug`, `info`, `warning`, `critical` | Logging functions — all available | -| `format_exc` | `traceback.format_exc()` — returns full traceback string (pre-loaded, do NOT `import traceback`) | +| `format_exc` | `traceback.format_exc()` — full traceback string (pre-loaded, do NOT `import traceback`) | -**Verified via llmage module dspy cleanup (2026-07-01)**: All 31 dspy files had their `import` statements removed and continue to work. The complete list of safely removable imports: `json`, `datetime`, `getID` (appPublic.uniqueID), `debug` (appPublic.log), `curDateString`/`timestampstr` (appPublic.timeUtils), `get_sor_context` (sqlor.dbpools), `time`, `DictObject` (appPublic.dictObject), `partial` (functools), `FileStorage` (ahserver.filestorage), `os`. +**Verified (llmage cleanup 2026-07-01)**: all 31 dspy files worked after removing imports of: `json`, `datetime`, `getID` (appPublic.uniqueID), `debug` (appPublic.log), `curDateString`/`timestampstr` (appPublic.timeUtils), `get_sor_context` (sqlor.dbpools), `time`, `DictObject` (appPublic.dictObject), `partial` (functools), `FileStorage` (ahserver.filestorage), `os`. -## DataViewer CRUD Endpoint Pattern - -When implementing full CRUD (Create/Update/Delete) for DataViewer widgets, the endpoints must return **Message widget JSON**, not raw data: +## Endpoint Patterns +### Code Component Data Endpoints +File: `/wwwroot/entity_name/list/index.dspy`. Must return an array of `{value, text}` (value as string) — `[]` on error/no data. ```python -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -"""Customer create API for DataViewer editable form""" -# No imports needed - json, DBPools, etc. are pre-loaded - -result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid request'}} - try: - name = params_kw.get('customer_name', '') - if not name: - result['options'] = {'title': 'Error', 'message': 'Name required', 'type': 'error'} - else: - dbname = get_module_dbname('module_name') - async with DBPools().sqlorContext(dbname) as sor: - await sor.sqlExe("INSERT INTO table_name (...) VALUES (...)", {...}) - - result = { - 'widgettype': 'Message', - 'options': {'title': 'Success', 'message': 'Created successfully', 'type': 'success'} - } -except Exception as e: - result['options'] = {'title': 'Error', 'message': f'Failed: {str(e)}', 'type': 'error'} - -return json.dumps(result, ensure_ascii=False) + result = [{"value": str(r.get('id')), "text": r.get('name', f"Record {r.get('id')}")} for r in get_all_records()] + return result +except Exception: + return [] ``` -**CRITICAL**: The return value MUST be a string (via `json.dumps()`). If the script reaches the end without hitting a `return` statement, ahserver throws `return data type error, `. Every code path must return a string. +### Single Record / Action Endpoints +- `/wwwroot/entity_name/get/index.dspy`: `record_id = params_kw.get('id')`; missing id → `{"error": "ID parameter required"}`; else record or `{"error": str(e)}`. +- `/wwwroot/entity_name/test/index.dspy`: return `{"status": "success"/"error", "message": ...}`. -## DataViewer Editable Configuration in .ui +### Login Endpoint +File: `/wwwroot/login.dspy`. Key points: +- `passwd = password_encode(password)` before comparison; `rzt = await check_user_password(request, username, passwd)` (RBAC auth). +- On success: `dbname = get_module_dbname('rbac')`, `async with DBPools().sqlorContext(dbname) as sor:` → `await sor.sqlExe("SELECT id, username, name, orgid FROM users WHERE username=${username}$", {'username': username})`; then `await remember_user(user.id, user.username, getattr(user, 'orgid', '') or '')`. +- Use `remember_user()` to create session — NOT `user_login()` (requires explicit import, fails in .dspy). +- Always return a string via `json.dumps(..., ensure_ascii=False)`, never `None`. -Configure CRUD operations in the DataViewer's `options.editable` block: +### CRUD List API Pattern (sqlor-based) +Return format: `{'success': bool, 'rows': [...], 'total': int}` via `json.dumps(result, ensure_ascii=False, default=str)`. +- Use `params_kw.get()` for pagination (`page`, `rows`, `sort`), `${param}$` syntax everywhere. +- Do **separate count + data queries**; `ns = {'page': int(...), 'rows': int(...), 'sort': ...}`. +- Merge ns dicts with `query_ns = dict(list(ns.items()) + list(where_ns.items()))` — `{**ns, **sql_ns}` FAILS. +- `sqlExe` with `page`/`rows` in ns returns `{'total': N, 'rows': [...]}` dict; without them returns a list of row objects. +- Convert rows: `[dict(r) if hasattr(r, 'keys') else r for r in rows]`. +- **CRITICAL**: all SELECT columns must exactly match the actual DB schema — verify with `DESCRIBE table_name` before writing queries (DDL files may differ from deployed schema). +### DataViewer CRUD Endpoint +Create/update/delete endpoints for DataViewer editable forms must return **Message widget JSON as a STRING**, not raw data: +```python +result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid request'}} +try: + ... + result = {'widgettype': 'Message', 'options': {'title': 'Success', 'message': 'Created successfully', 'type': 'success'}} +except Exception as e: + result['options'] = {'title': 'Error', 'message': f'Failed: {str(e)}', 'type': 'error'} +return json.dumps(result, ensure_ascii=False) +``` +**CRITICAL**: the return value MUST be a string (`json.dumps()`); any path ending without `return` → `return data type error, `. + +### DataViewer Editable Config in .ui +Configure in the DataViewer's `options.editable` block: ```json { "widgettype": "DataViewer", @@ -558,395 +116,124 @@ Configure CRUD operations in the DataViewer's `options.editable` block: "update_data_url": "/main/module/api/update.dspy", "delete_data_url": "/main/module/api/delete.dspy", "form_cheight": 8, - "fields": [ - {"name": "field_name", "label": "Label", "uitype": "text", "required": true} - ] + "fields": [{"name": "field_name", "label": "Label", "uitype": "text", "required": true}] } } } ``` +`new_data_url`/`update_data_url` = form submission URLs; `delete_data_url` = POST URL sending `{params: row_data}`. -The DataViewer (dataviewer.js) uses these URLs: -- `new_data_url` - Form submission URL for adding records -- `update_data_url` - Form submission URL for editing records -- `delete_data_url` - POST URL for deleting records (sends `{params: row_data}`) +### Cross-Module Database Access — `get_sor_context` is the ONLY pattern +- **REQUIRED**: `env = request._run_ns`; `async with get_sor_context(env, "module_name") as sor:` then `await sor.R('table', {'filter': 'value'})`. It resolves module → DB via the `module_dbname` config (Sage: module "tenant" → `sage` DB; pipeline-app: → `pipeline` DB), configured per deployment. +- ❌ NEVER hardcode DB names (`async with db.sqlorContext("pipeline")` breaks cross-deployment portability — the most common cross-module DSPY error). +- ❌ NEVER use `DBPools()` + `sqlorContext()` for cross-module access (that's for the current module's DB only). +- ❌ NEVER use `ServerEnv()` in .dspy — all server-env functions are injected as globals. +- If a cross-module function is registered via `load_{modulename}()` (e.g. `create_user_apikey` from dapi), call it directly: `create_user_apikey(sor, dappid, user_id, user_orgid)`. -## Common Pitfalls +### Batch Lookups with `$or` +`users = await sor.R('users', {'$or': [{'id': uid} for uid in user_ids]})`. Validate the ID list is non-empty first; chunk lists >100 items to avoid query complexity limits. -1. **Using `print()` instead of `return`** — `print()` writes to stdout and is NOT captured by ahserver. The framework expects `return` statements. Using `print()` causes `return data type error, ` because the script returns None. Real-world example: `top_models.dspy` had `print(json.dumps(models))` which returned None to the caller — fixed by changing to `return json.dumps(models, ensure_ascii=False, default=str)`. -2. **Import statements** - violates ahserver security model. All functions listed in the Server-Env table above are pre-loaded — including `getID`, `time`, `DictObject`, `partial`, `FileStorage`, `curDateString`, `timestampstr`. Never import them. If your module's function is needed in a .dspy, export it via `load_{modulename}()` in `init.py`. -3. **Jinja2 `.ui` files cannot execute Python** — `.ui` files are Jinja2 templates that render JSON, they cannot run database queries, async operations, or complex logic. When you need database access, convert to `.dspy` files. Example: `llmusage_ioinfo_display.ui` used `{% set sor = db.sqlorContext() %}` which failed with `NameError: name 'db' is not defined` — fixed by converting to `.dspy` with proper async database access. -4. **sqlPaging() performance pitfall** — `sor.sqlPaging(sql, ns)` wraps the SQL in a subquery `select count(*) from (...)` which is very slow for large tables (3-4 seconds). For better performance, separate count and data queries: -```python -# WRONG — sqlPaging is slow for large tables: -result = await sor.sqlPaging(sql, ns) +### Safe Attribute Access on SQLor Rows +SQLor returns row objects that may or may not support dict access. Use `getattr(user, 'orgid', '') or ''` (handles missing attribute AND None). For dict access use `dict(r) if hasattr(r, 'keys') else r`. -# CORRECT — separate count and data queries: -count_sql = f"SELECT count(*) as rcnt FROM table {where_clause}" -count_recs = await sor.sqlExe(count_sql, ns) -total = count_recs[0].rcnt if count_recs else 0 +## DSPY Code Review Checklist -data_sql = f"SELECT ... FROM table {where_clause} ORDER BY {sort} LIMIT {limit} OFFSET {offset}" -rows = await sor.sqlExe(data_sql, ns) -``` -5. **Extract reusable database operations to utility functions** — When multiple `.dspy` files need the same database operation (fetching a record, reading from FileStorage), create async functions in the module's `utils.py` and import them: -```python -# In module/utils.py: -async def get_record_by_id(record_id): - env = ServerEnv() - async with get_sor_context(env, 'module_name') as sor: - sql = "SELECT * FROM table WHERE id = ${id}$" - recs = await sor.sqlExe(sql, {'id': record_id}) - return dict(recs[0]) if recs else None +### Syntax & Security +- [ ] **No imports** — module DSPY files must have zero import statements. All needed names (`json`, `datetime`, `get_sor_context`, `DBPools`, `params_kw`, `request`, `uuid`, `time`, `os`, `DictObject`, `FileStorage`, logging functions) are pre-loaded. +- [ ] **No forbidden patterns** — no `eval()`, `exec()`, `__import__()`, `os.system()`, `subprocess`, `pickle.loads()`. +- [ ] **Valid Python AST** — bare `ast.parse()` rejects top-level `await`/`async with` ("await outside async function"); wrap first: `wrapped = 'async def __c__(params_kw, request, uid, org_id, json, DBPools, get_user, get_userorgid, get_module_dbname, getID, debug, sor, params_kw=None):\n' + '\n'.join(' ' + l if l.strip() else l for l in src.split('\n'))` then `ast.parse(wrapped)` (add injected names the file uses to the wrapper signature). **Mandatory after patching triple-quoted prompt constants** — a stray `"""` silently closes the string and dumps following prose as code; only ast.parse exposes it (caught live 2026-08 in cockpit_chat.dspy). +- [ ] **All branches return** — missing return → `return data type error, `. -# In .dspy file: -from module.utils import get_record_by_id -record = await get_record_by_id(record_id) -``` -6. **FileStorage requires realPath() for file I/O** — FileStorage stores files with webpath references, but actual file operations need the filesystem path: -```python -from ahserver.filestorage import FileStorage -import aiofiles +### SQL & Database +- [ ] **Parameterized queries** — `${param}$` syntax, never f-string interpolation or `%s` in SQL strings. +- [ ] **Decimal / SUM aggregate safety** — MySQL `SUM()` returns `Decimal`; wrap with `int()` (e.g. `int(r.total_size or 0)`) or pass `default=str` in `json.dumps()`. +- [ ] **Cross-module access** — `get_sor_context(env, 'module')`, not `DBPools().sqlorContext(dbname)`. +- [ ] **sqlExe return type awareness** — no `page`/`rows` in ns → list of row objects (`r.field` attrs); with `page`/`rows` → `{'total': N, 'rows': [...]}` dict. +- [ ] **Error handling** — at least try/except around DB ops with a fallback return. -async def read_storage_file(webpath): - fs = FileStorage() - real_path = fs.realPath(webpath) # Convert webpath to filesystem path - async with aiofiles.open(real_path, 'rb') as f: - return await f.read() -``` -3. **Implicit None return** - If any code path doesn't hit a `return` statement, ahserver throws `return data type error, `. **Every branch must end with `return result`.** +### Code Quality (KISS/DRY) +- [ ] **Sibling file consistency** — inconsistent return format (raw dict vs `json.dumps()`), divergent helper signatures, or different API patterns across `.dspy` files in one directory are red flags. +- [ ] **DRY — no duplicated helpers** — size formatters (`fmt_size`, `fmt`), date formatters, SQL builders duplicated across files → extract. +- [ ] **No hardcoded config values** — storage limits, API URLs, timeouts come from config. +- [ ] **f-string safety** — avoid f-strings in dict returns (see Pitfall: f-string braces); use `'prefix: ' + str(var)`. +- [ ] **No `print()`** — use `return` (see Core Rule 2). -4. **CRITICAL: Debug `NoneType` errors at the error location, NOT by adding broad try/except** — When a .dspy endpoint returns `return data type error, `, open the .dspy file FIRST. Do NOT start by adding try/except wrappers in Python functions, modifying database connections, or adjusting SQL. The error trace points directly at the failing .dspy — examine its format (JSON `{"python": {...}}` vs Python script), verify function registration, and check return paths. Broad `except Exception: return []` masks real errors and makes debugging impossible. +### Return Format +- [ ] **Consistent return style per directory** — either raw dict `return {...}` or `json.dumps({...})`, same everywhere. +- [ ] **DataViewer CRUD endpoints** — return `Message` widget JSON, not raw data. +- [ ] **Code component endpoints** — return `[{value, text}]` array. +- [ ] **JSON validity** — `Decimal`, `datetime`, `bytes` types break `json.dumps` without `default=str`. -5. **JSON-format vs Python-script-format DSPY** — `return data type error, ` is especially common with JSON-format DSPY files (`{"python": {"import": "...", "call": "..."}}`). The JSON-format processor handles `None` returns differently from Python-script format (`import json; data = await func(request); return json.dumps(data)`). If one DSPY in a module uses JSON format while all others use Python script format, it's likely a format inconsistency bug. Always check file format when debugging NoneType errors. This is the single most common dspy error — the dspy sets `result` in branches but forgets the final `return result` at module level. Even a trivial dspy like `result = {"text": "hello"}` will return None without an explicit `return result`. +## Testing and Validation +- Test endpoints directly: `http://localhost:8000/app-name/entity_name/list/`. +- Format expectations: code components → array of `{value, text}`; DataViewer → array of full record objects; Forms → single record or success/error object. +- Test error scenarios: missing params, invalid IDs, DB failures. -**Pattern for multi-branch dspy**: put `return result` at the very end, OUTSIDE all if/elif blocks: +## Integration with Bricks Framework (.ui) +- Reference endpoints: `{"uitype": "code", "data_url": "/app-name/entity_name/list/"}`. +- Pass params via query string: `{"data_url": "/app-name/entity_name/get/?id={{selectedRow.id}}"}`. -```python -if not user: - result = {...} -elif code: - result = {...} -else: - result = {...} +## Common Pitfalls (deduped, all known) -return result # ← REQUIRED, outside all branches -``` -4. **Wrong data format** - code components need `{value, text}` arrays -5. **Missing error handling** - causes 500 errors instead of graceful degradation -6. **Returning wrapper objects unnecessarily** - most components expect direct data -7. **SQL column mismatch with DDL** - SELECT columns in .dspy files MUST exactly match actual database schema. Always verify with `DESCRIBE table_name` before writing queries. DDL files may differ from deployed schema. -8. **CGI-style .dspy files** — Never use `os.environ`, `sys.stdin`, `os.read(0, ...)`, `print()`, or `asyncio.new_event_loop()`. Use `params_kw`, `sqlorContext`, `return`, and let ahserver handle the async context. **ahserver automatically parses ALL request data (query string + POST body, including JSON `application/json`) into `params_kw`** — no manual reading of stdin or `os.read(0, content_length)` needed. JSON POST bodies are preserved as nested dict/list structures: `params_kw.get('user', {})` returns the nested user object. ❌ Never write `content_length = int(os.environ.get('CONTENT_LENGTH', 0)); raw_data = os.read(0, content_length); post_data = json.loads(raw_data)` in a .dspy file. -9. **DataViewer CRUD endpoints returning raw JSON** - Create/update/delete endpoints called by DataViewer editable forms must return a `Message` widget JSON structure, not raw data dictionaries. -10. **Dict merge syntax `{**a, **b}` fails** - Use `dict(list(a.items()) + list(b.items()))` instead for merging parameter dictionaries in .dspy files. -11. **sqlExe return type depends on parameters** - `sor.sqlExe(sql, ns)` returns different types: - - **WITHOUT `page`/`rows` in ns**: returns a **list** of row objects — do NOT treat as dict (`ret['key']` will fail with TypeError) - - **WITH `page`/`rows` in ns**: returns a **dict** `{'total': N, 'rows': [...]}` — do NOT iterate directly as list - - Always check type or build result manually: - ```python - rows = await sor.sqlExe(sql, ns) # no page/rows - result = {'total': len(rows), 'rows': rows, 'stats': stats} - return json.dumps(result, ensure_ascii=False, default=str) - ``` -12. **Row objects need safe conversion** - sqlExe returns row objects that may or may not have `.keys()` method. Use `dict(r) if hasattr(r, 'keys') else r` for safe conversion. -13. **Sort column must exist in table** - sqlExe uses the `sort` parameter for ORDER BY. If the specified column doesn't exist in the table, query fails. Default 'id' may not always be available. -14. **API file location matters** - List API `.dspy` files must be in `wwwroot/api/` subdirectory (e.g., `wwwroot/api/customers_list.dspy`), while UI `.ui` files go directly in `wwwroot/`. -15. **Session expiration during testing** - Cookie sessions expire after `session_max_time` (default 3600s). Re-login via `/main/login.dspy?username=xxx&password=xxx` before testing if getting 401 errors. -22. **Connection pool dirty reads (multiserver)**: When multiple service instances share the same MySQL, a connection pool bug can cause `get_*.dspy` to read records that were just deleted by another instance. Root cause: `aiomysql.connect()` defaults to `autocommit=False`, and `sqlorContext` only calls `commit()` for writes (not reads). When a connection is reused from the pool, its REPEATABLE READ snapshot from a previous SELECT persists. Fix: in `mysqlor.enter()`, call `await self.conn.commit()` to end any lingering transaction before reuse. See sqlor repo commit `fab420c`. Symptom: high-frequency "get reads deleted record" reports in multi-instance deployments. -24. **CRITICAL: Filter NaN/null/empty before MySQL INSERT/UPDATE** — When receiving numeric parameters from bricks `UiFloat` widgets via `urlwidget` + `datawidget: "self"`, empty or invalid inputs may send `NaN`, `null`, or empty strings. MySQL cannot handle `nan` floats: `OperationalError: nan can not be used with MySQL`. Always sanitize: - -```python -discount_val = params_kw.get('discount') -if discount_val is not None: - s = str(discount_val).strip().lower() - if s in ('', 'nan', 'none', 'null'): - discount_val = None -``` - -Apply this to ALL numeric parameters from user input before `float()` conversion or sor.C/U. Also applies to `old_discount` comparison values sent as static params. - -23. **CRITICAL: SQL parameter syntax** - Use `${param}$` in SQL strings, NOT `%(param)s`. The `${param}$` placeholder is replaced by sqlor with proper escaping. Using `%(param)s` causes "format requires a mapping" errors. Example: `await sor.sqlExe("INSERT INTO t (col) VALUES (${col}$)", {'col': value})`. -17. **Optional DATE fields** - MySQL DATE columns reject empty strings `''`. Convert empty form values to `None`: `sign_date = params_kw.get('sign_date', '').strip() or None`. -18. **Safe row attribute access** - SQLor row objects may lack `.keys()` or dict access. Use `getattr(row, 'field', '') or ''` instead of `row.field` or `row['field']` to avoid AttributeError on missing/None fields. -19. **$or batch queries** - For looking up multiple records by ID, build `$or` conditions: `{'$or': [{'id': uid} for uid in ids]}`. Validate the ID list is non-empty before querying. -20. **CRITICAL: ServerEnv() forbidden in .dspy AND in Python helper functions** — The ahserver framework injects all necessary functions directly into the .dspy execution context via globals. **Never write `env = ServerEnv()` in a .dspy file.** Correct usage: `dbname = get_module_dbname('dapi')`, `db = DBPools()`, `create_apikey_func = create_user_apikey`. Using `getattr(env, 'func_name', None)` or `config = getConfig(); db.databases = config.databases` is also wrong — these are all available as bare names. - -**For Python functions in `init.py` called from dspy**: Use `env = request._run_ns`, NOT `env = ServerEnv()`. A bare `ServerEnv()` has no request binding — `get_user()`, `get_userorgid()`, `get_userid()` etc. will all be `None`. The correct pattern: - -```python -# ✅ CORRECT — request._run_ns has full request context -async def my_handler(request, params_kw): - env = request._run_ns - user_id = await env.get_user() # returns userid string - org_id = await env.get_userorgid() # returns orgid string - -# ❌ WRONG — bare ServerEnv() has no session/request binding -async def my_handler(request, params_kw): - env = ServerEnv() - user_id = await env.get_user() # None! 'NoneType' is not callable -``` - -**Symptom**: dspy returns 500, log shows `'NoneType' object is not callable` at calls like `env.get_userorgid()` or `env.get_user()`. - -25. **Bare function calls from `load_X()` registrations can be None in dspy context** — Functions registered via `env.func_name = func` in `load_discount()` (etc.) are placed on the `ServerEnv` singleton, which gets merged into the dspy execution namespace via `run_ns.update(ServerEnv())`. In practice, this merge can fail silently — the bare function name resolves to `None` in the dspy, producing `'NoneType' object is not callable`. When a bare function call returns this error, **use `request._run_ns.func()` instead of bare function calls:** - -```python -# ❌ Bare function call — may resolve to None in dspy context: -file_type = classify_file(file_name) # NameError or NoneType - -# ❌ Explicit import — PROHIBITED in .dspy (user-enforced rule): -from rag.pipeline import process_upload # BLOCKED - -# ❌ `request._run_ns.func()` — PROVEN UNRELIABLE in production (#6) -# ServerEnv registration in init_rag_module() does NOT propagate to DSPY exec context. -# Despite env.func = func being set correctly, request._run_ns.func is always None. -# -# ✅ THE ONLY RELIABLE PATTERN — inline all logic directly in the DSPY: -# Use only ahserver pre-loaded globals: json, uuid, DBPools, get_sor_context, -# request.read(), params_kw. For installed packages (PyPDF2, docx, pptx, openpyxl), -# import inline at point of use — these are venv-installed, not custom modules. - -env = request._run_ns -result = await env.process_upload(env, file_data, kb_id, folder_id, file_name) -return result -``` - -**Registration in init.py** (module's `init_rag_module` or `load_rag`): -```python -def init_rag_module(): - env = ServerEnv() - from .pipeline import process_upload - env.process_upload = process_upload - rf = RegisterFunction() - ... -``` - -**DSPY becomes a zero-import thin wrapper** (12 lines max): -```python -ns = params_kw.copy() -kb_id = ns.get('kb_id', '') -folder_id = ns.get('folder', '') -file_name = ns.get('file_name', 'upload.bin') -if not kb_id: - return json.dumps({"status": "error", "error": "kb_id required"}, ensure_ascii=False) -file_data = await request.read() -if not file_data: - return json.dumps({"status": "error", "error": "no file data"}, ensure_ascii=False) -env = request._run_ns -result = await env.process_upload(env, file_data, kb_id, folder_id, file_name) -return result -``` - -This pattern was verified on ragserver (yumoqing/rag.git) — bare function calls and explicit imports both fail; only `request._run_ns.func()` works reliably. Keep all business logic in Python modules (pipeline.py, utils.py); DSPY files are pure wire-up. - -```python -# ❌ May resolve to None in dspy context: -ret = await bind_customer(request, bind_params) # NoneType not callable - -# ✅ Reliable — explicit import bypasses namespace merge issues: -from discount.init import bind_customer, set_promote_discount -ret = await bind_customer(request, bind_params) # works -``` - -**Diagnosis**: create a minimal test .dspy: `result = {'text': str(type(bind_customer))}` — if output shows ``, the function isn't being found in the namespace. - -**VERIFIED DECISION (ragserver, 2026-07-29)**: ALL approaches were tested exhaustively: -1. `env.func = func` in `init_rag_module()` → `request._run_ns.func` always None in DSPY exec context -2. `from rag.pipeline import func` → blocked by user (imports not allowed in DSPY) -3. **Inline all logic directly in the DSPY** → the ONLY approach that works - -Use only ahserver pre-loaded globals (`json`, `uuid`, `DBPools`, `get_sor_context`, `request.read()`, `params_kw`, `request._run_ns.get_userorgid()`). For installed packages (`PyPDF2`, `docx`, `pptx`, `openpyxl`, `aiohttp`, `base64`), import inline at point of use — these are venv-installed packages, NOT custom module imports. The DSPY file becomes a self-contained script with zero custom imports. For building complete upload pipelines with text extraction + DB, put ALL logic in the DSPY — do NOT attempt to split across pipeline.py or module init.py. -26. **CRITICAL: f-string braces inside dict returns cause exec() parse error** — `exec()` interprets f-string `{e}`'s closing `}` as closing the outer dict, producing `SyntaxError: '{' was never closed`. - -```python -# ❌ exec() misreads the last } — thinks it closes the outer dict -return {"timeout": 5, "message": f"处理失败: {e}"} - -# ✅ Use string concatenation instead -return {"timeout": 5, "message": "处理失败: " + str(e)} -``` - -This also affects `exception(f'{var=}')` — the `=` inside `{var=}` is fine but the closing `}` before `)` triggers the same issue. Use `'prefix: ' + str(var)` for debug/exception calls too. — `sor.sqlExe(sql, ns)` without page/rows returns a list of **row objects** (like SimpleNamespace), not dictionaries. These objects support attribute access (`r.id`, `r.name`) but NOT dict access (`r['id']`, `r['name']`). Using dict access causes `TypeError: 'SimpleNamespace' object is not subscriptable`, which can be silently swallowed by `try/except` blocks, resulting in empty dropdowns or undefined values in the UI. - -**❌ Wrong (causes silent failure):** -```python -apps = await sor.sqlExe("select id, name from upapp", {}) -result = [{'value': r['id'], 'text': r['name']} for r in apps] # TypeError silently caught -``` - -**✅ Correct:** -```python -apps = await sor.sqlExe("select id, name from upapp", {}) -result = [{'value': str(r.id), 'text': r.name} for r in apps] # Attribute access -``` - -**Safe pattern with getattr:** -```python -apps = await sor.sqlExe("select id, name from upapp", {}) -result = [{'value': str(getattr(r, 'id', '')), 'text': getattr(r, 'name', '')} for r in apps] -``` - -**Why this matters:** When building dropdown data endpoints (like `get_upapps.dspy`), using dict access causes the endpoint to return an empty array `[]`, which makes dropdown fields show "undefined" in the UI. The error is invisible because the try/except catches it silently. - -## Module Deployment Workflow - -**CRITICAL**: Never edit code directly on test/production servers. All changes must follow this flow: - -1. Edit in local repo (`~/repos//`) -2. `git add` + `git commit` + `git push` -3. On test server: `git pull` in the module's directory -4. If server has no SSH key for git, scp changed files individually - -**Module directory structure** (Sage): -``` -/d/apitest/sage/ - pkgs/ - module_name/ ← git repo (for code) - wwwroot/ ← symlinked from ../../wwwroot/module_name - module_name/ ← Python package (copied to site-packages) - wwwroot/ - module_name -> ../pkgs/module_name/wwwroot ← symlink - py3/lib/python3.10/site-packages/ - module_name/ ← Python package (copied from pkgs during deploy) -``` - -Modules live under Sage's `pkgs/` directory, NOT under pipeline-app's `pkgs/`. Each module's `wwwroot/` is symlinked from Sage's main `wwwroot/`. Python code is copied to `site-packages/` for the Sage venv to find. - -When you cannot run direct database queries, create a temporary debug `.dspy` file to inspect table schemas: - -```python -# Debug: show table columns — no imports needed -result = {'keys': [], 'rows': []} -try: - dbname = get_module_dbname('module_name') - async with DBPools().sqlorContext(dbname) as sor: - ns = {'page': 1, 'rows': 50, 'sort': 'COLUMN_NAME'} - sql = "SELECT COLUMN_NAME, COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='dbname' AND TABLE_NAME='table_name'" - rows = await sor.sqlExe(sql, ns) - if isinstance(rows, dict): - rows = rows.get('rows', []) - if rows: - result['keys'] = list(dict(rows[0]).keys()) - result['rows'] = [list(dict(r).values()) for r in rows] - result['success'] = True -except Exception as e: - result['error'] = str(e) -return json.dumps(result, ensure_ascii=False, default=str) -``` - -Place in `wwwroot/api/debug_tables.dspy`, test via curl, then delete after getting schema info. - -## Best Practices Summary - -- ✅ Use `return` for all data responses -- ✅ Never use `import` statements -- ✅ Handle all exceptions gracefully -- ✅ Return component-appropriate data formats -- ✅ Validate all input parameters -- ✅ Keep .dspy files focused and minimal -- ✅ Only create .dspy files when standard CRUD endpoints are insufficient -- ✅ Follow consistent naming patterns (`/list/`, `/get/`, `/test/`, etc.) -- ✅ Use `getattr(row, 'field', '') or ''` for safe SQLor row attribute access -- ✅ Use bare `get_module_dbname('module')` for cross-module DB access (no `ServerEnv()` wrapper needed) -- ✅ Use `$or` conditions in sor.R for batch ID lookups +1. **`print()` instead of `return`** — stdout is ignored → `return data type error, `. (Real case: `top_models.dspy` fixed by `return json.dumps(models, ensure_ascii=False, default=str)`.) +2. **Import statements** — violates the ahserver security model; everything in the Server-Env table is pre-loaded. If a module function is needed, export it via `load_{modulename}()` in `init.py`. +3. **Jinja2 `.ui` files cannot execute Python** — they render JSON only; no DB queries/async/complex logic. (`llmusage_ioinfo_display.ui` used `{% set sor = db.sqlorContext() %}` → `NameError: name 'db' is not defined`; convert to `.dspy`.) +4. **`sqlPaging()` is slow for large tables** (3-4s; wraps SQL in `select count(*) from (...)`) — separate count + data queries instead. +5. **Extract reusable DB ops to async functions in module `utils.py`** — `async def get_record_by_id(...)` using `get_sor_context`; call from .dspy via `from module.utils import get_record_by_id` (note: see Pitfall 24 re import rules — module-utils imports are used in some modules, ragserver forbade them). +6. **FileStorage requires `realPath()` for file I/O** — `fs.realPath(webpath)` converts webpath → filesystem path before `aiofiles.open(...)`. +7. **Implicit None return** — EVERY branch must end with `return result`; a bare expression is local to the wrapped async function. **Multi-branch pattern**: set `result = {...}` in each if/elif/else, then a single `return result` at module level, OUTSIDE all branches. Even `result = {"text": "hello"}` returns None without explicit `return result`. +8. **Debug `NoneType` errors at the error location** — open the .dspy FIRST; do NOT start by adding broad try/except wrappers in Python functions or changing DB/SQL. `except Exception: return []` masks real errors. Check file format (JSON `{"python": {...}}` vs Python script) and return paths. +9. **JSON-format vs Python-script-format DSPY** — JSON format (`{"python": {"import": "...", "call": "..."}}`) handles None returns differently than script format. If one DSPY in a module uses JSON format while others use script format, it's likely the bug. This is the single most common dspy error: `result` set in branches but final `return result` forgotten. +10. **Wrong data format** — code components need `{value, text}` arrays; DataViewer needs full record arrays. +11. **Missing error handling** — causes 500s instead of graceful degradation. +12. **Returning wrapper objects unnecessarily** — most components expect direct data. +13. **SQL column mismatch with DDL** — SELECT columns MUST exactly match the deployed schema; verify with `DESCRIBE table_name`. +14. **CGI-style .dspy forbidden** — never `os.environ`, `sys.stdin`, `os.read(0, ...)`, `print()`, `asyncio.new_event_loop()`. ahserver auto-parses ALL request data (query string + POST body, incl. JSON `application/json`) into `params_kw`; JSON bodies preserved as nested dict/list. +15. **DataViewer CRUD returning raw JSON** — must return `Message` widget JSON. +16. **Dict merge `{**a, **b}` fails** — use `dict(list(a.items()) + list(b.items()))`. +17. **sqlExe return type depends on ns** — WITH `page`/`rows` → dict `{'total': N, 'rows': [...]}` (don't iterate as list); WITHOUT → list of row objects (don't do `ret['key']`). +18. **Row objects need safe conversion** — `dict(r) if hasattr(r, 'keys') else r`. +19. **Sort column must exist in table** — default 'id' may not always be available. +20. **API file location** — list API `.dspy` files go in `wwwroot/api/` (e.g. `wwwroot/api/customers_list.dspy`); `.ui` files go directly in `wwwroot/`. +21. **Session expiration during testing** — cookie sessions expire after `session_max_time` (default 3600s); re-login via `/main/login.dspy?username=xxx&password=xxx` on 401. +22. **Connection pool dirty reads (multiserver)** — `aiomysql.connect()` defaults to `autocommit=False`; `sqlorContext` only commits writes, so a reused pooled connection can keep a stale REPEATABLE READ snapshot → "get reads deleted record". Fix: in `mysqlor.enter()`, `await self.conn.commit()` to end lingering transactions before reuse. (Symptom: high-frequency stale reads in multi-instance deployments.) +23. **Filter NaN/null/empty before MySQL INSERT/UPDATE** — bricks `UiFloat` widgets via `urlwidget` + `datawidget: "self"` may send `NaN`, `null`, or `''` → MySQL `OperationalError: nan can not be used with MySQL`. Sanitize ALL numeric params before `float()` or sor.C/U: `s = str(v).strip().lower(); if s in ('', 'nan', 'none', 'null'): v = None`. +24. **SQL parameter syntax** — use `${param}$`, NOT `%(param)s` ("format requires a mapping" errors). +25. **Optional DATE fields** — MySQL DATE columns reject `''`; convert empty form values to None: `params_kw.get('sign_date', '').strip() or None`. +26. **Safe row attribute access** — `getattr(row, 'field', '') or ''`, not `row.field`/`row['field']`. +27. **CRITICAL: `ServerEnv()` forbidden in .dspy AND in Python helpers** — all functions are injected globals; `env = ServerEnv()` / `getattr(env, 'func_name', None)` / `getConfig(); db.databases = config.databases` are all wrong. For Python functions in `init.py` called from dspy: `env = request._run_ns` (bare `ServerEnv()` has no request binding — `get_user()`, `get_userorgid()` return None → 500 `'NoneType' object is not callable`). Correct: `user_id = await env.get_user()`. +28. **Bare function calls from `load_X()` registrations can be None in dspy** — ServerEnv-singleton merge into the exec namespace can fail silently → `'NoneType' object is not callable`. VERIFIED DECISION (ragserver, 2026-07-29): (a) `env.func = func` in init → `request._run_ns.func` always None; (b) `from rag.pipeline import func` → blocked (imports not allowed); (c) **inline all logic directly in the DSPY** is the ONLY reliable approach — use only pre-loaded globals; import installed venv packages (PyPDF2, docx, pptx, openpyxl, aiohttp, base64) inline at point of use (venv packages, not custom modules). DSPY = self-contained, zero custom imports. (Contrast: discount module verified `from discount.init import bind_customer, set_promote_discount` + `await bind_customer(request, bind_params)` works.) **Diagnosis**: minimal test dspy `result = {'text': str(type(bind_customer))}` — `` means the name isn't in the namespace. +29. **CRITICAL: f-string braces inside dict returns cause exec() parse error** — `exec()` misreads `}` as closing the outer dict → `SyntaxError: '{' was never closed`. ❌ `return {"timeout": 5, "message": f"处理失败: {e}"}` → ✅ `return {"timeout": 5, "message": "处理失败: " + str(e)}`. Also affects `exception(f'{var=}')` — use concatenation for debug/exception calls too. +30. **sqlExe rows are SimpleNamespace-like** — attribute access only (`r.id`); dict access (`r['id']`) raises `TypeError: 'SimpleNamespace' object is not subscriptable`, often SILENTLY swallowed by try/except → empty dropdowns / "undefined" in UI. Use `getattr(r, 'id', '')`. ## Complex Logic: Move to Python, DSPY as Thin Wrapper - -When a .dspy needs to call module-internal classes or functions not registered on ServerEnv (e.g., `EmailClient`, `PROVIDERS`, provider methods), the DSPY will hit `NameError`. **Never add imports to the DSPY.** Instead: - -1. Add the logic as a method on a provider class (e.g., `TransferGateway.check_transfer()`) -2. Register the provider on ServerEnv: `env.PROVIDERS = PROVIDERS` -3. The DSPY becomes a thin wrapper: +When a .dspy needs module-internal classes/functions not registered on ServerEnv (e.g. `EmailClient`, `PROVIDERS`), the DSPY hits `NameError`. **Never add imports to the DSPY.** Instead: (1) add logic as a provider-class method (e.g. `TransferGateway.check_transfer()`), (2) register on ServerEnv (`env.PROVIDERS = PROVIDERS`), (3) DSPY becomes a thin wrapper: ```python provider = env.PROVIDERS.get('transfer') title, msg = await provider.check_transfer(tcode, env) return {"widgettype": "Message", "options": {"title": title, "message": msg}} ``` - -**This also avoids f-string brace issues** (pitfall 26) — the Python method can use f-strings freely; only the DSPY wrapper uses concatenation. +This also avoids the f-string brace issue (Pitfall 29) — Python methods can use f-strings freely; only the DSPY wrapper uses concatenation. ## add_startup Blocks Server — Use Manually Triggered Actions - -`add_startup(coro)` awaits the coroutine during server startup. If the coroutine is an infinite `while True` loop, **it blocks the server indefinitely**. Never use `add_startup` with an infinite loop or long-running polling. Instead, trigger actions manually (e.g., a button calling a DSPY endpoint) or use `asyncio.create_task()` inside the startup callback to spawn non-blocking background tasks. +`add_startup(coro)` awaits the coroutine during startup; an infinite `while True` loop **blocks the server indefinitely**. Never use it with infinite loops/long polling. Trigger manually (button → DSPY endpoint) or spawn non-blocking tasks via `asyncio.create_task()` inside the startup callback. ## How DSPY Execution Works (ahserver wraps in async function) - -**CRITICAL**: The ahserver framework wraps your .dspy code in an async function and awaits it: - -```python -# ahserver baseProcessor.py line ~234-243: -txt = "async def myfunc(request,**ns):\n" + '\n'.join(lines) -exec(txt, lenv, lenv) -func = lenv['myfunc'] -return await func(request, **lenv) -``` - -This means: -- `async with`, `await`, and `async for` DO work inside .dspy files -- You MUST use explicit `return` — the function's return value is what gets passed to the caller -- A bare expression (like `result` on the last line) inside an `async with` block does NOT reach the outer scope — it's local to the async function - -**❌ WRONG — bare expression, function returns None:** -```python -async with db.sqlorContext(dbname) as sor: - data = await sor.R('table', {}) - result = [dict(r) for r in data] - result # ← local to function, not returned -``` - -**✅ CORRECT — explicit return:** -```python -async with db.sqlorContext(dbname) as sor: - data = await sor.R('table', {}) - return [dict(r) for r in data] -return [] -``` - -This pattern is used extensively in the RBAC permission CRUD dspy files (e.g., `get_permission.dspy`) and our `get_tree_data.dspy` / `new_tree_item.dspy`, all of which work correctly. +ahserver wraps the code in `async def myfunc(request, **ns):` then `exec(txt, lenv, lenv)`, `return await func(request, **lenv)` (baseProcessor.py ~line 234). Consequences: +- `async with`, `await`, `async for` DO work inside .dspy. +- You MUST use explicit `return` — the function's return value is what reaches the caller. +- A bare expression (e.g. `result` on the last line) inside an `async with` block is local to the function → returns None. ❌ bare `result` → ✅ `return [...]` inside the block, `return []` after it. ## Async/Await — Fully Supported in DSPY - -**VERIFIED (2026-07-29, ragserver)**: `async with`, `await`, and `async for` ALL work inside .dspy files. The ahserver framework wraps your code in `async def myfunc(request, **ns):` and awaits it. The earlier prohibition was incorrect — extensive testing on the ragserver module confirmed all async patterns work: - -```python -# ✅ ALL of these work in DSPY: -async with get_sor_context(env, 'rag') as sor: - recs = await sor.sqlExe("SELECT ...", {}) -file_data = await request.read() -async with aiohttp.ClientSession() as s: - r = await s.post('https://...', json={...}) -``` - -**Symptom of real async issues**: 500 with `'NoneType' object is not callable` — this is almost always a ServerEnv registration failure (see Pitfall 25), NOT an async/sync problem. The function is None, not uncallable because of async context. - -**PITFALL: `params_kw` unavailable in some DSPY contexts** — when a `.dspy` file is accessed as a standalone page endpoint (like `/discount/promote.dspy`), `params_kw` may not be in scope. Use `request._run_ns.params_kw` instead: -```python -# ✅ Safe — works in all DSPY contexts -code = request._run_ns.params_kw.get('code', '') - -# ❌ May fail — params_kw not always available -code = params_kw.get('code', '') -``` - -**PITFALL: `binds` with `script` actiontype causes 500 in DSPY files** — when a DSPY returns widget JSON containing a `binds` array with `actiontype: "script"`, the server-side JSON parser may attempt to evaluate the script string as Python, causing 500 errors. Avoid including `binds` in DSPY widget output; keep them in static `.ui` templates instead. - async with DBPools().sqlorContext(dbname) as sor: - recs = await sor.R('discount_promo_code', {'id': promo_id}) - ... -``` - -**Pattern**: export the async function via `load_discount()` (`env.generate_promo_qr = generate_promo_qr`), then call it from the DSPY with `await func_name(request, params_kw)`. The DSPY stays a thin 2-line wrapper. - -**Symptom**: access to `.dspy` returns 500, server log shows `return data type error, ` and `'NoneType' object is not callable` from `auth_api.py`. - -**Note**: This prohibition does NOT apply to CRUD wrapper DSPY files (see "CRUD Wrapper Pattern" below) — those wrappers use `await` legitimately because they delegate to pre-registered async functions. +**VERIFIED (2026-07-29, ragserver)**: `async with`, `await`, `async for` all work (the earlier prohibition was incorrect). E.g. `async with get_sor_context(env, 'rag') as sor:` / `file_data = await request.read()` / `async with aiohttp.ClientSession() as s: r = await s.post(...)`. +- Symptom 500 `'NoneType' object is not callable` is almost always a ServerEnv registration failure (Pitfall 28), NOT an async/sync problem. +- **PITFALL: `params_kw` unavailable in some DSPY contexts** (standalone page endpoints like `/discount/promote.dspy`) — use `request._run_ns.params_kw.get('code', '')` instead. +- **PITFALL: `binds` with `actiontype: "script"` causes 500 in DSPY output** — the server-side JSON parser may evaluate the script string as Python. Keep `binds` in static `.ui` templates, never in DSPY widget JSON. +- Async functions can be exported via `load_discount()` (`env.generate_promo_qr = generate_promo_qr`) and awaited from the DSPY (`await func_name(request, params_kw)`) — DSPY stays a thin 2-line wrapper. (Note: this contradicts Pitfall 28's ragserver finding — ServerEnv registration propagation varies by module; verify with the type() diagnosis.) ## CRUD Wrapper Pattern (Legitimate Exception) - -When a module's `init.py` registers CRUD functions via `load_{module}()` (e.g., `env.create_tablename = create_tablename`), the `wwwroot/api/*.dspy` files are **thin wrappers** that delegate to those functions. These wrappers use `ServerEnv()` and `print()` — this is a legitimate exception to the "no ServerEnv in dspy" rule. - -**CRITICAL**: `json` is pre-loaded in ALL dspy contexts (including wrappers). Do NOT `import json` — it is redundant and will cause pre-commit audit failures. The only import needed is `from ahserver.serverenv import ServerEnv`: - +When `init.py` registers CRUD functions via `load_{module}()` (`env.create_tablename = create_tablename`), the `wwwroot/api/{table}_create.dspy` / `{table}_update.dspy` / `{table}_delete.dspy` files become **thin wrappers** that delegate to those functions. These wrappers may use `ServerEnv()` and `print()` — a legitimate exception to the no-ServerEnv rule. +**CRITICAL**: `json` is pre-loaded in ALL dspy contexts — do NOT `import json` (redundant, causes pre-commit audit failures). The only allowed import is `from ahserver.serverenv import ServerEnv`: ```python from ahserver.serverenv import ServerEnv env = ServerEnv() @@ -957,10 +244,32 @@ else: result = await create_func(request, params_kw) print(result) ``` +**Applies ONLY to** `wwwroot/api/{table}_create|update|delete.dspy` wrappers delegating to init.py-registered CRUD functions. Business-logic .dspy files (queries, calculations, cross-module ops) must follow the standard pattern (no imports, no ServerEnv, use `return`). -**When this pattern applies**: Only for `wwwroot/api/{table}_create.dspy`, `{table}_update.dspy`, `{table}_delete.dspy` files that delegate to init.py-registered CRUD functions. +## Module Deployment Workflow +**CRITICAL**: never edit code directly on test/production servers. +1. Edit in local repo (`~/repos//`) → 2. `git add` + `git commit` + `git push` → 3. On test server: `git pull` in the module dir → 4. If the server has no SSH key for git, scp changed files individually. -**When NOT to use**: Business logic .dspy files that do actual work (queries, calculations, cross-module operations) must follow the standard pattern (no imports, no ServerEnv, use return). +**Module directory structure** (Sage — modules live under Sage's `pkgs/`, NOT pipeline-app's): +``` +/d/apitest/sage/ + pkgs/module_name/ ← git repo (code) + wwwroot/ ← symlinked from ../../wwwroot/module_name + module_name/ ← Python package (copied to site-packages) + wwwroot/module_name -> ../pkgs/module_name/wwwroot + py3/lib/python3.10/site-packages/module_name/ +``` +Python code is copied to `site-packages/` for the Sage venv; `wwwroot/` is symlinked from Sage's main `wwwroot/`. + +**Schema inspection without direct DB access**: create a temp debug `.dspy` at `wwwroot/api/debug_tables.dspy` querying `information_schema.COLUMNS` (`SELECT COLUMN_NAME, COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='dbname' AND TABLE_NAME='table_name'` with `ns = {'page': 1, 'rows': 50, 'sort': 'COLUMN_NAME'}`), test via curl, delete after use. + +## Best Practices Summary +- ✅ Use `return` for all data responses; never `import`; handle exceptions gracefully. +- ✅ Return component-appropriate formats; validate all inputs. +- ✅ Keep .dspy files focused/minimal; only create them when standard CRUD endpoints are insufficient. +- ✅ Consistent naming (`/list/`, `/get/`, `/test/`, etc.). +- ✅ `getattr(row, 'field', '') or ''` for SQLor rows; `$or` in sor.R for batch ID lookups. +- ✅ Bare `get_module_dbname('module')` for cross-module DB access (no `ServerEnv()` wrapper). ## Linked References - `references/user-sync-pattern.md` — Cross-module user sync API pattern @@ -970,4 +279,4 @@ else: - `references/pipeline-app-setup.md` — Pipeline-app config, debugging, and KTV setup - `references/sage-crontab-etl-pattern.md` — Cron DSPY endpoints + build.sh crontab + j2_ stat cards - `references/cross-table-column-pitfalls.md` — Column name mismatches across Sage tables (userorgid vs orgid) + catelogid length + GROUP BY ambiguity -- sqlor-database-module skill `references/dapi-table-architecture.md` — Full dapi module table structure \ No newline at end of file +- sqlor-database-module skill `references/dapi-table-architecture.md` — Full dapi module table structure diff --git a/skills_library/all/harnessed-module-development/SKILL.md b/skills_library/all/harnessed-module-development/SKILL.md index 74fe98f..76a81d8 100644 --- a/skills_library/all/harnessed-module-development/SKILL.md +++ b/skills_library/all/harnessed-module-development/SKILL.md @@ -15,45 +15,30 @@ linked_files: **Before modifying any existing Sage module, read and understand the complete existing implementation first.** -Pitfall: When asked to add a feature to an existing module (e.g., credit limit to accounting), the agent may: -- Invent new logic instead of extending existing patterns -- Modify the wrong layer (e.g., sageapi vs sage/pkgs/accounting) -- Miss integration points that already exist +Pitfall: When asked to add a feature to an existing module (e.g., credit limit to accounting), agents may invent new logic instead of extending existing patterns, modify the wrong layer (sageapi vs sage/pkgs/accounting), or miss existing integration points. Correct workflow: 1. Load relevant skills (e.g., `accounting-module-example`) to understand architecture 2. Read ALL related source files to understand existing patterns 3. Identify where the new feature hooks into existing code (e.g., `leg_accounting()` for credit limit) 4. Follow existing patterns exactly — don't invent new approaches -5. After implementation, verify against a checklist of all integration points +5. Verify against a checklist of all integration points -User correction example: "accounting有一套完整的记账功能，只是需要你读懂并增加信用额度能力" — the agent should have read the existing accounting module first, not invented new balance-update logic. +User correction example: "accounting有一套完整的记账功能，只是需要你读懂并增加信用额度能力" — read existing code first, never invent new balance-update logic. ## External Vendor API Integration (uapi Gateway) -**When a module needs to call external vendor/third-party APIs, use the Sage uapi gateway (`UpAppApi`) instead of writing direct HTTP client code.** This is the platform-native pattern. - -Details: `references/uapi-gateway-integration.md` - -Key pattern: -- Module config table maps `vendor → upappid + api_mapping(JSON)` -- `UpAppApi.call(upappid, apiname, callerid, params)` routes through uapi templates -- AK/SK managed via uapi's `upappkey` table, not in the calling module -- Different vendors have different API counts/logic — `api_mapping` is flexible JSON, zero code changes to add vendors - -**Do NOT write direct HTTP clients** (e.g., `volcengine_client.py`) — they duplicate uapi's signing, templating, and streaming capabilities. +**When a module needs to call external vendor/third-party APIs, use the Sage uapi gateway (`UpAppApi`) instead of writing direct HTTP client code.** This is the platform-native pattern. Key pattern: module config table maps `vendor → upappid + api_mapping(JSON)`; `UpAppApi.call(upappid, apiname, callerid, params)` routes through uapi templates; AK/SK lives in uapi's `upappkey` table (not the calling module); `api_mapping` is flexible JSON — zero code changes to add vendors. **Do NOT write direct HTTP clients** (e.g. `volcengine_client.py`) — they duplicate uapi's signing, templating, and streaming capabilities. Details: `references/uapi-gateway-integration.md`; full pattern in "External API Design Patterns" below. ## Module Architecture -- Reasoning layer handles context analysis, task decomposition, safety checks, planning -- Execution layer handles tool calls, memory, skills, workflows, remote skills +- Reasoning layer: context analysis, task decomposition, safety checks, planning +- Execution layer: tool calls, memory, skills, workflows, remote skills - Reasoning feeds execution plans to agent module - Shared database schema, common RBAC auth, complementary APIs ## Module Permission Registration: scripts/load_path.py Pattern -**CRITICAL: Each module's RBAC permissions are managed in the module's own `scripts/load_path.py` file, NOT in `sage/load_path.py`.** - -The `sage/load_path.py` is a legacy artifact. All new modules (and existing ones migrated) register permissions via their own `scripts/load_path.py`: +**CRITICAL: Each module's RBAC permissions are managed in the module's own `scripts/load_path.py`, NOT in `sage/load_path.py`** (the latter is a legacy artifact). Follow the `product_management/scripts/load_path.py` pattern: ``` module_name/ @@ -66,80 +51,49 @@ module_name/ ### scripts/load_path.py Template -Follow the `product_management/scripts/load_path.py` pattern: - ```python #!/usr/bin/env python3 -""" -module_name 模块 RBAC 权限管理脚本 - -使用方法: - cd ~/repos/sage - ./py3/bin/python ~/repos/module_name/scripts/load_path.py -""" - +"""module_name RBAC 权限脚本. Usage: cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py""" import subprocess, os, sys def find_sage_root(): - candidates = [ - os.path.expanduser("~/repos/sage"), - os.path.expanduser("~/sage"), - ] - for c in candidates: + for c in [os.path.expanduser("~/repos/sage"), os.path.expanduser("~/sage")]: if os.path.isdir(os.path.join(c, "py3")) and os.path.isdir(os.path.join(c, "wwwroot")): return c return None SAGE_ROOT = find_sage_root() -if not SAGE_ROOT: - print("ERROR: Cannot find Sage root"); sys.exit(1) - +if not SAGE_ROOT: print("ERROR: Cannot find Sage root"); sys.exit(1) PYTHON = os.path.join(SAGE_ROOT, "py3", "bin", "python") SET_PERM_SCRIPT = os.path.join(SAGE_ROOT, "set_role_perm.py") MOD = "module_name" PATHS_ANY = [f"/{MOD}/menu.ui"] - -PATHS_LOGINED = [ - f"/{MOD}", - f"/{MOD}/page.ui", - f"/{MOD}/api/endpoint.dspy", - # ... all module paths -] +PATHS_LOGINED = [f"/{MOD}", f"/{MOD}/page.ui", f"/{MOD}/api/endpoint.dspy"] # + all module paths def run_set_perm(role, path): - cmd = [PYTHON, SET_PERM_SCRIPT, role, path] - return subprocess.run(cmd, capture_output=True, text=True).returncode == 0 + return subprocess.run([PYTHON, SET_PERM_SCRIPT, role, path], capture_output=True, text=True).returncode == 0 def register_role_paths(role, paths): count = sum(1 for p in paths if run_set_perm(role, p)) - print(f" {role}: {count}/{len(paths)} paths registered") - return count + print(f" {role}: {count}/{len(paths)} paths registered"); return count def main(): - total = 0 - total += register_role_paths("any", PATHS_ANY) - total += register_role_paths("logined", PATHS_LOGINED) + total = register_role_paths("any", PATHS_ANY) + register_role_paths("logined", PATHS_LOGINED) print(f"Done. Total {total} permission entries registered.") if __name__ == "__main__": main() ``` -### Running the Script +Run: `cd ~/repos/sage && ./py3/bin/python ~/repos/module_name/scripts/load_path.py` -```bash -cd ~/repos/sage -./py3/bin/python ~/repos/module_name/scripts/load_path.py -``` - -### Rules - -- **NEVER modify `sage/load_path.py`** for module permissions — use `module/scripts/load_path.py` -- Every new page, API, CRUD directory, and `.dspy` file needs a corresponding path entry +Rules: +- **NEVER modify `sage/load_path.py`** for module permissions +- Every new page, API, CRUD directory, and `.dspy` file needs a path entry - Paths follow URL convention: `/modulename/path` (no `wwwroot` in URLs) - Use `logined` for authenticated endpoints, `any` for public ones -- After running, restart Sage to reload RBAC cache +- Restart Sage after running to reload RBAC cache ## CRUD JSON: data_filter Pattern @@ -158,17 +112,12 @@ When a CRUD list needs search/filter functionality, define `data_filter` in the {"field": "upappid", "op": "=", "var": "upappid_input"} ] }, - "filter_labels": { - "name_input": "名称", - "model_input": "识别名", - "providerid_input": "供应商", - "upappid_input": "上位系统" - } + "filter_labels": {"name_input": "名称", "model_input": "识别名", "providerid_input": "供应商", "upappid_input": "上位系统"} } } ``` -### Backend .dspy: DBFilter Integration +Backend `.dspy` receives `data_filter` as a JSON string, parses it, and feeds it to `DBFilter`: ```python #!/usr/bin/env python3 @@ -176,146 +125,74 @@ import json from sqlor.filter import DBFilter result = {'success': False, 'rows': [], 'total': 0} - try: dbname = get_module_dbname('module_name') - page = int(params_kw.get('page', 1)) - rows_per_page = int(params_kw.get('rows', 20)) + page = int(params_kw.get('page', 1)); rows_per_page = int(params_kw.get('rows', 20)) offset = (page - 1) * rows_per_page - - # Parse data_filter JSON string from frontend - filterjson_str = params_kw.get('data_filter') - filterjson = None - if filterjson_str: - try: - filterjson = json.loads(filterjson_str) - except (json.JSONDecodeError, TypeError): - filterjson = None - + try: + filterjson = json.loads(params_kw.get('data_filter')) if params_kw.get('data_filter') else None + except (json.JSONDecodeError, TypeError): + filterjson = None async with DBPools().sqlorContext(dbname) as sor: - where_clause = '' - filterdic = {} + where_clause, filterdic = '', {} if filterjson: - # Preprocess LIKE values: add % wildcards if not already present ns = dict(params_kw) - for key, val in ns.items(): + for key, val in ns.items(): # auto-add % wildcards for LIKE vars if _is_like_var(filterjson, key) and val and '%' not in val: ns[key] = f'%{val}%' - - dbf = DBFilter(filterjson) - conds = dbf.gen(ns) - if conds: - where_clause = f' WHERE {conds}' - filterdic = ns - - # Count + paginated query using where_clause + filterdic - count_sql = f"select count(*) as cnt from tablename{where_clause}" - # ... execute count and data queries + conds = DBFilter(filterjson).gen(ns) + if conds: where_clause, filterdic = f' WHERE {conds}', ns + # count + paginated query: f"select count(*) as cnt from tablename{where_clause}" then data query result['success'] = True - except Exception as e: result['error'] = str(e) - return json.dumps(result, ensure_ascii=False, default=str) - def _is_like_var(filterjson, varname): """Check if a var is used with LIKE operator in the filter tree.""" - if not filterjson: - return False + if not filterjson: return False for key, val in filterjson.items(): if key.upper() in ('AND', 'OR') and isinstance(val, list): for item in val: - if _is_like_var(item, varname): - return True + if _is_like_var(item, varname): return True elif key.upper() == 'NOT' and isinstance(val, dict): - if _is_like_var(val, varname): - return True + if _is_like_var(val, varname): return True elif isinstance(val, dict) and val.get('var') == varname: - if val.get('op', '').upper() == 'LIKE': - return True + if val.get('op', '').upper() == 'LIKE': return True return False ``` -### data_filter Rules +Rules: +1. `data_filter` lives under `params`; uses `AND`/`OR`/`NOT` tree matching `sqlor.filter.DBFilter` +2. `var` names map to URL params from the frontend search form; `filter_labels` gives display labels +3. LIKE fields need `%` wildcards — auto-add on backend if not present +4. `DBFilter.gen(ns)` returns the WHERE clause; `ns` holds variable values +5. Dropdown fields (providerid, upappid) use `browserfields.alters` with `uitype: "code"` + `dataurl` for code table data -1. `data_filter` lives under `params` in the CRUD JSON -2. Uses `AND`/`OR`/`NOT` tree structure matching `sqlor.filter.DBFilter` -3. `var` names map to URL params sent by the frontend search form -4. `filter_labels` provides display labels for the search form fields -5. Backend `.dspy` receives `data_filter` as a JSON string, parses it, feeds to `DBFilter` -6. LIKE fields need `%` wildcards added on the backend (auto-add if not present) -7. `DBFilter.gen(ns)` returns the WHERE clause string; `ns` contains variable values -8. Dropdown fields (providerid, upappid) use `browserfields.alters` with `uitype: "code"` + `dataurl` for code table data +## CRITICAL: URL & Path Rules -## CRITICAL: menu.ui URL Must Match JSON Alias - -The `menu.ui` URLs must match the `alias` (or `tblname` if no alias) defined in the JSON CRUD files: - -| Menu URL | Must match JSON `alias` or `tblname` | -|---|---| -| `{{entire_url('/module/alias_name')}}` | JSON file must have `"alias": "alias_name"` or `"tblname": "alias_name"` | - -**Common mistake**: Using a short name in menu URL but a different name in JSON alias: -- WRONG: menu URL `/harnessed_agent/sessions` but JSON alias is `hermes_sessions` -- CORRECT: menu URL `/harnessed_agent/hermes_sessions` matching JSON alias `hermes_sessions` - -Also, `.ui` wrapper files that use `entire_url('crud_alias')` to load CRUD pages must use the exact alias from the JSON definition, not a different name. - -## CRITICAL: WSS WebSocket URL Routing — ALL paths include `/wss/` prefix - -Server logs confirm RBAC checks the FULL path including `/wss/` prefix: - -``` -[debug] userid=None, path='/wss/harnessed_reasoning/reasoning_console.wss' permission check failed -``` - -All paths MUST include `/wss/`: - -- **Frontend/UI `entire_url()`**: `{{entire_url('/wss/harnessed_reasoning/reasoning_console.wss')}}` -- **RBAC permission paths**: `/wss/harnessed_reasoning/reasoning_console.wss` -- **`set_role_perm.py` path arg**: `/wss/harnessed_reasoning/reasoning_console.wss` - -| WRONG | CORRECT | -|-------|---------| -| `{{entire_url('reasoning_console.wss')}}` (in UI) | `{{entire_url('/wss/harnessed_reasoning/reasoning_console.wss')}}` | -| `/harnessed_reasoning/reasoning_console.wss` (in RBAC) | `/wss/harnessed_reasoning/reasoning_console.wss` | - -## CRITICAL: .ui File References Must Not Include Module Prefix - -When a `.ui` file in `wwwroot/` references another `.ui` file in the same `wwwroot/`, use just the filename: -- WRONG: `{{entire_url('harnessed_agent/memory.ui')}}` (resolves to double-prefixed path) -- CORRECT: `{{entire_url('memory.ui')}}` (resolves correctly relative to current module) - -## CRITICAL: URL Path Rules in JSON Config +### menu.ui URL Must Match JSON Alias +The `menu.ui` URLs must match the `alias` (or `tblname` if no alias) defined in the JSON CRUD files: `{{entire_url('/module/alias_name')}}` requires JSON to have `"alias": "alias_name"` (or `"tblname": "alias_name"`). +- WRONG: menu URL `/harnessed_agent/sessions` but JSON alias `hermes_sessions` +- CORRECT: menu URL `/harnessed_agent/hermes_sessions` matching JSON alias +- `.ui` wrappers using `entire_url('crud_alias')` must use the exact alias from the JSON definition ### `wwwroot` is INVISIBLE in URLs The `wwwroot` directory is the document root — it NEVER appears in URL paths. +- WRONG: `{{entire_url('../wwwroot/api/xxx.dspy')}}` / `/module/wwwroot/page.ui` +- CORRECT: `{{entire_url('../api/xxx.dspy')}}` / `/module/page.ui` +- From `json/` directory: same-module refs use `../api/endpoint.dspy` or `../crud_alias`; cross-module uses absolute `/module_name/api/endpoint.dspy` +- CRUD subtables `url` must use `../` prefix: `"url": "{{entire_url('../handover_items_list')}}"` targeting the CRUD alias in another JSON file -| WRONG | CORRECT | -|-------|---------| -| `{{entire_url('../wwwroot/api/xxx.dspy')}}` | `{{entire_url('../api/xxx.dspy')}}` | -| `/module/wwwroot/page.ui` | `/module/page.ui` | +### .ui File References Must Not Include Module Prefix +When a `.ui` file in `wwwroot/` references another `.ui` in the same `wwwroot/`, use just the filename: `{{entire_url('memory.ui')}}` — NOT `{{entire_url('harnessed_agent/memory.ui')}}` (resolves to double-prefixed path). -### Relative paths from `json/` directory -When a JSON CRUD file in `json/` references files in `wwwroot/`: -- Same module: `../api/endpoint.dspy` or `../crud_alias` -- Cross module: absolute path `/module_name/api/endpoint.dspy` - -### CRUD subtables `url` pattern -```json -"subtables": [{ - "field": "handover_id", - "title": "明细", - "url": "{{entire_url('../handover_items_list')}}", - "subtable": "customer_handover_items" -}] -``` -The URL must use `../` prefix to escape the `json/` directory, targeting the CRUD alias defined in another JSON file. +### CRITICAL: WSS WebSocket URL Routing — ALL paths include `/wss/` prefix +Server logs confirm RBAC checks the FULL path including `/wss/` (e.g. `[debug] userid=None, path='/wss/harnessed_reasoning/reasoning_console.wss' permission check failed`). Use that path verbatim everywhere: frontend `entire_url()` → `{{entire_url('/wss/harnessed_reasoning/reasoning_console.wss')}}`; RBAC permission paths and `set_role_perm.py` args → `/wss/harnessed_reasoning/reasoning_console.wss`. WRONG: `/harnessed_reasoning/reasoning_console.wss` (no `/wss/`) → `permission check failed`. ## CRITICAL: `editable` Section Required -Every JSON list/crud definition MUST have an `editable` section. Without it, the framework doesn't know where to submit forms for create/update/delete operations. +Every JSON list/crud definition MUST have an `editable` section — without it the framework doesn't know where to submit create/update/delete forms: ```json { @@ -333,61 +210,62 @@ Every JSON list/crud definition MUST have an `editable` section. Without it, the } ``` -**Pitfall**: `wwwroot` directories contain `.ui` and `.dspy` files, but `json/` CRUD configs must reference them via relative paths that skip `wwwroot`. The framework resolves `../api/xxx.dspy` from the module root, not from `json/`. +Pitfall: `json/` CRUD configs must reference wwwroot files via relative paths skipping `wwwroot` — the framework resolves `../api/xxx.dspy` from the module root, not from `json/`. + +## CRUD JSON Strict Validation Checklist + +Validate EVERY field reference against the model definition in `models/`. Known mismatches: +| File | Wrong Field | Correct Field (from model) | +|------|-------------|----------------------------| +| opportunities_list.json | `org_id` | (does not exist — remove) | +| opportunities_list.json | `sales_stage` | `current_stage` | +| opportunities_list.json | `source` | `source_type` | +| sales_stages_list.json | `is_active` | `is_won_stage` / `is_lost_stage` | +| stage_history_list.json | `changed_by` | `changed_by_id` / `changed_by_name` | + +Every CRUD JSON MUST have: +1. `tblname` root key matching a table in `models/` +2. `params` with at least `sortby` and `browserfields` +3. `editable` with `new_data_url`/`update_data_url`/`delete_data_url` (even if read-only) +4. All field names in `browserfields.exclouded`, `browserfields.alters`, `editexclouded` must exist in the model +5. `alters` use `uitype: "code"` with `data` array — never nest `style` objects +6. `subtables[].url` uses `{{entire_url('../alias')}}` with `../` prefix +7. `editor.binds[].actiontype` ∈ {`urlwidget`, `method`, `script`, `registerfunction`, `event`} + +When adding model fields, ALWAYS update `init/data.json` seed data (missing fields → config gaps after fresh deployment). + +### Model float/decimal Fields +MUST have BOTH `length` (int) and `dec` (int) as separate numeric keys. WRONG: `"length": "15,2"` (string). CORRECT: `"length": 15, "dec": 2`. + +### ID Generation: Always Use `getID()`, Never `uuid.uuid4()` +`id` columns are VARCHAR(32); `str(uuid.uuid4()).replace('-','')` is a 32-char hex string that can exceed column length → `DataError: (1406, "Data too long for column 'id' at row 1")`: +```python +# WRONG: new_id = str(uuid.uuid4()).replace('-', '') +# CORRECT: +from appPublic.uniqueID import getID +new_id = getID() +``` +Applies to ALL Python backend code (core.py) AND `.dspy` API files — same scheme as the framework's `uniqueID` module. ## CRITICAL: TabPanel Correct Syntax -### Wrong: Using `Tab` widgettype with `tabs` parameter -```json -// WRONG — "Tab" widgettype doesn't exist, "tabs" is invalid -{ - "widgettype": "Tab", - "options": { - "tabs": [{"title": "Sessions"}, {"title": "Config"}] - } -} -``` +Widgettype `"TabPanel"` (NOT `"Tab"` — doesn't exist); parameter `items` array (NOT `tabs`); each item has `name`, `label`, `content`; `content` directly embeds a widget description object (e.g. urlwidget); `tab_pos`: `"top"` (default)/`"bottom"`/`"left"`/`"right"`. -### Correct: Using `TabPanel` widgettype with `items` parameter ```json -// CORRECT -{ - "widgettype": "TabPanel", - "options": { - "tab_pos": "top", - "items": [ - { - "name": "sessions", - "label": "推理会话", - "icon": "history", - "content": { - "widgettype": "urlwidget", - "options": { - "url": "{{entire_url('crud_alias_or_file.ui')}}" - } - } - } - ] - } -} +{"widgettype": "TabPanel", "options": {"tab_pos": "top", "items": [ + {"name": "sessions", "label": "推理会话", "icon": "history", + "content": {"widgettype": "urlwidget", "options": {"url": "{{entire_url('crud_alias_or_file.ui')}}"}}} +]}} ``` -Key rules: -- Widgettype: `"TabPanel"` (NOT `"Tab"`) -- Parameter: `items` array (NOT `tabs`) -- Each item: `name`, `label`, `content` -- `content`: directly embeds a widget description object (e.g., urlwidget) -- `tab_pos`: `"top"` (default), `"bottom"`, `"left"`, `"right"` - ## CRUD UI File Pattern -Tabular for list views: +Tabular (list views): ```json { "widgettype": "Tabular", "options": { - "width": "100%", - "height": "100%", + "width": "100%", "height": "100%", "data_url": "{{entire_url('api/list_endpoint.dspy')}}", "data_method": "GET", "page_rows": 20, @@ -395,29 +273,17 @@ Tabular for list views: "fields": [ {"name": "id", "width": 80, "frozen": true}, {"name": "field_name", "title": "中文标题", "width": 150}, - { - "name": "status", - "title": "状态", - "width": 100, - "uitype": "code", - "data": [ - {"value": "active", "text": "活跃"}, - {"value": "inactive", "text": "非活跃"} - ] - } + {"name": "status", "title": "状态", "width": 100, "uitype": "code", + "data": [{"value": "active", "text": "活跃"}, {"value": "inactive", "text": "非活跃"}]} ], "editexclouded": ["id", "created_at"] }, - "editable": { - "new_data_url": null, - "update_data_url": null, - "delete_data_url": null - } + "editable": {"new_data_url": null, "update_data_url": null, "delete_data_url": null} } } ``` -Form for config/edit views: +Form (config/edit views): ```json { "widgettype": "Form", @@ -429,47 +295,27 @@ Form for config/edit views: "method": "POST", "layout": "vertical", "fields": [...], - "buttons": [ - {"type": "submit", "label": "保存", "variant": "primary"} - ], + "buttons": [{"type": "submit", "label": "保存", "variant": "primary"}], "maxWidth": "500px" }, - "binds": [ - { - "wid": "self", - "event": "submited", - "actiontype": "script", - "script": "await bricks.show_resp_message_or_error(event.params)" - } - ] + "binds": [{"wid": "self", "event": "submited", "actiontype": "script", + "script": "await bricks.show_resp_message_or_error(event.params)"}] } ``` ## .dspy API Pattern -**CRITICAL: ahserver `.dspy` files return data via `return`, NOT `print()`.** Using `print()` sends output to stdout which ahserver does not capture — the caller receives `NoneType`. Always use `return json.dumps(...)`. +**CRITICAL: ahserver `.dspy` files return data via `return`, NOT `print()`** — print() goes to stdout, ahserver receives `NoneType`. Always `return json.dumps(result, ensure_ascii=False, default=str)`. -```python -# WRONG — print sends to stdout, ahserver receives None: -print(json.dumps(result)) - -# CORRECT — ahserver captures the return value: -return json.dumps(result, ensure_ascii=False, default=str) -``` - -List endpoint: +List endpoint (rows/total): ```python #!/usr/bin/env python3 import json - result = {'success': False, 'rows': [], 'total': 0} - try: dbname = get_module_dbname('module_name') user_id = await get_user() - sql = """SELECT id, name, status, created_at FROM table_name - WHERE user_id = ${user_id}$ ORDER BY created_at DESC""" - + sql = """SELECT id, name, status, created_at FROM table_name WHERE user_id = ${user_id}$ ORDER BY created_at DESC""" async with DBPools().sqlorContext(dbname) as sor: data = await sor.sqlExe(sql, {'user_id': user_id}) if isinstance(data, dict): @@ -479,564 +325,205 @@ try: result['rows'] = [dict(r) for r in (data or [])] result['total'] = len(result['rows']) result['success'] = True - except Exception as e: result['error'] = str(e) - return json.dumps(result, ensure_ascii=False, default=str) ``` -Get config endpoint: -```python -#!/usr/bin/env python3 -import json +Get config endpoint: same skeleton; `result = {'success': False, 'config': {}}`; `SELECT * FROM config_table WHERE user_id = ${user_id}$ LIMIT 1`; set `result['config'] = dict(rows[0])` or defaults. -result = {'success': False, 'config': {}} - -try: - dbname = get_module_dbname('module_name') - user_id = await get_user() - sql = """SELECT * FROM config_table WHERE user_id = ${user_id}$ LIMIT 1""" - - async with DBPools().sqlorContext(dbname) as sor: - rows = await sor.sqlExe(sql, {'user_id': user_id}) - if rows and len(rows) > 0: - config = dict(rows[0]) - result['config'] = config - else: - result['config'] = {...default values...} - result['success'] = True - -except Exception as e: - result['error'] = str(e) - -return json.dumps(result, ensure_ascii=False, default=str) -``` - -Save config endpoint (returns Message widget): +Save config endpoint (returns Message widget — both success and error): ```python #!/usr/bin/env python3 import json, uuid, time - result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid', 'type': 'error'}} - try: dbname = get_module_dbname('module_name') user_id = await get_user() now = time.strftime('%Y-%m-%d %H:%M:%S') - async with DBPools().sqlorContext(dbname) as sor: rows = await sor.sqlExe("SELECT id FROM table WHERE user_id = ${user_id}$", {'user_id': user_id}) if rows: await sor.sqlExe("UPDATE table SET ... WHERE id = ${id}$", {...}) else: await sor.sqlExe("INSERT INTO table ... VALUES (...)", {...}) - result = {'widgettype': 'Message', 'options': {'title': 'Success', 'message': '保存成功', 'type': 'success'}} - except Exception as e: result['options'] = {'title': 'Error', 'message': '保存失败: ' + str(e), 'type': 'error'} - return json.dumps(result, ensure_ascii=False) ``` -## Menu.ui Pattern - -Simplified menu with auth check: -```json -{ - "widgettype": "Menu", - "options": { - "target": "PopupWindow", - "popup_options": {"archor": "cc", "width": "70%", "height": "70%"}, - "cwidth": 10, - "items": [ -{% if get_user() %} - {"name": "entry_name", "label": "入口名", "url": "{{entire_url('page.ui')}}"} -{% endif %} - ] - } -} -``` - -Note: JSON validation will fail on `.ui` files with Jinja2 templates — this is expected and normal. - ## File Organization ``` module_name/ -├── json/ # CRUD metadata definitions -│ ├── table_crud.json # List/CRUD config with editable section -│ └── table_edit.json # Edit form config -├── models/ # Database table definitions -│ ├── table.json # Canonical JSON format (for CRUD + DDL) -│ └── table.xlsx # Original Excel source (multi-sheet) +├── json/ # CRUD metadata (table_crud.json list, table_edit.json form) +├── models/ # table.json (canonical, CRUD+DDL) + table.xlsx (source) ├── wwwroot/ -``` - -**Note**: `models/` may contain both `.json` and `.xlsx` files. JSON is the canonical format used by CRUD and DDL generation. Convert xlsx→json with `~/repos/sage/xlsx2json_models.py`. See `database-table-definition-spec` skill for format details. -│ ├── menu.ui # Module menu -│ ├── index.ui # Main page -│ ├── page.ui # UI components -│ ├── table_crud.ui # Generated or manual CRUD UI -│ └── api/ -│ ├── table_list.dspy # Data list API -│ ├── table_create.dspy # Create API -│ ├── table_update.dspy # Update API -│ └── table_delete.dspy # Delete API +│ ├── menu.ui, index.ui, page.ui, table_crud.ui +│ └── api/ # table_list/create/update/delete.dspy └── module_name/ ├── __init__.py # Empty - ├── init.py # load_module() function, registers to ServerEnv + ├── init.py # load_module() registers to ServerEnv └── core.py # Business logic ``` -## Debugging Reference +Convert xlsx→json with `~/repos/sage/xlsx2json_models.py` (see `database-table-definition-spec` for format). -See `references/module-init-deferred-io.md` for robust module initialization patterns — deferred I/O, graceful degradation, and per-subsystem try/except wrapping. +## Debugging Reference (references/) -See `references/websocket-ui-debugging.md` for: -- Step-by-step WebSocket UI diagnosis (log verification, curl template check, widget conflict detection) -- Server log pattern recognition: `WS Registered` in logs means backend is fine — problem is frontend -- Common root cause table for "等待连接" symptoms -- Browser console error decoding: `wid not find`, `HTML not registered`, `ReferenceError` patterns - -See `references/sage-login-debugging.md` for: -- Login flow: rfexe('password', params_kw) RC4 encryption, users table query, remember_user -- Common failures: commented-out rf.register, silent password mismatch, RBAC 401 -- RBAC cache requires Sage restart after set_role_perm.py -- Debugging checklist: app/rf.py, conf/config.json password_key, server log patterns - -See `references/bricks-ui-pitfalls.md` for: -- Widgettype casing: `Html` NOT `HTML`; `Scroll` does not exist (use VBox + CSS) -- Form buttons cannot have external binds — use standalone Button widgets -- Multi-line input via `uitype: "text"` + `height` on Form fields -- Raw JS WebSocket preferred over `bricks.WebSocket` widget (avoids ReferenceError timing issues) -- Session passing: `bricks.app.get_session()` → WebSocket sub-protocol - -See `references/sage-session-debugging.md` for: -- Session diagnosis checklist: auth_api.py truncation, Redis, wwwroot diff, Jinja2 template calls, URL resolution in user_panel.ui, RBAC permissions -- Architecture overview: remember_user() → auth.remember() → ticket → Redis session storage -- Known root causes: user_panel.ui path change, global_menu.ui get_user_roles() at template level - -See `references/sage-auth-and-websocket.md` for: -- Cookie-based session flow and Redis storage -- WebSocket authentication via Sec-WebSocket-Protocol -- RBAC permission path formats (with/without /wss/) -- Common auth pitfalls (cookie secure flag, RBAC blocking, field names) -- Users table schema and login field names -- Role types (anonymous, any, logined, owner) -- Debugging commands for auth issues - -See `references/sage-deployment-architecture.md` for: -- sage/wwwroot/ is gitignored — all content is symlinks from module repos -- global_menu.ui architecture (lives in dashboard_for_sage, symlinked to sage/wwwroot/) -- Module discovery technique for global_menu (scanning wwwroot/index.ui) -- bricks/ static files deployment (header.tmpl, footer.tmpl, JS, CSS) -- Error diagnosis: `'NoneType' object has no attribute 'be_call'` = missing bricks static file -- Production deployment checklist for new module content - -See `references/sage-background-coroutines.md` for: -- Registry of all modules using `add_cleanupctx` / `add_startup` hooks -- Multi-process safety matrix (which hooks must be extracted) -- Standalone program pattern for background tasks -- Cookie secure/samesite pitfall for HTTP development -- RBAC permission for login endpoints -- Login form field names and database schema -- **Sage wwwroot symlink requirement** — new module files must be linked -- **Git branch strategy** — main branch only, no feature branches -- **Self-testing requirement** — verify before reporting - -See `references/sage-testing-environment.md` for: -- Remote and local test environment details (URLs, credentials, database) -- Browser testing prerequisites (Chrome CDP, Redis, server status) -- 502 Bad Gateway troubleshooting flowchart -- RBAC permission testing and cache refresh -- Password encoding and LLM configuration patterns - -See `references/sage-i18n-system.md` for: -- Sage i18n architecture (MiniI18N backend + bricks.js frontend) -- msg.txt encoding rules and file locations -- Scanning strategy: MUST use regex fallback for template-containing .ui files -- CRUD, .dspy, and dashboard string extraction patterns - -See `references/sage-environment.md` for: -- Password encoding/decoding patterns -- Common Sage commands (startup, RBAC, DDL) -- harnessed_agent and harnessed_reasoning required tables -- Known issues and workarounds - -See `references/vendor-callback-pattern.md` for: -- External vendor webhook/callback endpoint patterns (`.dspy` + Python handler) -- Vendor POST parsing (JSON body + query params fallback) -- Idempotency handling for retried callbacks -- RBAC `paths_any` registration for vendor endpoints -- Client query API pattern for authenticated resource lookups -- **Bearer token auth model**: dapi auto-resolves user_id/org_id, no manual `downapp_id` needed -- **Never expose internal IDs** to client APIs — only vendor-side identifiers - -See `references/debugging-guide.md` for: -- Deployed code vs source code identification -- Debug logging patterns (`info()` vs `debug()`) -- Common error: `'NoneType' object has no attribute 'get'` -- Non-JSON LLM API response handling -- Encrypted API key decryption - -**Verifying Jinja2 template rendering via curl** - -When a `.ui` file uses Jinja2 templates like `{{ get_user() }}`, verify they render correctly: +- `module-init-deferred-io.md` — deferred I/O, graceful degradation, per-subsystem try/except +- `websocket-ui-debugging.md` — WS diagnosis; `WS Registered` = backend fine, problem frontend; "等待连接" causes; `wid not find`/`HTML not registered`/`ReferenceError` +- `sage-login-debugging.md` — login flow (RC4, users, remember_user); silent password mismatch; RBAC 401; restart after set_role_perm.py +- `bricks-ui-pitfalls.md` — `Html` NOT `HTML`; no `Scroll` (VBox+CSS); Form buttons no external binds; `uitype:"text"`+height; raw JS WS; get_session() +- `sage-session-debugging.md` — remember_user()→auth.remember()→ticket→Redis; wwwroot diff; Jinja2 calls +- `sage-auth-and-websocket.md` — cookie+Redis; WS auth Sec-WebSocket-Protocol; RBAC path formats; roles; users schema +- `sage-deployment-architecture.md` — wwwroot gitignored (symlinks); global_menu.ui in dashboard_for_sage; `'NoneType'...be_call` = missing bricks static +- `sage-background-coroutines.md` — add_cleanupctx/add_startup registry; safety matrix; standalone pattern; main-branch only +- `sage-testing-environment.md` — test URLs/credentials; Chrome CDP; 502 flowchart; RBAC cache refresh +- `sage-i18n-system.md` — MiniI18N+bricks.js; msg.txt encoding; regex fallback for template .ui files +- `sage-environment.md` — password encoding; commands; required tables +- `vendor-callback-pattern.md` — vendor webhook `.dspy`; POST parsing (JSON+query fallback); idempotency; paths_any; Bearer auto-resolves (no downapp_id); no internal IDs +- `debugging-guide.md` — deployed vs source; info() vs debug(); 'NoneType' get; non-JSON LLM; key decryption +- `event-dispatcher-cache-pattern.md` — sqlor C/U/D events; LRU+TTL; naming `{db}:{tbl}:{c|u|d}:{before|after}`; RBAC `this`/`self` bug +- `user-isolation-pattern.md` — context chain; users/{user_id}/; _get_user_dir(); per-user ws_push; hermes_skills +- `reasoning-visualization.md` — event flow; per-user WS callback; skill search +- `multi-process-cache-invalidation.md` — Redis Pub/Sub pattern +### Verifying Jinja2 template rendering via curl +When a `.ui` file uses Jinja2 templates like `{{ get_user() }}`, verify rendering: ```bash -# Unauthenticated request (no session cookie) — get_user() renders as 'None' curl -s http://localhost:9180/module/page.ui | grep -o "user_id:.*" | head -1 - -# The rendered output should show the template was processed: -# user_id: 'None'})); <-- correct (not logged in, get_user() returns None) -# user_id: '{{ get_user() }}' <-- WRONG (template not processed at all) -# user_id: 'current_user' <-- WRONG (hardcoded string, not a template) +# 'user_id: None' = correct (not logged in); raw '{{ get_user() }}' = template NOT processed +# (served as static file, not through ahserver renderer); 'current_user' = hardcoded, wrong ``` -If the raw `{{ get_user() }}` appears in the curl output, the template is not being processed — check that the file is served through ahserver's template renderer, not as a static file. - -See `references/event-dispatcher-cache-pattern.md` for: -- EventDispatcher lifecycle events dispatched by sqlor (C/U/D before/after) -- Module cache status: RBAC (partial + bug), Pricing (partial), Llmage (none) -- LRU cache implementation with async-safe locks and TTL -- Event handler registration pattern in module init.py -- Known bug: RBAC uses `this` instead of `self` in userperm.py -- Naming convention: `{dbname}:{tablename}:{c|u|d}:{before|after}` -- **CRITICAL: EventDispatcher is process-scoped — in SO_REUSEPORT multi-process deployment, events fired in one worker are invisible to all other workers. Migrate to Redis Pub/Sub for cross-process cache invalidation. See `references/multi-process-cache-invalidation.md`.** - -See `references/user-isolation-pattern.md` for: -- Context propagation chain (reasoning -> execute_tool -> tool wrapper) -- User-isolated directory structure (~/.hermes/users/{user_id}/) -- `_get_user_dir()` helper pattern for tool wrappers -- Per-user WebSocket callbacks (ws_push_callbacks dict) -- Table name fix: `hermes_skills` (NOT `harnessed_skills`) - -See `references/reasoning-visualization.md` for: -- Complete event flow for reasoning visualization (context/plan/safety/execution) -- Per-user WebSocket callback registration pattern in .wss endpoint -- File-based skill search across user and shared directories - -**Symptom: "等待连接" / WebSocket never connects** -If the reasoning console shows "等待连接..." and never transitions to "已连接": -1. Check HTML widget JavaScript for hardcoded `user_id: 'current_user'` — must be `{{ get_user() }}` -2. Verify RBAC permission exists for the `.wss` path (without `/wss/` prefix) -3. Check Redis is running (Sage sessions depend on `redis://127.0.0.1:6379`) -4. Verify `.wss` file exists at `wwwroot/endpoint.wss` and defines `async def myfunc(request, **kwargs)` - ## Testing: Use Browser Tools, NOT curl -**CRITICAL: When testing Sage web features, ALWAYS use browser tools (`browser_navigate`, `browser_click`, `browser_type`, etc.) — NEVER curl.** +**CRITICAL: When testing Sage web features, ALWAYS use browser tools (`browser_navigate`, `browser_click`, `browser_type`, etc.) — NEVER curl.** curl cannot carry the browser's session cookies through the RBAC authentication flow → false negatives. The browser handles cookie sessions (AIOHTTP_SESSION), JS widget rendering (bricks), and form submissions. -curl cannot carry the browser's session cookies through the RBAC authentication flow, so curl tests produce false negatives. The browser automatically handles: -- Cookie-based session management (AIOHTTP_SESSION) -- JavaScript widget rendering (bricks framework) -- Form submissions via the bricks UI layer - -**Correct testing pattern:** ```python -# Navigate to the page browser_navigate(url='http://localhost:9180/module/page.ui') -browser_snapshot() # Check what's rendered - -# For login-required pages, interact with the login form: -browser_navigate(url='http://localhost:9180/index.ui') -browser_type(ref='@e38', text='superuser') # username field -browser_type(ref='@e39', text='Kyy@123456') # password field -browser_click(ref='@e26') # submit button -sleep(2) - -# Then navigate to the target page -browser_navigate(url='http://localhost:9180/harnessed_reasoning/reasoning_console.ui') browser_snapshot() -``` - -**Wrong: Using curl for authentication testing:** -```bash -# WRONG - curl cannot maintain session state through RBAC login -curl -b /tmp/cookies.txt https://sage.example.com/module/page.ui +# Login-required pages: navigate to /index.ui, type username/password (superuser / Kyy@123456), submit, then navigate to target +browser_navigate(url='http://localhost:9180/harnessed_reasoning/reasoning_console.ui') ``` ## Common Pitfalls Checklist -- [ ] **Stay focused on the current module** — when the user says "你跑飞啦，停，聚焦在XX模块", immediately stop all cross-module investigation and return to the current module's issues. Do not jump to other modules, dashboards, or unrelated features until the current module's problems are resolved. The user will explicitly redirect you when ready. -- [ ] **Understand the requirement before coding** — if the user says "完全错误" (completely wrong), you have fundamentally misunderstood the request. Stop immediately, re-read the user's message, and ask clarifying questions rather than continuing down the wrong path -- [ ] **Be proactive with configuration** — when you have the necessary credentials or paths (API keys, model URLs, database configs), configure them immediately without waiting for user confirmation. The user expects you to "该配什么配什么" (configure what needs configuring) rather than asking permission for each step. -- [ ] **Use existing RBAC modules, don't recreate** — never write custom login logic in `sage/wwwroot/` when RBAC provides `/rbac/user/login.ui`. Never duplicate files that already exist in module directories. -- [ ] **Follow established development standards** — the user has zero tolerance for errors at critical moments. Always load relevant skill docs (`bricks-framework`, `crud-definition-spec`, `module-development-spec`) before making changes. Systematically verify all affected files, not just the ones you touched. +- [ ] **Stay focused on the current module** — on "你跑飞啦，停，聚焦在XX模块", stop cross-module investigation immediately; user redirects when ready +- [ ] **Understand the requirement before coding** — "完全错误" = fundamentally misunderstood: stop, re-read, ask +- [ ] **Be proactive with configuration** — with credentials/paths in hand, configure immediately ("该配什么配什么"), don't ask permission per step +- [ ] **Use existing RBAC modules, don't recreate** — never custom login in `sage/wwwroot/` when `/rbac/user/login.ui` exists; don't duplicate existing module files +- [ ] **Follow established standards & verify all affected files** — load `bricks-framework`, `crud-definition-spec`, `module-development-spec` before changes; zero tolerance at critical moments - [ ] `entire_url` paths must NOT include `wwwroot` -- [ ] JSON CRUD files MUST have `editable` section with `new_data_url`, `update_data_url`, `delete_data_url` -- [ ] TabPanel uses `items` NOT `tabs`, widgettype is `TabPanel` NOT `Tab` -- [ ] TabPanel `content` directly embeds widget description (e.g., urlwidget) -- [ ] SQL parameters use `${param}$` format in `.dspy` files -- [ ] `sor.sqlExe(sql, ns)` ALWAYS requires the second `ns` argument — use `{}` when no parameters -- [ ] `.dspy` files return `json.dumps()` strings -- [ ] CRUD save endpoints return Message widget format -- [ ] `get_module_dbname('module_name')` to get database name -- [ ] `await get_user()` for current user ID -- [ ] JSON files in `json/` directory reference wwwroot files via `../` relative paths -- [ ] `.ui` files' `entire_url()` arguments MUST be quoted strings, NOT bare variables -- [ ] Same rule applies to JSON subtables `url` fields -- [ ] .dspy files are wrapped by framework in `async def myfunc()` — py_compile will show "await outside function" errors, which is expected and normal -- [ ] CRUD .dspy API files go in `wwwroot/api/` directory -- [ ] DELETE operations must include `AND user_id = ${user_id}$` for multi-user isolation -- [ ] UPDATE operations must include `AND user_id = ${user_id}$` for multi-user isolation -- [ ] **Python backend: NEVER use `sor.sqlExe()` with ORDER BY or LIMIT** — use `sor.R(table, ns_dict)` with Python slicing instead -- [ ] **Python backend: ALL queries must filter by user_id** — `sor.R('table', {'user_id': user_id, 'sort': 'field desc'})` -- [ ] **Python backend: `sor.R()` signature is `R(tablename, ns, filters=None)`** — `ns` (2nd arg) is a SINGLE dict containing BOTH filter conditions AND sort/page options. NO `ns=` keyword, NO 3rd-arg filters for normal queries -- [ ] **Python backend: NEVER create `DBPools()` in `__init__()`** — create it locally in each function that needs database access -- [ ] **Python backend: NEVER use `db.sqlorContext('default')`** — always pass the actual module name (e.g., `'harnessed_agent'`, `'customer_management'`) -- [ ] **Debugging: `rf.register('password', ...)` in `app/rf.py` must be uncommented** — login `.dspy` calls `await rfexe('password', params_kw)` to RC4-encrypt the password before DB lookup. If commented out, login silently fails with "user name or password error". After uncommenting, restart Sage -- [ ] **Debugging: `decode_password` in `app/rf.py` has typo `config.getConfig()`** — should be `config = getConfig()` -- [ ] **Debugging: always report the root cause when something fails** — user has zero tolerance for silent failures. When a test or fix doesn't work, immediately explain WHY (log output, curl verification, specific error message), not just "it didn't work". User will explicitly ask "出错了不报告出错原因吗" if you skip this -- [ ] **Debugging: test features yourself using available tools before asking the user** — use curl, server logs, or programmatic probes to verify fixes. User expects self-testing ("你自己操作浏览器测试"), not "please try it yourself". When browser_navigate or other UI tools are unavailable (Chrome zombie processes, etc.), use curl + log analysis as alternative verification -- [ ] **Debugging: when server logs show `WS Registered` but UI shows "等待连接", check RBAC permission path** — server logs reveal the exact path RBAC checks (e.g., `path='/wss/.../xxx.wss'`). If the permission is registered without `/wss/`, it will fail with `permission check failed` -- [ ] **Debugging: deployed code runs from site-packages, not repo** — use log line numbers to identify version -- [ ] **Debugging: config.website.ssl may be None even when hasattr returns True** — `hasattr(config.website, 'ssl')` returns True if the key exists in JSON, but the value may be `None`. Always check `if self.conf.website.ssl:` before accessing attributes -- [ ] **Debugging: use `info()` for debug output, not `debug()`** — debug may be filtered by log level -- [ ] **Tool wrappers: core tools must execute real operations, not return mock dicts** — `read_file`, `write_file`, `terminal`, `execute_code`, `memory`, `skill_manage`, `todo` must all do real work -- [ ] **Tool wrappers: accept `context` param for user isolation** — memory, skills, todo, execute_code wrappers must use `_get_user_dir()` -- [ ] **Widgettype casing matters**: use `Html` (mixed case), NOT `HTML` (all caps). `HTML` will fail with "widgetBuild(): HTML not registered". Also: `Scroll` widgettype does NOT exist — use `VBox` with `style: "overflow-y: auto;"` instead -- [ ] **Form internal buttons cannot have external binds**: Buttons inside `Form.options.buttons` are handled by Form's submit mechanism. If you need custom click handlers (not form submit), use standalone `Button` widgets OUTSIDE the Form. Otherwise you get "desc wid not find" errors -- [ ] **Frontend JS in HTML widgets: use `{{ get_user() }}` for user_id, NOT hardcoded `'current_user'` string** — JavaScript inside `.ui` HTML widgets runs client-side and has no server session; user_id must be injected via Jinja2 template rendering -- [ ] **Raw JS WebSocket is more reliable than `bricks.WebSocket` widget**: The widget's event binds require handler functions to exist before widget initialization. If `Html` widget renders after `WebSocket` widget, you get `ReferenceError`. Use a single `Html` widget with raw JS `new WebSocket(url, session)` instead +- [ ] JSON CRUD files MUST have `editable` with `new_data_url`/`update_data_url`/`delete_data_url` +- [ ] TabPanel: widgettype `TabPanel` NOT `Tab`; `items` NOT `tabs`; `content` directly embeds widget description +- [ ] SQL params use `${param}$` in `.dspy`; `sor.sqlExe(sql, ns)` ALWAYS needs the 2nd `ns` arg (`{}` if none) +- [ ] `.dspy` returns `json.dumps()` strings; CRUD save endpoints return Message widget format +- [ ] `get_module_dbname('module_name')` for db name; `await get_user()` for user ID +- [ ] `json/` files reference wwwroot files via `../` relative paths; `entire_url()` args MUST be quoted strings (same for subtable `url`) +- [ ] .dspy wrapped by framework in `async def myfunc()` — py_compile "await outside function" errors are expected/normal +- [ ] CRUD .dspy API files go in `wwwroot/api/` +- [ ] DELETE/UPDATE must include `AND user_id = ${user_id}$` for multi-user isolation +- [ ] **Python backend: NEVER `sor.sqlExe()` with ORDER BY or LIMIT** — use `sor.R(table, ns_dict)` + Python slicing; ALL queries filter by user_id +- [ ] **`sor.R(tablename, ns, filters=None)`** — 2nd arg `ns` is a SINGLE dict with BOTH filters AND sort/page options; no `ns=` keyword, no 3rd-arg filters for normal queries +- [ ] **NEVER create `DBPools()` in `__init__()`** — create locally in each function needing DB access +- [ ] **NEVER `db.sqlorContext('default')`** — always the actual module dbname (e.g. `'harnessed_agent'`, `'customer_management'`) +- [ ] Debugging: `rf.register('password', ...)` in `app/rf.py` must be uncommented (else login silently fails "user name or password error"); `decode_password` typo `config.getConfig()` → `getConfig()`; restart Sage after +- [ ] **Always report the root cause** — zero tolerance for silent failures ("出错了不报告出错原因吗"); explain WHY with logs/curl/error text +- [ ] **Self-test with available tools** before asking the user ("你自己操作浏览器测试"); curl + log analysis as fallback when browser unavailable +- [ ] `WS Registered` in logs but UI "等待连接" → RBAC path missing or without `/wss/` (server log shows exact path checked) +- [ ] Deployed code runs from site-packages, not repo — use log line numbers to identify version +- [ ] `hasattr(config.website, 'ssl')` is True even when value is None — check `if self.conf.website.ssl:` before accessing attributes +- [ ] Use `info()` for debug output, not `debug()` (may be filtered by log level) +- [ ] Core tool wrappers must execute REAL operations, not return mock dicts (read_file, write_file, terminal, execute_code, memory, skill_manage, todo) +- [ ] Tool wrappers accept `context` param and use `_get_user_dir()` for user isolation +- [ ] Widgettype casing: `Html` NOT `HTML` (else "widgetBuild(): HTML not registered"); `Scroll` doesn't exist — use VBox + `style: "overflow-y: auto;"` +- [ ] Form internal buttons can't have external binds — standalone Button widgets OUTSIDE the Form for custom handlers (else "desc wid not find") +- [ ] Frontend JS in HTML widgets: use `{{ get_user() }}` for user_id, NOT hardcoded `'current_user'` (JS runs client-side, no server session) +- [ ] Raw JS WebSocket more reliable than `bricks.WebSocket` widget (avoids ReferenceError timing — handlers must exist before widget init) -## Python Backend Code: Database Query Patterns +## Python Backend: Database Query Patterns -**CRITICAL: `sor.R()` signature is `R(tablename, ns, filters=None)`**. The 2nd arg `ns` is a SINGLE dict containing BOTH filter conditions AND sort/page options. No `ns=` keyword argument needed. +**CRITICAL: `sor.R()` signature is `R(tablename, ns, filters=None)`** — the 2nd arg `ns` is a SINGLE dict containing BOTH filter conditions AND sort/page options. No `ns=` keyword needed. -### Sorting: Put `sort` in the same dict as filters - -```python -# WRONG -- ns= keyword argument: -rows = await sor.R('users', {'status': 'active'}, ns={'sort': 'created_at desc'}) - -# WRONG -- sqlExe with ORDER BY: -sql = "SELECT * FROM users WHERE status = :status ORDER BY created_at DESC" -rows = await sor.sqlExe(sql, {'status': 'active'}) - -# CORRECT -- filter conditions + sort in ONE dict (2nd arg): -rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'}) -``` - -### Limiting: Use Python Slicing NOT `sqlExe("LIMIT n")` - -```python -# WRONG -- sqlExe with LIMIT: -sql = "SELECT * FROM users WHERE status = :status LIMIT 10" -rows = await sor.sqlExe(sql, {'status': 'active'}) - -# CORRECT -- sor.R with sort, then Python slicing: -rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'}) -rows = (rows or [])[:10] -``` - -### OFFSET: Use Python Slicing - -```python -# CORRECT: -rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'}) -rows = (rows or [])[20:20 + 10] # [offset:offset+limit] -``` - -### Multi-user Isolation: Every Query Needs user_id - -```python -# CORRECT -- user isolation + sort in one dict: -rows = await sor.R('hermes_sessions', {'user_id': user_id, 'sort': 'started_at desc'}) -rows = (rows or [])[:50] -``` - -### $or Conditions: Put everything in the ns dict - -```python -rows = await sor.R('hermes_skills', { - 'user_id': user_id, - '$or': [ - {'name': {'$like': '%keyword%'}}, - {'description': {'$like': '%keyword%'}} - ] -}) -rows = (rows or [])[:2] -``` - -### sor.R vs sor.sqlExe Decision +- **Sorting** — put `'sort'` in the same dict: `rows = await sor.R('users', {'status': 'active', 'sort': 'created_at desc'})`. WRONG: `sor.R('users', {'status': 'active'}, ns={'sort': ...})`; WRONG: `sqlExe` with ORDER BY +- **Limit/OFFSET** — Python slicing, NOT SQL LIMIT: `rows = (rows or [])[:10]`; offset: `(rows or [])[20:20+10]` +- **Multi-user isolation** — every query filters user_id: `await sor.R('hermes_sessions', {'user_id': user_id, 'sort': 'started_at desc'})` +- **$or conditions** go in the ns dict: `await sor.R('hermes_skills', {'user_id': user_id, '$or': [{'name': {'$like': '%keyword%'}}, {'description': {'$like': '%keyword%'}}]})` | Use `sor.R(table, ns_dict)` | Use `sor.sqlExe(sql, params)` | |---|---| | Simple CRUD reads with filtering | INSERT/UPDATE/DELETE operations | -| Need sorting via `'sort': '...'` in ns dict | Complex joins or subqueries | +| Sorting via `'sort'` in ns dict | Complex joins or subqueries | | Multi-user isolation with user_id filter | When sor.R can't express the query | ## LLM Client Pattern (harnessed_agent) -**CRITICAL: harnessed_agent is an LLM CLIENT, not a server.** It calls external LLM provider APIs — it does NOT serve `/v1/chat/completions` endpoints to others. +**CRITICAL: harnessed_agent is an LLM CLIENT, not a server** — it calls external LLM provider APIs (`aiohttp POST /v1/chat/completions` to OpenAI/DashScope/DeepSeek/SiliconFlow); it does NOT serve chat endpoints to others. -### Architecture - -``` -harnessed_agent (client) --aiohttp POST--> LLM Provider (OpenAI/DashScope/DeepSeek/SiliconFlow) - /v1/chat/completions -``` - -The `llm_client.py` module provides 5 functions registered to ServerEnv: -- `llm_chat(messages, model, temperature, ...)` -> OpenAI response dict -- `llm_chat_stream(messages, ...)` -> async generator yielding {delta, finish_reason, raw} -- `llm_simple(prompt, system)` -> plain text string -- `llm_list_models()` -> provider model list -- `llm_get_config()` -> current config (key masked) - -### Usage in .dspy files +`llm_client.py` registers 5 functions to ServerEnv: `llm_chat(messages, model, temperature, ...)` → OpenAI response dict; `llm_chat_stream(messages, ...)` → async generator yielding `{delta, finish_reason, raw}`; `llm_simple(prompt, system)` → plain text; `llm_list_models()` → provider model list; `llm_get_config()` → current config (key masked). ```python -# Standard chat call -result = await llm_chat( - messages=[ - {"role": "system", "content": "You are helpful"}, - {"role": "user", "content": "Hello"} - ], - model="qwen3-max", - temperature=0.7 -) -# result matches OpenAI format: {"choices": [{"message": {"content": "..."}}], "usage": {...}} - -# Stream mode -async for chunk in llm_chat_stream(messages=[...]): - text = chunk['delta'] # accumulated text - -# Simple text-only call +result = await llm_chat(messages=[{"role": "system", "content": "You are helpful"}, {"role": "user", "content": "Hello"}], model="qwen3-max", temperature=0.7) +# result: {"choices": [{"message": {"content": "..."}}], "usage": {...}} +async for chunk in llm_chat_stream(messages=[...]): text = chunk['delta'] answer = await llm_simple("What is 2+2?", system="Answer briefly") ``` -### Provider Presets +Provider presets (`harnessed_agent_config` table): `llm_provider` (preset name: `dashscope` default / `openai` / `deepseek` / `siliconflow` / empty=custom); `llm_service_url` (base URL, auto-filled from preset or custom); `llm_api_key` (Bearer token); `default_model` (e.g. `qwen-plus`); `default_temperature` / `top_p` (floats: length=5, dec=2). +Preset URLs: dashscope `https://dashscope.aliyuncs.com/compatible-mode/v1`; openai `https://api.openai.com/v1`; deepseek `https://api.deepseek.com/v1`; siliconflow `https://api.siliconflow.cn/v1` -Configured in `harnessed_agent_config` table: - -| Field | Description | -|---|---| -| `llm_provider` | Preset name: `dashscope` (default), `openai`, `deepseek`, `siliconflow`, or empty for custom | -| `llm_service_url` | Base URL (auto-filled from preset, or custom URL) | -| `llm_api_key` | Bearer token for authentication | -| `default_model` | Default model name (e.g. `qwen-plus`) | -| `default_temperature` | Default temperature (float, length=5, dec=2) | -| `top_p` | Default top_p (float, length=5, dec=2) | - -Preset URLs: -- dashscope: `https://dashscope.aliyuncs.com/compatible-mode/v1` -- openai: `https://api.openai.com/v1` -- deepseek: `https://api.deepseek.com/v1` -- siliconflow: `https://api.siliconflow.cn/v1` - -### Resilience Requirements - -LLM client MUST implement: -- **Retry with exponential backoff**: 3 attempts for transient errors (timeout, 500, connection failure) -- **429 rate limit handling**: Read `Retry-After` header, wait and retry -- **Structured logging**: Use `appPublic.log` (info/warning/error) for request params, response timing, token counts -- **Error propagation**: Return OpenAI-compatible error dict `{"error": {"message": "...", "type": "...", "code": N}}` +Resilience REQUIRED: retry with exponential backoff (3 attempts: timeout/500/connection failure); 429 rate limit → read `Retry-After`, wait, retry; structured logging via `appPublic.log` (info/warning/error: request params, response timing, token counts); error propagation as OpenAI-compatible `{"error": {"message": ..., "type": ..., "code": N}}`. ## harnessed_reasoning Pattern: LLM-Based Reasoning Engine -**harnessed_reasoning** is a REAL reasoning engine, not a mock. It uses harnessed_agent's LLM client and tool execution system to perform actual AI reasoning and task execution. +**harnessed_reasoning is a REAL reasoning engine, not a mock** — uses harnessed_agent's LLM client + tool execution. Flow: `reasoning_console.ui` (Form) → `reasoning_submit.dspy` → `hermes_reason_and_execute()` → LLM planning (`llm_chat`) + tool execution (`harnessed_execute_tool`) → results stored in DB. -### Architecture +1. **Context gathering**: `harnessed_get_intelligent_memory_context` + session search + skill search +2. **LLM planning**: `llm_chat()` with system prompt incl. tool descriptions → JSON execution plan +3. **Safety check**: validates plan against configurable rules (strict/moderate/lenient) — blocks dangerous commands like `rm -rf /` +4. **Tool execution**: if safe & `execute_immediately=True`, `harnessed_execute_tool()` per action +5. **Error recovery**: auto-recovers (read_file not found → search_files; permission denied → strip sudo prefix) +6. **Session storage**: `harnessed_reasoning_sessions` table -``` -User Input -> reasoning_console.ui (Form) - | - v - reasoning_submit.dspy - | - v - hermes_reason_and_execute() - / \ - / \ - LLM Planning Tool Execution - (llm_chat) (harnessed_execute_tool) - \ / - \ / - v v - Execution Plan + Results -> stored in DB -``` +17 tools: read_file, write_file, search_files, patch, terminal, process, execute_code, memory, skill_manage, skill_view, todo, session_search, cronjob, clarify, delegate_task, text_to_speech, vision_analyze -### How It Works - -1. **Context Gathering**: Calls `harnessed_get_intelligent_memory_context` + session search + skill search to build real context -2. **LLM Planning**: Calls `llm_chat()` with a reasoning system prompt that includes available tool descriptions, asking LLM to return a JSON execution plan -3. **Safety Check**: Validates the plan against configurable safety rules (strict/moderate/lenient) — blocks dangerous commands like `rm -rf /` -4. **Tool Execution**: If safe and `execute_immediately=True`, calls `harnessed_execute_tool()` for each action in the plan -5. **Error Recovery**: Auto-recovers from common failures (e.g., read_file not found -> search_files; permission denied -> strip sudo prefix) -6. **Session Storage**: Stores all reasoning sessions in `harnessed_reasoning_sessions` table - -### Available Tools (17) - -read_file, write_file, search_files, patch, terminal, process, execute_code, memory, skill_manage, skill_view, todo, session_search, cronjob, clarify, delegate_task, text_to_speech, vision_analyze - -### Reasoning Config - -Configured in `harnessed_reasoning_config` table: -- `model_name`: LLM model for planning (default: `qwen3-max`) -- `temperature` / `top_p`: LLM parameters -- `system_prompt`: Custom reasoning system prompt (overrides default) -- `safety_mode`: `strict` / `moderate` / `lenient` -- `max_reasoning_steps`, `max_tool_calls_per_step`: Execution limits -- `enable_error_recovery`: Auto-recovery on tool failures +Config (`harnessed_reasoning_config`): `model_name` (default `qwen3-max`), `temperature`/`top_p`, `system_prompt` (overrides default), `safety_mode` (strict/moderate/lenient), `max_reasoning_steps`, `max_tool_calls_per_step`, `enable_error_recovery`. ## Sage Multi-Process Deployment Architecture ### SO_REUSEPORT: Multiple Workers Share One Port - -ahserver's `ConfiguredServer.run()` sets `reuse_port=True` on Linux, allowing multiple `sage.py` processes to bind to the same port. The kernel distributes incoming connections across workers (similar to nginx/gunicorn worker model). - -**Pattern in `start.sh`:** +`ConfiguredServer.run()` sets `reuse_port=True` on Linux — multiple `sage.py` processes bind the same port; kernel distributes connections (nginx/gunicorn-style). `start.sh`: ```bash WORKERS=$(nproc) # auto-detect CPU cores -for (( i=0; i "logs/sage_worker_${i}.log" 2>&1 & done ``` - -All workers listen on the same `$PORT`. No load balancer or port range needed. +No load balancer or port range needed. ### Background Coroutines Must NOT Run in Every Worker +`add_cleanupctx(coro)` (async ctx manager: startup+cleanup) / `add_startup(coro)` (startup only) / `asyncio.create_task(...)` in module `init.py` attach to aiohttp hooks — in multi-process mode EVERY worker starts its own copy → duplicate work, double-charging, DB race conditions. Known: `llmage` `add_cleanupctx(start_backend)` → `backend_accounting()` (10s billing loop); `unipay` `add_startup(setup_callback_path)` (route registration — safe to duplicate, aiohttp registration is idempotent per process; background *tasks* MUST be extracted). -**The Problem**: Sage modules register background coroutines via `add_cleanupctx()` and `add_startup()`. These are attached to aiohttp's `app.cleanup_ctx` and `app.on_startup` hooks. In multi-process mode, EVERY worker would start its own copy of the background task — causing duplicate work, double-charging, race conditions on DB records. +**Fix**: extract background coroutines into standalone programs, start them ONCE in `start.sh` BEFORE sage.py workers, remove `add_cleanupctx`/`add_startup` from `init.py`. -**Identify background coroutines**: Search module `init.py` files for: -- `add_cleanupctx(coro)` — runs at server startup, cleaned up on shutdown -- `add_startup(coro)` — runs on aiohttp app startup -- `asyncio.create_task(...)` inside init or startup hooks - -**Known modules using these patterns:** -| Module | Hook | Background Task | What It Does | -|--------|------|-----------------|--------------| -| `llmage` | `add_cleanupctx(start_backend)` | `backend_accounting()` | Periodic LLM usage billing loop (every 10s) | -| `unipay` | `add_startup(setup_callback_path)` | Registers payment callback routes | (route registration, safe to duplicate) | - -**The Fix**: Extract background coroutines into standalone programs, start them once in `start.sh` BEFORE the sage.py workers, and remove the `add_cleanupctx`/`add_startup` calls from module `init.py`. - -**Example: Extract `backend_accounting` from llmage:** - -1. Create standalone program `bin/backend_accounting.py`: +Standalone program template (from llmage `backend_accounting`): ```python #!/usr/bin/env python import os, sys, asyncio, signal os.chdir(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -sys.path.insert(0, 'py3/lib/python3.10/site-packages') -sys.path.insert(0, 'pkgs') - +sys.path.insert(0, 'py3/lib/python3.10/site-packages'); sys.path.insert(0, 'pkgs') from appPublic.folderUtils import ProgramPath from appPublic.jsonConfig import getConfig from sqlor.dbpools import DBPools from appPublic.log import MyLogger, info, exception from llmage.accounting import get_accounting_llmusages, llm_accounting, llm_accoung_failed -# Init config + DB p = ProgramPath() config = getConfig(NS={'workdir': os.getcwd(), 'ProgramPath': p}) -DBPools(config.databases) +DBPools(config.databases) # standalone must init manually (sage.py workers get it from webapp()) async def backend_accounting(): info('backend accounting started ...') @@ -1044,19 +531,16 @@ async def backend_accounting(): try: lus = await get_accounting_llmusages() except Exception as e: - exception(f'{e}') - lus = [] + exception(f'{e}'); lus = [] for lu in lus: try: await llm_accounting(lu) except Exception as e: - exception(f'{e}, {lu.id=}') - await llm_accoung_failed(lu.id) + exception(f'{e}, {lu.id=}'); await llm_accoung_failed(lu.id) await asyncio.sleep(10) def main(): - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) + loop = asyncio.new_event_loop(); asyncio.set_event_loop(loop) signal.signal(signal.SIGTERM, lambda s, f: [t.cancel() for t in asyncio.all_tasks(loop)] or loop.stop()) try: loop.run_until_complete(backend_accounting()) @@ -1069,212 +553,100 @@ if __name__ == '__main__': main() ``` -2. Remove from `llmage/init.py`: -```python -# REMOVE these lines: -# from ahserver.configuredServer import add_cleanupctx -# async def start_backend(app): ... -# add_cleanupctx(start_backend) -``` +start.sh order: 1) background programs first: `nohup $PYTHON bin/backend_accounting.py > logs/backend_accounting.log 2>&1 &`, record `name:pid` in `sage_backend.pid`; 2) sage workers (plain pid per line in `sage.pid`). stop.sh: kill workers from sage.pid, then background programs via `while IFS=: read name pid; do kill $pid 2>/dev/null; done < sage_backend.pid`. -3. Update `start.sh` to start it first: -```bash -# Step 1: Start independent background programs -nohup $PYTHON bin/backend_accounting.py > logs/backend_accounting.log 2>&1 & -echo "backend_accounting:$!" >> sage_backend.pid - -# Step 2: Start Sage web workers (SO_REUSEPORT) -for (( i=0; i "logs/sage_worker_${i}.log" 2>&1 & - echo "$!" >> sage.pid -done -``` - -4. Update `stop.sh` to stop both: -```bash -# Stop workers from sage.pid -while read pid; do kill $pid 2>/dev/null; done < sage.pid - -# Stop background programs from sage_backend.pid (format: name:pid) -while IFS=: read name pid; do kill $pid 2>/dev/null; done < sage_backend.pid -``` +Pitfalls: +- `add_cleanupctx` = async context manager (yield; startup + cleanup); `add_startup` = startup only. Both per-process hooks +- PID file: `name:pid` format for background programs (named stop), plain `pid` for workers +- Background programs must handle SIGTERM (cancel asyncio tasks) or `kill` leaves DB connections inconsistent +- Standalone programs must init `DBPools(config.databases)` manually; use `MyLogger` with a dedicated log file (they don't inherit Sage's log config) ### Sage standalone scripts — DBPools pattern - -When writing standalone scripts (data migration, seeding, export) that need database access outside the Sage server context: - ```python from appPublic.jsonConfig import getConfig # NOT appPublic.getConfig from sqlor.dbpools import DBPools import asyncio - config = getConfig('.') db = DBPools(config.databases) dbname = list(config.databases.keys())[0] - async def run(): - async with db.sqlorContext(dbname) as sor: - # use sor + async with db.sqlorContext(dbname) as sor: ... asyncio.run(run()) ``` +Inside Sage server context (`.dspy`, module code): use `get_sor_context(env, 'modulename')` instead. -**Inside Sage server context** (`.dspy`, module code): use `get_sor_context(env, 'modulename')` instead. +### EventDispatcher Cache Invalidation is Process-Scoped — Use Redis Pub/Sub +sqlor C/U/D lifecycle events only fire within the current process. In SO_REUSEPORT deployment, worker A's invalidation never reaches workers B–N → stale cache. **Fix**: Redis Pub/Sub channel `sage:cache:invalidate` — each worker keeps a local cache (fast reads) + subscribes; publish on invalidation, all workers evict; add TTL fallback for missed messages. See `references/multi-process-cache-invalidation.md`. Naming: `{dbname}:{tablename}:{c|u|d}:{before|after}`. Known bug: RBAC uses `this` instead of `self` in userperm.py. ## CRITICAL: uapi.headers Template Rendering — No Helper Functions Needed -The `uapi.headers` field is a **JSON template string**. Template variables are rendered at runtime before each HTTP request: +`uapi.headers` is a JSON **template string**; variables render at runtime before each HTTP request: `{{apikey}}` → decrypted API key from `upappkey` table; `{{jsondata}}` → JSON request body; `{{response}}` → response transformation template. -| Template variable | Renders to | -|---|---| -| `{{apikey}}` | The decrypted API key from `upappkey` table | -| `{{jsondata}}` | The JSON request body | -| `{{response}}` | Response transformation template | +- Vidu-style Token auth: `{"Content-Type": "application/json", "Authorization": "Token {{apikey}}"}` — NO helper function; do NOT add `token()` to `uapi/appapi.py`, the template engine handles `{{apikey}}` natively +- OpenAI-compatible: `Bearer {{apikey}}`; complex signing: `{{bearer(apikey)}}` helper -**For Vidu-style Token auth, use `{{apikey}}` directly in the headers JSON — NO helper function needed:** +## External API Design Patterns (uapi detail) -```json -{"Content-Type": "application/json", "Authorization": "Token {{apikey}}"} -``` - -Do NOT add a `token()` or similar helper function to `uapi/appapi.py` — the template engine handles `{{apikey}}` substitution natively. - -Other auth patterns: `Bearer {{apikey}}` (OpenAI-compatible), `{{bearer(apikey)}}` (with bearer() helper if needed for complex signing). - -## Pitfalls - -- **`add_cleanupctx` vs `add_startup`**: `add_cleanupctx` is an async context manager (`yield` pattern) that runs on startup and cleanup. `add_startup` runs on app startup only. Both are per-process hooks. -- **Route registration via `add_startup`** (like unipay's `setup_callback_path`) is technically safe to duplicate across workers since aiohttp route registration is idempotent within each process. But background *tasks* (loops, periodic jobs, consumers) MUST be extracted. -- **PID file format**: Use `name:pid` format for background program PIDs to enable named stop commands. Use plain `pid` format for worker PIDs (one per line). -- **Signal handling**: Background programs should handle `SIGTERM` to gracefully cancel asyncio tasks. Without this, `kill` will leave DB connections in inconsistent state. -- **DBPools in standalone programs**: Must initialize `DBPools(config.databases)` before importing module functions that use it. Unlike sage.py workers (where `webapp()` does this), standalone programs must do it manually. -- **Logging**: Use `MyLogger` with a dedicated log file path. Standalone programs don't inherit Sage's log config. - -### EventDispatcher Cache Invalidation is Process-Scoped — Use Redis Pub/Sub for Multi-Process - -**EventDispatcher events from sqlor C/U/D lifecycle hooks only fire within the current process.** In Sage's SO_REUSEPORT multi-process deployment, when worker A modifies data and fires a cache invalidation event, workers B through N never receive it — they continue serving stale cached data. - -**Fix:** Use Redis Pub/Sub as the cross-process invalidation channel. Each worker maintains a local cache (fast reads) and subscribes to a shared `sage:cache:invalidate` channel. When any worker publishes an invalidation message, all workers evict the corresponding local cache entry. Add a TTL fallback to handle missed messages. - -See `references/multi-process-cache-invalidation.md` for the complete pattern and reference implementation. - -### External API Design Patterns - -- **DO NOT write direct HTTP clients for vendor APIs — use Sage's uapi gateway instead.** Sage provides a complete external API routing system via the `uapi` module (`pkgs/uapi/`). Write a direct HTTP client (like `volcengine_client.py`) only when the uapi gateway cannot express the protocol. For standard REST/JSON APIs, use uapi. - -#### uapi Gateway Pattern - -**Tables**: `upapp` (vendor definition), `uapi` (API endpoints per vendor), `upappkey` (API credentials), `uapiio` (I/O schemas). +**Tables**: `upapp` (vendor definition), `uapi` (API endpoints per vendor; `upappid + name` combo unique — contains `path`, `httpmethod`, `headers` JSON template w/ `{{apikey}}`, `data` JSON template w/ params, `response` output transformation), `upappkey` (API credentials `apikey`/`secretkey`, encrypted), `uapiio` (I/O schemas). ``` Client → dapi(Bearer) → your_module/api/xxx.dspy - ↓ - vendor_id → lookup upappid + apiname in your module's config table - ↓ - from uapi.uapi import UpAppApi - ua = UpAppApi(request) - resp = await ua.call(upappid, apiname, callerid, params) - ↓ - uapi gateway → renders headers/data/response templates → HTTP → vendor + → vendor_id → lookup upappid+apiname in module config table + → from uapi.uapi import UpAppApi; resp = await UpAppApi(request).call(upappid, apiname, callerid, params) + → uapi gateway renders headers/data/response templates → HTTP → vendor ``` -**Module's config table should register per-vendor API mappings:** -```python -# In your module's vendor_config table: -# vendor: "volcengine" -# upappid: "upapp-volcengine-01" # references upapp.id -# apiname_create_session: "CreateVisualValidateSession" -# apiname_get_result: "GetVisualValidateResult" -# apiname_create_asset: "CreateAsset" -# ... one apiname per vendor operation -``` +Module config table registers per-vendor mappings: `vendor: "volcengine"`, `upappid: "upapp-volcengine-01"` (refs upapp.id), `apiname_create_session: "CreateVisualValidateSession"`, `apiname_get_result: "GetVisualValidateResult"`, `apiname_create_asset: "CreateAsset"` — one apiname per vendor operation. -**Calling a vendor API from your module's .dspy or init.py:** ```python from uapi.uapi import UpAppApi - async def call_vendor_api(vendor_id, action, params): - # 1. Look up the vendor config to get upappid + apiname config = await get_vendor_config(vendor_id) upappid = config.upappid apiname = getattr(config, f'apiname_{action}') - callerid = await get_user() - - # 2. Call through uapi gateway ua = UpAppApi(request) - resp = await ua.call(upappid, apiname, callerid, params) - - # 3. Parse response (uapi.response template may have transformed it) - return json.loads(resp.decode('utf-8')) + resp = await ua.call(upappid, apiname, await get_user(), params) + return json.loads(resp.decode('utf-8')) # response template may have transformed it ``` -**uapi configuration (set up once per vendor via Sage admin):** -- `upapp`: Define the vendor (name, base URL, auth type) -- `uapi`: Register each API endpoint. The `upappid + name` combo is unique. Contains `path`, `httpmethod`, `headers` (JSON template with `{{apikey}}`), `data` (JSON template with params), `response` (output transformation template) -- `upappkey`: Store `apikey`/`secretkey` per upapp (encrypted) - -**When to use direct HTTP vs uapi:** | Use uapi gateway | Use direct HTTP client | |---|---| | Standard REST/JSON APIs | Non-HTTP protocols (gRPC, WebSocket to vendor) | -| Vendor has OpenAPI/REST interface | Vendor requires complex HMAC signing not expressible in templates | -| Multiple vendors for same operation (easy to add via config) | Vendor API is a one-off with no reuse pattern | -| Need response templating/transformation | Need streaming/chunked responses with custom parsing | +| Vendor has OpenAPI/REST interface | Complex HMAC signing not expressible in templates | +| Multiple vendors for same op (config-driven, zero code) | One-off API with no reuse pattern | +| Need response templating/transformation | Streaming/chunked responses with custom parsing | -- **Bearer token auth: dapi module auto-resolves identity** — all client-facing `.dspy` APIs get `user_id` via `await get_user()` and `org_id` via `await get_userorgid()`. Never add `downapp_id`, `client_id`, or manual identification parameters to API endpoints. The Bearer token IS the identifier. -- **Never expose internal DB IDs to clients** — client API responses and parameters should only use vendor-side identifiers (e.g., `vendor_group_id` not `local_group_id`). Internal IDs are meaningless to downstream systems and create unnecessary coupling. -- **Client-facing upload endpoints accept vendor-side IDs** — validate ownership by looking up `rl_org_group(org_id, vendor_group_id)`, then use the internal `local_group_id` only for FK relationships in local tables. -- **Vendor callbacks are `paths_any`** — vendor POSTs have no session, cannot authenticate. Register callback endpoints in `paths_any`, not `paths_logined`. -- **Callback idempotency is mandatory** — vendors may retry; check for existing mapping/status before inserting. +- **Bearer token auth: dapi auto-resolves identity** — client-facing `.dspy` APIs get `user_id` via `await get_user()` and `org_id` via `await get_userorgid()`. NEVER add `downapp_id`/`client_id`/manual identification params — the Bearer token IS the identifier +- **Never expose internal DB IDs to clients** — client API responses/params use only vendor-side identifiers (`vendor_group_id` not `local_group_id`) +- **Upload endpoints accept vendor-side IDs** — validate ownership via `rl_org_group(org_id, vendor_group_id)`, use internal `local_group_id` only for local FK relationships +- **Vendor callbacks are `paths_any`** (vendor POSTs have no session) — NOT paths_logined; **callback idempotency mandatory** (vendors retry — check existing mapping/status before insert) + +## Schema Migration Pattern (xlsx models) -### Schema Migration Pattern (xlsx models) When modifying model fields (adding/removing columns): -1. Update the `.xlsx` file in `models/` -2. Create a production migration script that: (a) checks if the old column exists, (b) creates new tables/indexes, (c) migrates data, (d) optionally drops the old column -3. Make migration idempotent — safe to run multiple times -4. Copy updated files to `pkgs/` directory and reinstall: `cd pkgs/module && pip install -e .` -5. Restart Sage after code changes +1. Update the `.xlsx` in `models/` +2. Migration script: (a) check old column exists, (b) create new tables/indexes, (c) migrate data, (d) optionally drop old column — make it idempotent (safe to run multiple times) +3. Copy updated files to `pkgs/` and reinstall: `cd pkgs/module && pip install -e .` +4. Restart Sage after code changes -### pyproject.toml Dependencies -ONLY declare `sqlor` and `bricks_for_python`. Do NOT declare `ahserver`, `apppublic`, `appbase`, `rbac` — these are installed by `build.sh`, not pip. +## pyproject.toml Dependencies +ONLY declare `sqlor` and `bricks_for_python`. Do NOT declare `ahserver`, `apppublic`, `appbase`, `rbac` — these are installed by `build.sh`, not pip: ```toml -dependencies = [ - "sqlor", - "bricks_for_python", -] +dependencies = ["sqlor", "bricks_for_python"] ``` -### Python Backend: DBPools() Lifecycle, Singleton Fork Safety, and sqlorContext() Module Name +## Python Backend: DBPools() Lifecycle, Singleton Fork Safety, sqlorContext() Module Name Three critical rules for all Python backend code (core.py, etc.): -**Rule 1: DBPools() must be created in function scope, NEVER in `__init__()`** -```python -# WRONG: -class MyClass: - def __init__(self): - self.db = DBPools() +**Rule 1: `DBPools()` must be created in function scope, NEVER in `__init__()`** — WRONG: `self.db = DBPools()` in `__init__`; CORRECT: `db = DBPools()` inside each async method that needs DB access. -# CORRECT: -class MyClass: - def __init__(self): - pass +**Rule 2: DBPools is a Singleton (`@SingletonDecorator`)** — in forked child processes (where `.dspy` files run) the inherited parent instance persists; `DBPools(config.databases)` silently discards new args and returns the old instance with empty/stale `databases`. - async def query(self): - db = DBPools() - async with db.sqlorContext(dbname) as sor: - ... -``` - -**Rule 2: DBPools is a Singleton — in forked child processes, must manually set `db.databases` AND use `env.get_module_dbname()`** - -`DBPools` is decorated with `@SingletonDecorator`. The decorator caches the first instance: `__call__` returns the cached instance if it exists, completely ignoring new arguments. In forked child processes (where `.dspy` files execute), the inherited parent instance persists, so `DBPools(config.databases)` returns the old instance with empty/stale `databases`. The passed `config.databases` argument is silently discarded. - -Additionally, `sqlorContext()` receives a database **key** (e.g., `'crm_db'`), NOT a module name. Hardcoding `'harnessed_agent'` as the key fails because that key doesn't exist in `config.databases`. The main app's `get_module_dbname()` resolves all modules to the actual database key. - -**Complete required pattern for ALL database access in Python backend code:** +**Rule 3: `sqlorContext()` takes a database KEY (e.g. `'crm_db'`), NOT a module name** — hardcoding `'harnessed_agent'` fails (key doesn't exist in `config.databases`); `env.get_module_dbname()` resolves modules to the actual key. +Complete required pattern for ALL database access: ```python from ahserver.serverenv import ServerEnv from appPublic.jsonConfig import getConfig @@ -1282,290 +654,324 @@ from sqlor.dbpools import DBPools async def my_query(): env = ServerEnv() - dbname = env.get_module_dbname('my_module') # resolves DB key dynamically (e.g., 'crm_db') + dbname = env.get_module_dbname('my_module') # resolves DB key dynamically (e.g. 'crm_db') config = getConfig() - db = DBPools() # Returns Singleton instance (fork-safe) - db.databases = config.databases # MUST force-set to override inherited empty dict + db = DBPools() # Singleton instance (fork-safe) + db.databases = config.databases # MUST force-set: overrides inherited empty dict async with db.sqlorContext(dbname) as sor: rows = await sor.R('table', {...}) ``` -This 5-line template handles: -1. Singleton fork safety (`db.databases = config.databases` overwrites inherited empty dict) -2. Dynamic database resolution (`env.get_module_dbname()` returns the actual DB key like `'crm_db'`) -3. Proper context management +**NEVER** hardcode a database name in `sqlorContext()`. **NEVER** pass `config.databases` as a constructor arg to `DBPools()` — silently ignored due to the Singleton. -**NEVER** hardcode a database name string in `sqlorContext()`. **NEVER** pass `config.databases` as a constructor argument to `DBPools()` — it will be silently ignored due to the Singleton. Always use the 5-line pattern above. - -### harnessed_agent Tool Permission: Internal Calls Must Not Be Blocked - -The `_get_user_permissions()` method in `harnessed_agent/core.py` must NOT restrict permissions for empty/missing context. Internal workflow calls (e.g., reasoning engine executing tools) often pass `context=None`, and if the method returns only read-only permissions for anonymous users, tools like `write_file`, `memory`, `clarify` will fail with `"Insufficient permissions to execute tool 'X'"`. - -**Fix**: Grant full permissions unconditionally regardless of whether `context` is present: +## harnessed_agent: Tool Permissions, No Mocks, Context Propagation +### Internal Calls Must Not Be Blocked +`_get_user_permissions()` in `harnessed_agent/core.py` must NOT restrict permissions for empty/missing context — internal workflow calls pass `context=None`; restricting makes `write_file`, `memory`, `clarify` fail with "Insufficient permissions to execute tool 'X'". Grant full permissions unconditionally: ```python -def _get_user_permissions(self, context: Dict[str, Any]) -> List[str]: +def _get_user_permissions(self, context): # Internal system calls should not be blocked by permission checks - return [ - 'file_read', 'file_write', - 'system_execute', 'system_manage', - 'browser_access', - 'ai_vision', 'ai_tts', - 'memory_manage', 'memory_read', - 'skill_read', 'skill_manage', - 'task_manage', 'task_delegate', - 'user_interact', 'schedule_manage', - 'config_read' - ] + return ['file_read', 'file_write', 'system_execute', 'system_manage', 'browser_access', + 'ai_vision', 'ai_tts', 'memory_manage', 'memory_read', 'skill_read', 'skill_manage', + 'task_manage', 'task_delegate', 'user_interact', 'schedule_manage', 'config_read'] ``` -### CRUD JSON Strict Validation Checklist +### Tool Implementation: No Mocks +`harnessed_agent/tools/base_tools.py` wrappers MUST execute real operations, not return `status: "mock_implementation"` dicts (reasoning engine reports fake success, LLM hallucinates file locations). Real implementations: read_file/write_file/search_files/patch = actual file I/O; terminal = `asyncio.create_subprocess_shell`; execute_code = temp `.py` + `python3`; memory = `~/.hermes/memory.json`; skill_view/skills_list = `~/.hermes/skills/`; todo = `~/.hermes/todo.json`. Browser (`browser_*`), vision (`vision_analyze`), TTS (`text_to_speech`) need external drivers — may legitimately return structured `note` responses. -When creating or modifying CRUD JSON files in `json/`, validate EVERY field reference against the model definition in `models/`. Common field name mismatches found: - -| File | Wrong Field | Correct Field (from model) | -|------|-------------|----------------------------| -| opportunities_list.json | `org_id` | (does not exist — remove) | -| opportunities_list.json | `sales_stage` | `current_stage` | -| opportunities_list.json | `source` | `source_type` | -| sales_stages_list.json | `is_active` | `is_won_stage` / `is_lost_stage` | -| stage_history_list.json | `changed_by` | `changed_by_id` / `changed_by_name` | - -**Every CRUD JSON file MUST have:** -1. `tblname` root key matching a table in `models/` -2. `params` dict with at least `sortby` and `browserfields` -3. `editable` dict with `new_data_url`, `update_data_url`, `delete_data_url` (even if read-only, provide the URLs) -4. All field names in `browserfields.exclouded`, `browserfields.alters`, and `editexclouded` must exist in the model -5. `alters` entries must use `uitype: "code"` with `data` array — never nest `style` objects -6. `subtables[].url` must use `{{entire_url('../alias')}}` format with `../` prefix -7. `editor.binds[].actiontype` must be one of: `urlwidget`, `method`, `script`, `registerfunction`, `event` -When adding new fields to model definitions, ALWAYS update `init/data.json` seed data with the new fields. Missing fields in seed data cause configuration gaps after fresh deployment. - -### Model float/decimal Fields -float and decimal fields in model JSON MUST have BOTH `length` (int) and `dec` (int) as separate numeric keys. WRONG: `"length": "15,2"` (string). CORRECT: `"length": 15, "dec": 2`. - -### ID Generation: Always Use `getID()`, Never `uuid.uuid4()` -Database `id` columns are VARCHAR(32). `uuid.uuid4().replace('-', '')` produces a 32-char hex string that can exceed the column length and cause `DataError: (1406, "Data too long for column 'id' at row 1")`. - -**Always use `appPublic.uniqueID.getID()` for ID generation:** +### Keep base_tools.py Exports in Sync +Tool dicts (`file_tools`, `system_tools`, `skill_tools`, ...) are imported by BOTH `__init__.py` and `registration.py`. After modifying `base_tools.py`, verify all three are consistent — removing/renaming a dict or merging two (e.g. skill_tools into memory_tools) without updating the others → `ImportError`/`KeyError` at registration time. +### Tool Wrappers Must Accept `context` for User Isolation ```python -# WRONG - produces 32-char hex string, often too long: -import uuid -new_id = str(uuid.uuid4()).replace('-', '') +HERMES_DIR = os.path.expanduser("~/.hermes") -# CORRECT - produces compatible ID: -from appPublic.uniqueID import getID -new_id = getID() +def _get_user_dir(base_dir, context=None): + """User-isolated subdirectory; falls back to global dir if no user context.""" + user_id = (context or {}).get('user_id') or (context or {}).get('userid') + return os.path.join(base_dir, "users", str(user_id)) if user_id else base_dir + +async def wrapped_skill_manage(action, name, context=None, **kwargs): + user_dir = _get_user_dir(HERMES_DIR, context) + skills_dir = os.path.join(user_dir, "skills", name) + # ... file operations in user-isolated directory ``` -This applies to ALL Python backend code (core.py) AND `.dspy` API files. The `getID()` function uses the same ID generation scheme as the framework's `uniqueID` module, ensuring compatibility with all database column definitions. +**Context propagation chain** (every step must pass context): +1. `reasoning_console.wss` → `engine.reason_and_execute(user_id=X)` +2. `_execute_tool()` → `harnessed_execute_tool(tool, params, context={user_id: X})` +3. → `agent.execute_tool_call(tool, params, context)` +4. → `_execute_tool_with_retry()` injects `context` via `inspect.signature` — **CRITICAL: check signature first**, else tools without context fail "unexpected keyword argument" +5. Tool wrapper receives `context`, uses `_get_user_dir()` -### CRITICAL: Reuse Existing RBAC Login — Do NOT Write Your Own +User-isolated structure: `~/.hermes/users/{user_id}/` → `skills/` (SKILL.md per subdir), `memory.json`, `todo.json`, `tmp/` (execute_code temp files). Context-accepting wrappers: `wrapped_skill_manage`/`wrapped_skill_view`/`wrapped_skills_list` → `users/{user_id}/skills/`; `wrapped_memory` → memory.json; `wrapped_todo` → todo.json; `wrapped_execute_code` → tmp/. -When a Sage module needs login/authentication, ALWAYS use the existing RBAC user login system. Do NOT create a new `up_login.dspy` or `login.ui` in `sage/wwwroot/` or any module's `wwwroot/`. +### harnessed_execute_tool: Must Accept and Pass `context` +```python +# CORRECT — accepts and forwards context: +async def harnessed_execute_tool(tool_name, parameters, context=None): + return await get_harnessed_agent().execute_tool_call(tool_name, parameters, context) +``` +If `context` is omitted, the `user_id` embedded in it is lost and tools execute as `"anonymous"`. -| WRONG | CORRECT | -|-------|---------| -| Create `sage/wwwroot/up_login.dspy` | Use `rbac/user/up_login.dspy` | -| Create `sage/wwwroot/login.ui` | Use `rbac/user/login.ui` | -| Write custom password hashing | Use `ServerEnv.password_encode()` / `password_decode()` | +### _get_current_user_id Must NOT Raise +Return `"anonymous"` when context is missing, NOT raise ValueError (internal calls/system workflows often lack full context): +```python +def _get_current_user_id(self, context): + user_id = (context or {}).get('user_id') or (context or {}).get('userid') + return str(user_id) if user_id else "anonymous" +``` -RBAC login at `/rbac/user/login.ui` already handles: RC4 password encryption with `config.password_key`, account lockout detection, session management via `remember_user()`, and redirect to userinfo. All other modules authenticate via this shared session. +### Table Name: hermes_skills (NOT harnessed_skills) +Model is `models/hermes_skills.json`; all SQL in core.py must use `hermes_skills`. `harnessed_skills` does NOT exist → "table not found". -### Password Handling: Use ServerEnv, NOT rf (RegisterFunction) +### Per-User WebSocket Callbacks (NOT shared ws_push) +A shared `ws_push` attribute breaks with concurrent users (they overwrite each other's callbacks): +```python +class HermesReasoningEngine: + ws_push_callbacks: Dict[str, callable] = {} # per-user callbacks -In `.dspy` files that need password encryption (e.g., login forms, user creation): + async def _push(self, event_type, data=None, user_id=None): + if user_id and user_id in self.ws_push_callbacks: + await self.ws_push_callbacks[user_id]({'event': event_type, 'data': data}) +``` +In `reason_and_execute()`: set `self._current_user_id = user_id`, pass to all `_push()` calls; cleanup `finally: self._current_user_id = None`. In `.wss` endpoint: `engine.ws_push_callbacks[user_id] = callback`; cleanup `engine.ws_push_callbacks.pop(user_id, None)`. + +### Shared Skills Permission: Owner Org Only +Shared skills (`~/.hermes/skills/`) readable by ALL users, writable ONLY by owner organization users (`org_id='0'`); non-owner gets `"共享技能仅允许所有者机构用户修改"`. `reason_and_execute()` must set `self._current_org_id` from ServerEnv's `orgid`/`org_id` attribute and include it in the context dict: `context = {"user_id": user_id, "org_id": self._current_org_id, ...}`. + +### user_id Must Be in Context for Tool Execution +If `user_id` is missing from the context passed to `harnessed_execute_tool`, tools run as `'anonymous'` → permission checks and data isolation failures: +```python +context = await self._get_memory_context(user_id, request, config) +context['user_id'] = user_id # CRITICAL: must be in context for tool execution +# _get_memory_context initializes: {"user_id": user_id, "memory_entries": [], "recent_sessions": [], "skills": []} +``` + +## Reasoning Engine: Execution Details + +### LLM Call: Do NOT Pass `model` Parameter +`harnessed_reasoning/core.py` `_llm_call()` must NOT pass `model=` to `llm_chat()` — let `llm_chat` resolve `default_model` from `harnessed_agent_config`: +```python +# WRONG: result = await env.llm_chat(messages=messages, model=model, ...) +# CORRECT: result = await env.llm_chat(messages=messages, temperature=temperature, max_tokens=max_tokens, **extra) +``` + +### execute_immediately Parameter Parsing +Frontend may send `true` (boolean or string), not just `'1'` — support multiple truthy values: +```python +execute_val = str(params_kw.get('execute_immediately', '1')).lower() +execute_immediately = execute_val in ('1', 'true', 'yes', 'on') +``` + +### Module Function Signature Consistency +When a `.dspy` passes `user_id` to a module function, ALL functions in the call chain must accept it — else `unexpected keyword argument 'user_id'`: +```python +async def hermes_reason_and_execute(request: str, execute_immediately: bool = True, user_id: str = None): + return await engine.reason_and_execute(request, execute_immediately=execute_immediately, user_id=user_id) + +async def reason_and_execute(self, request: str, execute_immediately: bool = True, user_id: str = None): + if not user_id: user_id = "anonymous" +``` + +### Non-JSON LLM Response Handling +LLM API may return HTML error page / proxy block — check content-type before parsing and log the body: +```python +if resp.status == 200: + content_type = resp.content_type + if 'json' not in content_type: + err_text = await resp.text() + error(f"[llm_response] Non-JSON from {url}, Content-Type={content_type}") + error(f"[llm_response] Body (first 2000): {err_text[:2000]}") + return {'error': {'message': f'Non-JSON response ({content_type})', 'type': 'content_type_error'}} + return await resp.json() +``` + +### Store Session: JSON Serialization Safety +`json.dumps(plan)` may hit non-serializable types (datetime, custom objects). Clean before serialization (or use `default=str`): +```python +def clean_plan(obj): + if isinstance(obj, dict): return {k: clean_plan(v) for k, v in obj.items()} + if isinstance(obj, list): return [clean_plan(i) for i in obj] + if isinstance(obj, datetime): return obj.isoformat() + return obj + +data['execution_plan_json'] = json.dumps(clean_plan(plan), ensure_ascii=False) # or json.dumps(plan, default=str) +await sor.C('harnessed_reasoning_sessions', data) +``` + +### Database Table Must Exist Before Use +`Failed to store session: 'NoneType' object has no attribute 'get'` often means the table doesn't exist — `sqlor.getTableDesc()` returns `None` for missing tables and `C()` crashes on `None['fields']`. Ensure tables are created via `build.sh` before running. + +### LLM Config Database Isolation & Debugging +`llm_client.py` `_get_llm_config()` reads `harnessed_agent_config`. When called from another module (e.g. `integrated_crm_app` calling `llm_chat()`), lookup can fail (`Failed to fetch LLM config from DB 'default': 'NoneType' object has no attribute 'get'` → hardcoded `model=qwen3-max` fallback, NOT the user's configured value). Root causes: (1) db context — `env.get_module_dbname('harnessed_agent')` throws → falls back to `default`; (2) table missing in queried database; (3) `llm_api_key` stored encrypted → must decrypt via `env.password_decode()`. + +Correct implementation — try module DB first, then `default`; decrypt api_key; sort `updated_at desc`: +```python +async def _get_llm_config(): + dbnames_to_try = ['default'] + try: + env = ServerEnv() + module_db = env.get_module_dbname('harnessed_agent') + if module_db and module_db not in dbnames_to_try: + dbnames_to_try.insert(0, module_db) + except Exception as e: + error(f"[llm_config] Exception: {e}") + for dbname in dbnames_to_try: + try: + async with DBPools().sqlorContext(dbname) as sor: + ns = {'sort': 'updated_at desc'} + if user_id: ns['user_id'] = user_id + rows = (await sor.R('harnessed_agent_config', ns)) or [] + if rows: + row = rows[0] + if row.get('llm_api_key'): + row['llm_api_key'] = ServerEnv().password_decode(row['llm_api_key']) + return row + except Exception as e: + error(f"Failed to fetch LLM config from DB '{dbname}': {e}") + return {} +``` + +### Encrypted API Key Decryption +Fields stored encrypted (like `llm_api_key`, `api_key`) must be decrypted before use: +```python +api_key = row.get('llm_api_key', '') +if api_key: + api_key = ServerEnv().password_decode(api_key) +``` + +## CRITICAL: Reuse Existing RBAC Login — Do NOT Write Your Own + +ALWAYS use the existing RBAC user login system. NEVER create `up_login.dspy`/`login.ui` in `sage/wwwroot/` or any module's `wwwroot/`: +- Login page: `/rbac/user/login.ui`; login handler: `/rbac/user/up_login.dspy` +- RBAC already handles: RC4 password encryption with `config.password_key`, account lockout detection, session management via `remember_user()`, redirect to userinfo, multiple login methods (password, SMS, WeChat) +- All modules authenticate via this shared session; custom login in `sage/wwwroot/` breaks the RBAC auth flow → session inconsistencies + +## Password Handling: Use ServerEnv, NOT rf (RegisterFunction) ```python -# CORRECT — uses ServerEnv's password_encode from ahserver.globalEnv: +# CORRECT — ServerEnv's password_encode from ahserver.globalEnv (reads key from config.password_key, RC4): from ahserver.globalEnv import password_encode encrypted_pw = password_encode(params_kw.password) -# WRONG — do NOT use rfexe('password', ...) or app/rf.py: -await rfexe('password', params_kw) # This is deprecated/legacy +# WRONG — legacy RF pattern, do NOT use: +await rfexe('password', params_kw) # rf.register('password', ...) is deprecated; may not be registered ``` +Decrypt with `ServerEnv().password_decode(value)`. -The `rf` (RegisterFunction) pattern with `rf.register('password', ...)` is legacy. All new code must use `ServerEnv.password_encode()` / `password_decode()` which properly reads the encryption key from `config.password_key`. +## User ID Retrieval +- In `.dspy`: `userid = await get_user()` (returns user ID string); org: `await get_userorgid()` -### User ID Retrieval Patterns +## Sage Authentication & Cookie Pitfalls -**In `.dspy` files:** +### Cookie Secure Flag for HTTP Development +`ahserver/auth_api.py`'s `EncryptedCookieStorage` defaults to `secure=True` → blocks cookies on HTTP (localhost). Check the SSL value, not just key existence: ```python -userid = await get_user() # Returns user ID string +ssl_enabled = False +if hasattr(self.conf.website, 'ssl') and self.conf.website.ssl: # hasattr is True even when value is None! + ssl_cfg = self.conf.website.ssl + if hasattr(ssl_cfg, 'crtfile') and hasattr(ssl_cfg, 'keyfile'): + ssl_enabled = True +storage = EncryptedCookieStorage(secret, secure=ssl_enabled, # False for HTTP, True for HTTPS + samesite='Lax', httponly=True, max_age=24*60*60) ``` -### Password Encoding in .dspy Files +### Login Form Field Names & Users Table +- Form fields: `username` and `passwd` (NOT `loginid`/`password`); `passwd` encrypted via `password_encode()` +- Table `users` (NOT `user`); login matches on `username` (e.g. `superuser`), not `id`; `id` is the user ID (e.g. `user-01`) used for RBAC/session +- Schema: `id VARCHAR(32) PK`, `username VARCHAR(100)`, `password VARCHAR(255)`, `orgid VARCHAR(32)`, `user_status VARCHAR(1)` ('0'=active), `login_fail_count INT`, `last_login_fail DATETIME` -When a `.dspy` file needs to encrypt a password (e.g., for login forms), use `password_encode()` from `ahserver.globalEnv`, NOT the RF (register function) pattern: +### RBAC Permission Roles +| Role | Description | +|------|-------------| +| `anonymous` | Unauthenticated users | +| `any` | All users (including anonymous) | +| `logined` | Authenticated users only | +| `owner.*` | Owner organization roles | -```python -# CORRECT - use ServerEnv's password_encode directly: -from ahserver.globalEnv import password_encode -params_kw['password'] = password_encode(params_kw.password) +Feature pages requiring login → `logined`; public pages (login, registration) → `any`/`anonymous`. -# WRONG - don't use rfexe('password', params_kw): -await rfexe('password', params_kw) # RF mode may not be registered -``` +## Module Navigation & Adding Features -The `password_encode()` function automatically retrieves the password key from config and uses RC4 encryption. Always use this function rather than manually calling RC4 or relying on registered functions. - -### RBAC Login Convention - -**NEVER create custom login files in `sage/wwwroot/`.** The RBAC module provides a complete, battle-tested login system at `/rbac/user/login.ui` with: -- Password encryption via `password_encode()` -- Account lockout detection (failed attempt tracking) -- Session management via `remember_user()` -- Multiple login methods (password, SMS code, WeChat) - -If you need login functionality, always use the existing RBAC login endpoint: -- Login page: `/rbac/user/login.ui` -- Login handler: `/rbac/user/up_login.dspy` - -Creating duplicate login logic in `sage/wwwroot/` breaks the RBAC authentication flow and causes session inconsistencies. - -**CRITICAL Pitfall: User ID in JavaScript within HTML widgets of `.ui` files** - -When a `.ui` file contains an `HTML` widget with JavaScript that needs the current user ID (e.g., for WebSocket messages), you MUST use Jinja2 template injection `{{ get_user() }}`. JavaScript has no access to server-side session — it only sees the rendered HTML string. - -```javascript -// WRONG — hardcoded string literal; backend receives "current_user" not the real user ID -ws.send(JSON.stringify({cmd: 'connect', user_id: '{{ get_user() }}'})); -ws.send(JSON.stringify({cmd: 'start_reasoning', request: text, user_id: 'current_user'})); - -// CORRECT — Jinja2 template rendered server-side into the actual user ID -ws.send(JSON.stringify({cmd: 'connect', user_id: '{{ get_user() }}'})); -ws.send(JSON.stringify({cmd: 'start_reasoning', request: text, user_id: '{{ get_user() }}'})); -``` - -**Symptom**: WebSocket connects but shows "等待连接" status, reasoning requests fail silently, or all operations run as `user_id='anonymous'` because the backend receives the literal string `"current_user"`. - -**Rule**: Every JavaScript string that passes `user_id` to the server (WebSocket connect, command messages, AJAX calls) must use `{{ get_user() }}` template syntax when inside a `.ui` file's HTML widget. - -## CRITICAL: WebSocket UI Pattern — Reasoning Console Layout - -**Working pattern for reasoning console UI** (multi-line input + WebSocket + step timeline): - -### Recommended Layout Structure - -**APPROACH A: WebSocket widget + Html event handlers** (framework-managed connection) +### menu.ui Pattern +Sage modules are navigated via `menu.ui` files, not standalone `index.ui` pages. Main Sage `wwwroot/menu.ui` references module submenus: ```json -{ - "subwidgets": [ - { - "widgettype": "WebSocket", - "id": "reasoning_ws", - "options": {"ws_url": "{{entire_url('/wss/module/endpoint.wss')}}", "with_session": true}, - "binds": [ - {"wid": "self", "event": "onopen", "actiontype": "script", "script": "onWsOpen()"}, - {"wid": "self", "event": "ontext", "actiontype": "script", "script": "onWsMessage(event.params)"} - ] - }, - {"widgettype": "Html", "id": "ws_logic", "options": {"html": ""}} - ] -} +{"name": "llmage", "label": "模型管理", "submenu": "{{entire_url('/llmage/menu.ui')}}"} ``` - -**APPROACH B: Pure Html widget with raw JS WebSocket** (full control, recommended) +Module `menu.ui` (Menu widget; `"url"` = direct nav to page or CRUD alias, `"submenu"` = nested menu.ui, `"items"` = inline submenu, all URLs via `{{entire_url()}}`): ```json -{ - "subwidgets": [ - {"widgettype": "Html", "id": "ws_logic", "options": {"html": ""}} - ] -} +{"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"width": "60%", "height": "75%"}, "items": [ + {"name": "feature1", "label": "功能1", "url": "{{entire_url('/module/feature.ui')}}"}, + {"name": "feature2", "label": "功能2", "url": "{{entire_url('/module/alias_name')}}"} +]}} +``` +Menu with auth check (JSON validation fails on `.ui` files with Jinja2 templates — expected/normal): +```json +{"widgettype": "Menu", "options": {"target": "PopupWindow", "popup_options": {"archor": "cc", "width": "70%", "height": "70%"}, "cwidth": 10, "items": [ +{% if get_user() %}{"name": "entry_name", "label": "入口名", "url": "{{entire_url('page.ui')}}"}{% endif %} +]}} ``` -### Key rules -1. **Form with uitype "text" + height** gives multi-line textarea (not single-line Input) -2. **Form buttons** with `binds` on specific button names — buttons inside Form.options.buttons are handled by Form's internal submit mechanism. If you need custom click handlers (not form submit), use standalone `Button` widgets OUTSIDE the Form instead -3. **WebSocket binds on `bricks.WebSocket` are fragile**: event handler functions must exist in global scope BEFORE widget initialization. If the Html widget defining them appears after the WebSocket widget, you get "ReferenceError: onWsOpen is not defined". **APPROACH B (raw JS)** avoids this entirely -4. **Widgettype casing**: `Html` (mixed case), NOT `HTML`. `Scroll` does not exist — use VBox + `style: "overflow-y: auto;"` -5. **Session passing**: With `with_session: true`, bricks passes session via WebSocket protocol header (Sec-WebSocket-Protocol), which ahserver's WebsocketProcessor reads to identify the user -6. **JavaScript reads Form value** via `bricks.getWidgetById('input_form', bricks.app).get_value('user_input')` -7. **User ID in JS**: Use `{{ get_user() }}` Jinja2 template for server-side injection +### Adding New Module Features +1. Create the `.ui` file in module's `wwwroot/` +2. **Symlink to Sage wwwroot** (critical for local development): `cd ~/repos/sage/wwwroot/module_name && ln -sf ~/repos/module_name/wwwroot/new_feature.ui .` +3. Add menu entry in `module_name/wwwroot/menu.ui` +4. Register RBAC: `./py3/bin/python set_role_perm.py logined /module_name/new_feature.ui` +5. For API `.dspy` files: symlink into `module_name/api/` and register `/module_name/api/new_api.dspy` -### RBAC permissions for WebSocket -```bash -# Path MUST include /wss/ prefix (server logs confirm RBAC checks full path) -python set_role_perm.py "logined" "/wss/harnessed_reasoning/reasoning_console.wss" -``` +## WebSocket Real-Time Event Push (Reasoning Console Pattern) -## WebSocket-Based Real-Time Process Visualization +### WSS URL paths +See "WSS WebSocket URL Routing" under URL & Path Rules — `/wss/` prefix mandatory everywhere (UI `{{entire_url}}`, JS URL, RBAC registration, `set_role_perm.py` arg). Verify via server log `[debug] userid=None, path='/wss/...' permission check failed` — use that path verbatim. -### CRITICAL: WSS URL Paths — RBAC Checks Full Path with `/wss/` Prefix +### UI Layout (two approaches) +**A: WebSocket widget + Html handlers** (framework-managed): `{"widgettype": "WebSocket", "id": "...", "options": {"ws_url": "{{entire_url('/wss/module/endpoint.wss')}}", "with_session": true}, "binds": [{"wid": "self", "event": "onopen", "actiontype": "script", "script": "onWsOpen()"}, {"wid": "self", "event": "ontext", "actiontype": "script", "script": "onWsMessage(event.params)"}]}` plus an Html widget defining `window.onWsOpen`/`onWsMessage` — fragile ordering (see rule 3). +**B: Pure Html widget + raw JS WebSocket (recommended, full control)** — single Html widget with inline `\n\n" - } -} -``` - -### Event Flow Pattern - -``` -action_start -> step_context -> step_plan -> step_safety - -> execution_start - -> step_1_start -> tool_call -> tool_result -> step_1_complete - -> step_2_start -> tool_call -> tool_result -> step_2_complete - -> execution_complete - -> action_complete -``` - -### Message Format - -Server pushes messages with this structure: -```json -{"event": "step_name", "data": {"message": "Description", "...": "..."}, "timestamp": 1234567890} -``` - -Or for type-based routing: -```json -{"type": "error", "data": {"message": "Error details"}} -``` - -### User ID in WebSocket - -The `user_id` is passed from the frontend in the `connect` and command messages. For production, extract the real user ID from the websocket handshake headers (cookie/session) rather than trusting the frontend-provided value. - -### URL for WebSocket Connection - -In JavaScript, construct the WebSocket URL dynamically — **include `/wss/` prefix**: -```javascript -var protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'; -var url = protocol + '//' + window.location.host + '/wss/module_name/endpoint.wss'; -``` - -**Note**: RBAC permission registration for this endpoint uses the path WITH `/wss/`: +### RBAC for WebSocket ```bash python set_role_perm.py "logined" "/wss/harnessed_reasoning/reasoning_console.wss" ``` -Server logs confirm RBAC receives the full path including `/wss/` prefix. -The `.wss` extension is handled by ahserver's `WebsocketProcessor` which wraps the endpoint in `myfunc(request, **kwargs)` and provides `ws_pool` and `ws_data` in kwargs. - -### Sage Module Navigation: menu.ui Pattern - -Sage modules are navigated through `menu.ui` files, not standalone `index.ui` pages. The main Sage `wwwroot/menu.ui` references module submenus: - -```json -{ - "name": "llmage", - "label": "模型管理", - "submenu": "{{entire_url('/llmage/menu.ui')}}" -} -``` - -### Module menu.ui Structure - -Each module has its own `menu.ui` with items linking to feature pages: - -```json -{ - "widgettype": "Menu", - "options": { - "target": "PopupWindow", - "popup_options": {"width": "60%", "height": "75%"}, - "items": [ - {"name": "feature1", "label": "功能1", "url": "{{entire_url('/module/feature.ui')}}"}, - {"name": "feature2", "label": "功能2", "url": "{{entire_url('/module/alias_name')}}"} - ] - } -} -``` - -- `"url"`: Direct navigation to a page or CRUD alias -- `"submenu"`: Nested submenu loading another menu.ui -- `"items"`: Inline submenu items (alternative to submenu URL) -- Menu items use `{{entire_url()}}` for all URL values - -### Adding New Module Features - -When adding a new feature page to a module: - -1. Create the `.ui` file in module's `wwwroot/` -2. **Symlink to Sage wwwroot** (critical for local development): - ```bash - cd /home/hermesai/repos/sage/wwwroot/module_name - ln -sf /home/hermesai/repos/module_name/wwwroot/new_feature.ui . - ``` -3. Add menu entry in `module_name/wwwroot/menu.ui` -4. Register RBAC permissions: - ```bash - ./py3/bin/python set_role_perm.py logined /module_name/new_feature.ui - ``` -5. For API `.dspy` files, also symlink and register permissions: - ```bash - cd /home/hermesai/repos/sage/wwwroot/module_name/api - ln -sf /home/hermesai/repos/module_name/wwwroot/api/new_api.dspy . - ./py3/bin/python set_role_perm.py logined /module_name/api/new_api.dspy - ``` - -## Sage Authentication and Cookie Pitfalls - -### Cookie Secure Flag for HTTP Development - -`ahserver/auth_api.py`'s `EncryptedCookieStorage` defaults to `secure=True`, which blocks cookies on HTTP (localhost). For local HTTP development: - -```python -# Check if SSL is actually enabled (not just if key exists) -ssl_enabled = False -if hasattr(self.conf.website, 'ssl') and self.conf.website.ssl: - ssl_cfg = self.conf.website.ssl - if hasattr(ssl_cfg, 'crtfile') and hasattr(ssl_cfg, 'keyfile'): - ssl_enabled = True - -storage = EncryptedCookieStorage(secret, - secure=ssl_enabled, # False for HTTP, True for HTTPS - samesite='Lax', # Lax for same-site, None for cross-domain - httponly=True, - max_age=24*60*60 -) -``` - -**Pitfall**: `hasattr(config.website, 'ssl')` returns `True` even when the value is `None`. Always check `if self.conf.website.ssl:` before accessing attributes. - -### Login Form Field Names - -Sage login uses `username` and `passwd` fields (NOT `loginid` and `password`): -- Form field: `username` (maps to `users.username` column) -- Form field: `passwd` (encrypted via `password_encode()`) -- Database table: `users` (NOT `user`) -- `username` column stores the login identifier (e.g., `superuser`) -- `id` column is the user ID (e.g., `user-01`) - -### RBAC Permission Roles - -| Role | Description | -|------|-------------| -| `anonymous` | Unauthenticated users | -| `any` | All users (including anonymous) | -| `logined` | Authenticated users only | -| `owner.*` | Owner organization roles | - -For module feature pages that require login, use `logined` role. For public pages (login, registration), use `any` or `anonymous`. - -### WebSocket RBAC Paths: ALWAYS Include /wss/ Prefix - -**CRITICAL:** Server logs confirm RBAC checks the FULL path INCLUDING the `/wss/` prefix. Always include `/wss/` in RBAC permission registration. - -| Context | Path | Why | -|---------|------|-----| -| Frontend `{{entire_url(...)}}` | `/wss/module/endpoint.wss` | Nginx needs `/wss/` to route to WebSocket handler | -| RBAC permission registration | `/wss/module/endpoint.wss` | **MUST include `/wss/`** — server logs confirm RBAC receives full path | -| `set_role_perm.py` path arg | `/wss/module/endpoint.wss` | **MUST include `/wss/`** — must match what RBAC actually checks | - -**Example:** -```bash -# CORRECT - register WITH /wss/ -python set_role_perm.py "logined" "/wss/harnessed_reasoning/reasoning_console.wss" - -# WRONG - without /wss/ will fail RBAC check -python set_role_perm.py "logined" "/harnessed_reasoning/reasoning_console.wss" -``` - -**Verification**: Check server logs for the exact path RBAC receives: -``` -[debug] userid=None, path='/wss/harnessed_reasoning/reasoning_console.wss' permission check failed -``` -Use this path verbatim in `set_role_perm.py`. - -### Sage Multi-Process Deployment Architecture - -### Users Table Schema - -```sql -CREATE TABLE users ( - id VARCHAR(32) PRIMARY KEY, -- User ID (e.g., 'user-01') - username VARCHAR(100), -- Login name (e.g., 'superuser') - password VARCHAR(255), -- Encrypted password - orgid VARCHAR(32), -- Organization ID - user_status VARCHAR(1), -- '0'=active - login_fail_count INT, - last_login_fail DATETIME, - ... -); -``` - -**Important**: Login matches on `username`, not `id`. The `id` is used for RBAC and session tracking. - - -Fields stored encrypted in the database (like `llm_api_key`, `api_key`) must be decrypted before use. Use `ServerEnv.password_decode()`: - -```python -api_key = row.get('llm_api_key', '') -if api_key: - env = ServerEnv() - api_key = env.password_decode(api_key) -``` - -### harnessed_reasoning LLM Call: Do NOT Pass `model` Parameter - -`harnessed_reasoning/core.py`'s `_llm_call()` must NOT pass `model=` to `llm_chat()`. Let `llm_chat` use `default_model` from `harnessed_agent_config` table: - -```python -# WRONG — passes hardcoded/override model, overriding harnessed_agent_config.default_model: -result = await env.llm_chat(messages=messages, model=model, ...) - -# CORRECT — let llm_chat resolve model from config: -result = await env.llm_chat(messages=messages, temperature=temperature, max_tokens=max_tokens, **extra) -``` - -### execute_immediately Parameter Parsing - -Frontend may send `true` (boolean or string), not `'1'`. Must support multiple truthy values: - -```python -# WRONG — only accepts string '1': -execute_immediately = params_kw.get('execute_immediately', '1') == '1' - -# CORRECT — supports '1', 'true', 'yes', 'on': -execute_val = str(params_kw.get('execute_immediately', '1')).lower() -execute_immediately = execute_val in ('1', 'true', 'yes', 'on') -``` - -### Module Function Signature Consistency - -When a `.dspy` file passes `user_id` to a Python module function, ALL functions in the call chain must accept it as a parameter: - -```python -# .dspy: -user_id = await get_user() -result = await hermes_reason_and_execute(request=text, user_id=user_id, ...) - -# Python entry function: -async def hermes_reason_and_execute(request: str, execute_immediately: bool = True, user_id: str = None): - engine = get_harnessed_reasoning_engine() - return await engine.reason_and_execute(request, execute_immediately=execute_immediately, user_id=user_id) - -# Python method: -async def reason_and_execute(self, request: str, execute_immediately: bool = True, user_id: str = None): - if not user_id: - user_id = "anonymous" -``` - -**Pitfall**: If any function in the chain doesn't accept `user_id`, you get `unexpected keyword argument 'user_id'`. - -### harnessed_agent Tool Implementation: No Mocks - -The tool wrappers in `harnessed_agent/tools/base_tools.py` **must execute real operations**, not return `status: "mock_implementation"` dicts. If tools return mock results, the reasoning engine will report fake success and the LLM will hallucinate file locations and content. - -**Verify real implementations exist for core tools:** -- `read_file` / `write_file` / `search_files` / `patch` — actual Python file I/O -- `terminal` — `asyncio.create_subprocess_shell` -- `execute_code` — writes to temp `.py` file, runs via `python3` -- `memory` — reads/writes `~/.hermes/memory.json` -- `skill_view` / `skills_list` — scans `~/.hermes/skills/` directory -- `todo` — reads/writes `~/.hermes/todo.json` - -Browser tools (`browser_*`), vision (`vision_analyze`), and TTS (`text_to_speech`) require external drivers/APIs and can legitimately return structured `note` responses. - -### harnessed_execute_tool: Must Accept and Pass `context` Parameter - -The global entry function `harnessed_execute_tool` MUST accept `context` and pass it through to `agent.execute_tool_call()`. If `context` is omitted, the `user_id` embedded in it is lost and tools execute as `"anonymous"`. - -```python -# WRONG — context parameter missing, user_id dropped: -async def harnessed_execute_tool(tool_name: str, parameters: Dict[str, Any]): - agent = get_harnessed_agent() - return await agent.execute_tool_call(tool_name, parameters) - -# CORRECT — accepts and forwards context: -async def harnessed_execute_tool(tool_name: str, parameters: Dict[str, Any], context: Dict[str, Any] = None): - agent = get_harnessed_agent() - return await agent.execute_tool_call(tool_name, parameters, context) -``` - -### harnessed_agent Tool Registration: Keep base_tools.py Exports in Sync - -`base_tools.py` defines tool dictionaries (`file_tools`, `system_tools`, `skill_tools`, etc.) that are imported by both `__init__.py` and `registration.py`. If you remove or rename a dictionary in `base_tools.py` but forget to update the other two files, you get `ImportError` or `KeyError` at tool registration time. - -**Rule**: After modifying `base_tools.py`, verify these three files are consistent: -- `base_tools.py` — dictionary definitions at the bottom -- `__init__.py` — import statements -- `registration.py` — import statements + `_register_*_tools` function calls - -A common pitfall is merging two dictionaries (e.g., putting `skill_tools` entries into `memory_tools`) and then `registration.py` still expects `skill_tools` to exist independently. - -### Tool Wrappers Must Accept `context` Parameter for User Isolation - -Tool wrappers in `harnessed_agent/tools/base_tools.py` that access file state (memory, skills, todo, temp files) **MUST accept an optional `context` parameter** and use `_get_user_dir()` to resolve user-isolated paths. - -**Complete pattern:** - -```python -# In base_tools.py: -HERMES_DIR = os.path.expanduser("~/.hermes") - -def _get_user_dir(base_dir: str, context: Optional[Dict[str, Any]] = None) -> str: - """Get user-isolated subdirectory. Falls back to global dir if no user context.""" - user_id = None - if context: - user_id = context.get('user_id') or context.get('userid') - if user_id: - return os.path.join(base_dir, "users", str(user_id)) - return base_dir - -# Tool wrapper example: -async def wrapped_skill_manage(action: str, name: str, context: Optional[Dict[str, Any]] = None, **kwargs): - user_dir = _get_user_dir(HERMES_DIR, context) - skills_dir = os.path.join(user_dir, "skills", name) - # ... file operations in user-isolated directory -``` - -**Context propagation chain** (every step must pass context): -1. `reasoning_console.wss` -> `engine.reason_and_execute(user_id=X)` -2. `_execute_tool()` -> `harnessed_execute_tool(tool, params, context={user_id: X})` -3. `harnessed_execute_tool()` -> `agent.execute_tool_call(tool, params, context)` -4. `_execute_tool_with_retry()` -> injects `context` into params via inspect.signature -5. Tool wrapper receives `context` kwarg, uses `_get_user_dir()` - -**CRITICAL**: `_execute_tool_with_retry` MUST check `inspect.signature` before injecting context, otherwise tools that don't accept context will fail with "unexpected keyword argument". - -**User-isolated directory structure:** -``` -~/.hermes/users/{user_id}/ -├── skills/ # User-created skills (SKILL.md per subdirectory) -├── memory.json # User-specific memory -├── todo.json # User-specific todo list -└── tmp/ # User-specific temp files (execute_code) -``` - -### _get_current_user_id Must NOT Raise ValueError - -`HermesAgent._get_current_user_id()` must return `"anonymous"` when context is missing, NOT raise `ValueError`. Internal tool calls and system workflows often don't pass full context. - -```python -def _get_current_user_id(self, context: Dict[str, Any]) -> str: - user_id = None - if context: - user_id = context.get('user_id') or context.get('userid') - if not user_id: - return "anonymous" # NOT raise ValueError - return str(user_id) -``` - -### Table Name: hermes_skills (NOT harnessed_skills) - -The skills table is named `hermes_skills` in the model definition (`models/hermes_skills.json`). All SQL operations in `core.py` must use `hermes_skills`. The table name `harnessed_skills` does NOT exist and will cause "table not found" errors. - -### Per-User WebSocket Callbacks for Reasoning Engine - -`HermesReasoningEngine` must use per-user WebSocket callbacks, not a shared `ws_push` attribute. Multiple concurrent users will overwrite each other's callbacks if using a single shared attribute. - -**Correct pattern:** -```python -class HermesReasoningEngine: - ws_push_callbacks: Dict[str, callable] = {} # Per-user callbacks - _current_user_id = None # Set during execution - - async def _push(self, event_type, data=None, user_id=None): - if user_id and user_id in self.ws_push_callbacks: - await self.ws_push_callbacks[user_id]({'event': event_type, 'data': data}) -``` - -In `reason_and_execute()`: set `self._current_user_id = user_id`, pass to all `_push()` calls. Clean up in `finally: self._current_user_id = None`. - -In `.wss` endpoint: use `engine.ws_push_callbacks[user_id] = callback`, cleanup with `engine.ws_push_callbacks.pop(user_id, None)`. - -### harnessed_reasoning Tool Execution: context Must Be Passed Through Entire Chain - -The reasoning engine's `_execute_tool` MUST pass `context` to `harnessed_execute_tool`. Without it, tool wrappers receive no user context and all file operations (skills, memory, todo, temp files) go to the global `~/.hermes/` directory instead of user-isolated paths. - -**Critical chain (all links required):** -``` -reasoning/core.py _execute_tool() -> env.harnessed_execute_tool(tool, params, context) - -> harnessed_agent/core.py execute_tool_call(tool, params, context) - -> _execute_tool_with_retry(func, params, ..., context) - -> inspects function signature; if 'context' in params, injects it - -> wrapped_skill_manage(..., context) # user isolation activated -``` - -**Tool wrappers that accept `context` parameter:** -- `wrapped_skill_manage`, `wrapped_skill_view`, `wrapped_skills_list` -> `~/.hermes/users/{user_id}/skills/` -- `wrapped_memory` -> `~/.hermes/users/{user_id}/memory.json` -- `wrapped_todo` -> `~/.hermes/users/{user_id}/todo.json` -- `wrapped_execute_code` -> `~/.hermes/users/{user_id}/tmp/` - -### Reasoning Engine: WebSocket Push Must Be Per-User - -The reasoning engine MUST NOT use a single shared `ws_push` callback. Use `ws_push_callbacks: Dict[str, callable]` keyed by `user_id` to prevent cross-user event leakage: - -```python -# Class attribute (not instance): -ws_push_callbacks: Dict[str, callable] = {} - -# Set during _run_reasoning: -engine.ws_push_callbacks[user_id] = lambda msg: _ws_push(user_id, msg) - -# In _push(): -if user_id and user_id in self.ws_push_callbacks: - await self.ws_push_callbacks[user_id](msg) - -# Cleanup in finally: -engine.ws_push_callbacks.pop(user_id, None) -``` - -### Shared Skills Permission: Owner Org Only - -Shared skills (`~/.hermes/skills/`) are readable by ALL users but writable ONLY by owner organization users (org_id='0'). Non-owner attempts to modify shared skills receive `"共享技能仅允许所有者机构用户修改"`. - -### Reasoning Engine: org_id Must Be in Context for Shared Skill Checks - -`reason_and_execute()` must set `self._current_org_id` from ServerEnv's `orgid`/`org_id` attribute, then include it in the context dict passed to tool execution: -```python -context = {"user_id": user_id, "org_id": self._current_org_id, ...} -``` - -### harnessed_reasoning Tool Execution: user_id Must Be in Context - -The reasoning engine's `_execute_plan` passes a `context` dict to `harnessed_execute_tool`. If `user_id` is missing from this context, tools execute as `user_id='anonymous'`, causing permission checks and data isolation failures. - -**Ensure `user_id` is injected into the context dict before tool execution:** - -```python -# In hermes_reason_and_execute (reasoning entry point): -context = await self._get_memory_context(user_id, request, config) -context['user_id'] = user_id # CRITICAL: must be in context for tool execution - -# Then in _execute_plan, this context is passed to each tool call: -tool_result = await self._execute_tool(tool, params, context) -``` - -Also ensure `_get_memory_context` initializes context with `user_id`: -```python -context = {"user_id": user_id, "memory_entries": [], "recent_sessions": [], "skills": []} -``` - -### Non-JSON LLM Response Handling - -When LLM API returns non-JSON (e.g., HTML error page, proxy block), check content-type before parsing and log the body: - -```python -if resp.status == 200: - content_type = resp.content_type - if 'json' not in content_type: - err_text = await resp.text() - error(f"[llm_response] Non-JSON from {url}, Content-Type={content_type}") - error(f"[llm_response] Body (first 2000): {err_text[:2000]}") - return {'error': {'message': f'Non-JSON response ({content_type})', 'type': 'content_type_error'}} - - return await resp.json() -``` - -### Store Session: JSON Serialization Safety - -When storing reasoning sessions with `json.dumps(plan)`, the plan object may contain non-serializable types (datetime, custom objects). Clean the plan before serialization: - -```python -def clean_plan(obj): - if isinstance(obj, dict): - return {k: clean_plan(v) for k, v in obj.items()} - elif isinstance(obj, list): - return [clean_plan(i) for i in obj] - elif isinstance(obj, datetime): - return obj.isoformat() - return obj - -clean_plan_obj = clean_plan(plan) -data['execution_plan_json'] = json.dumps(clean_plan_obj, ensure_ascii=False) -await sor.C('harnessed_reasoning_sessions', data) -``` - -### Database Table Must Exist Before Use - -`Failed to store session: 'NoneType' object has no attribute 'get'` often means the database table doesn't exist. `sqlor`'s `getTableDesc()` returns `None` if the table is missing, and `C()` crashes when accessing `None['fields']`. - -Always ensure tables are created via `build.sh` before running the module. - -## LLM Config Database Isolation & Debugging - -**LLM configuration lookup must correctly handle the caller module's database context and user isolation.** - -### The Bug Pattern - -`harnessed_agent`'s `llm_client.py` uses `_get_llm_config()` to read from `harnessed_agent_config` table. When called by another module (e.g., `integrated_crm_app` calling `llm_chat()`), the config lookup may fail: - -``` -Failed to fetch LLM config from DB 'default': 'NoneType' object has no attribute 'get' -LLM config not found in any database -LLM chat: model=qwen3-max <-- hardcoded fallback, NOT the user's configured value -``` - -### Root Cause Analysis - -1. **Database context**: `_get_llm_config()` uses `env.get_module_dbname('harnessed_agent')` to find the correct database. If this throws, it falls back to `default`. -2. **Missing data**: The table may not exist in the queried database. -3. **Encrypted fields**: `llm_api_key` is stored encrypted and must be decrypted via `env.password_decode()`. - -### Correct Implementation - -```python -async def _get_llm_config() -> Dict[str, Any]: - dbnames_to_try = ['default'] - try: - env = ServerEnv() - module_db = env.get_module_dbname('harnessed_agent') - if module_db and module_db not in dbnames_to_try: - dbnames_to_try.insert(0, module_db) - except Exception as e: - error(f"[llm_config] Exception: {e}") - - for dbname in dbnames_to_try: - try: - async with DBPools().sqlorContext(dbname) as sor: - ns = {'sort': 'updated_at desc'} - if user_id: - ns['user_id'] = user_id - rows = await sor.R('harnessed_agent_config', ns) - rows = rows or [] - if rows: - row = rows[0] - api_key = row.get('llm_api_key', '') - if api_key: - env = ServerEnv() - api_key = env.password_decode(api_key) - return row - except Exception as e: - error(f"Failed to fetch LLM config from DB '{dbname}': {e}") - - return {} -``` - -### harnessed_reasoning LLM Call: Do NOT Pass `model` Parameter, ns) - rows = rows or [] - if rows: - return rows[0] - else: - warning(f"No rows in DB '{dbname}' for user_id={repr(user_id)}") - except Exception as e: - error(f"Failed to fetch LLM config from DB '{dbname}': {type(e).__name__}: {e}") - - error("LLM config not found in any database") - return {} -``` - -### Encrypted API Key Decryption - -Fields stored encrypted in the database (like `llm_api_key`) must be decrypted before use: - -```python -api_key = row.get('llm_api_key', '') -if api_key: - env = ServerEnv() - api_key = env.password_decode(api_key) -``` - -### harnessed_reasoning LLM Call: Do NOT Pass `model` Parameter - -`harnessed_reasoning/core.py`'s `_llm_call()` must NOT pass `model=` to `llm_chat()`. Let `llm_chat` use `default_model` from `harnessed_agent_config` table: - -```python -# WRONG — passes hardcoded/override model, overriding harnessed_agent_config.default_model: -result = await env.llm_chat(messages=messages, model=model, ...) - -# CORRECT — let llm_chat resolve model from config: -result = await env.llm_chat(messages=messages, temperature=temperature, max_tokens=max_tokens, **extra) -``` - -### User ID Retrieval Patterns - -### harnessed_reasoning LLM Call: Do NOT Pass `model` Parameterllm_response] Non-JSON from {url}, Content-Type={content_type}") - error(f"[llm_response] Body (first 2000): {err_text[:2000]}") - return {'error': {'message': f'Non-JSON response ({content_type})', 'type': 'content_type_error'}} - -return await resp.json() -``` - -### Store Session: JSON Serialization Safety - -When storing reasoning sessions with `json.dumps(plan)`, the plan object may contain non-serializable types (datetime, custom objects). Always use `default=str`: - -```python -plan_str = json.dumps(plan, ensure_ascii=False, default=str) -data = { - 'execution_plan_json': plan_str, - ... -} -await sor.C('harnessed_reasoning_sessions', data) -``` - -### Module Function Signature: Accept user_id Parameter - -When a `.dspy` file passes `user_id` to a Python module function, the module function MUST accept it as a parameter: - -```python -# Python module function - accept user_id explicitly: -async def reason_and_execute(self, request: str, execute_immediately: bool = True, user_id: str = None): - if not user_id: - user_id = "anonymous" - ... -``` - -**Pitfall**: If the entry function passes `user_id` but the underlying method doesn't accept it, you get `unexpected keyword argument 'user_id'`. +### "等待连接" / WebSocket never connects — checklist +1. HTML widget JS has hardcoded `user_id: 'current_user'` → must be `{{ get_user() }}` +2. RBAC permission for the `.wss` path missing or registered without `/wss/` prefix (server log shows the exact path RBAC checks) +3. Redis not running (Sage sessions depend on `redis://127.0.0.1:6379`) +4. `.wss` file missing at `wwwroot/endpoint.wss` or doesn't define `async def myfunc(request, **kwargs)` diff --git a/skills_library/all/module-development-spec/SKILL.md b/skills_library/all/module-development-spec/SKILL.md index 219a1e5..d9977d8 100644 --- a/skills_library/all/module-development-spec/SKILL.md +++ b/skills_library/all/module-development-spec/SKILL.md @@ -9,712 +9,168 @@ trigger_conditions: - Task involves creating read-only/dashboard modules that display data without CRUD (no models/json needed) --- -## Work logs / delivery archive +## Work Logs / Delivery Archive -For production module/application development, update a dated work log before finishing the session. Use the actual current date from the system (do not assume the date from earlier messages) and keep late-night follow-up fixes under their real date or explicitly label them as next-day follow-ups. If no log file exists, create one under an appropriate docs/archive path such as `docs/work-log-YYYY-MM-DD.md`. +For production module/application development, update a dated work log (use the REAL current system date; late-night follow-up fixes go under their real date or are labeled next-day follow-ups) before finishing, e.g. `docs/work-log-YYYY-MM-DD.md`. Entry should include: scope/background + repo/module name; timeline/commit list; key technical decisions and pitfalls; verification performed (incl. environment-limited checks that couldn't run); current branch/commit state. Do NOT mark a task done without this archive when deliverables must be retained. -A useful log entry should include: -- Scope/background and repository/module name. -- Timeline or commit list for the date. -- Key technical decisions and pitfalls discovered. -- Verification performed and any environment-limited verification that could not run. -- Current branch/commit state. - -Do not leave a task as "done" without this archive when the user expects deliverables to be retained for later inspection. - - -This skill defines the complete workflow for developing standardized modules that integrate with the ahserver ecosystem using bricks-framework for frontend and sqlor for database operations. +This skill defines the complete workflow for standardized modules: ahserver ecosystem + bricks-framework frontend + sqlor database backend. ## Module Philosophy — Host-Agnostic -**Modules are NOT tied to any specific application.** A module is a self-contained unit that any host can load: -- pipeline-app loads it → it's pipeline-app's module -- sage loads it → it's sage's module -- Any future application loads it → it's that application's module - -**Core principle**: A module only depends on: -1. Foundation packages (sqlor, ahserver for ServerEnv, appPublic for utilities) -2. Its own data tables (or shared tables like sage's appcodes) -3. Other modules it explicitly imports - -A module does NOT depend on: -- Any specific host application's entry point -- Host-specific configuration files -- Host-specific wwwroot paths - -The `load_{module}()` function is the ONLY integration point. It registers functions to ServerEnv. The host decides how to wire it in. - -**Interaction-layer modules** (like pipeline-task) have NO data tables — they are pure thin wrappers that call other modules' functions via ServerEnv. They provide .dspy + .ui files for user interaction only. +- A module is a self-contained unit ANY host can load (pipeline-app / sage / future apps). It depends ONLY on: ① foundation packages (sqlor, ahserver ServerEnv, appPublic utilities); ② its own data tables (or shared tables like sage's appcodes); ③ other modules it explicitly imports. +- It does NOT depend on any host entry point, host-specific config files, or host wwwroot paths. +- `load_{module}()` is the ONLY integration point — it registers functions to ServerEnv; the host decides how to wire it in. +- **Interaction-layer modules** (e.g. pipeline-task) have NO data tables — pure thin wrappers calling other modules' functions via ServerEnv; they provide .dspy + .ui only. ## Directory Structure + ``` -mymodule/ # Main module directory (replace mymodule with actual name) -├── mymodule/ # Python package directory -│ ├── __init__.py # Required Python package file -│ ├── init.py # Module initialization script -│ └── *.py # Additional source files -├── wwwroot/ # Frontend scripts and resources (.ui, .dspy files) -├── models/ # Database table definitions as JSON files -├── json/ # CRUD operation definitions as JSON files -├── init/ # Module initialization data -│ └── data.json # Initial data in specified format -├── scripts/ # Supporting scripts -│ └── load_path.py # RBAC permission registration script (see RBAC section below) -├── skill/ # MANDATORY — AI agent reference documentation -│ └── SKILL.md # Agent-facing module spec: data model, endpoints, pitfalls -├── pyproject.toml # Python packaging configuration -└── README.md # Module documentation +mymodule/ # module root (replace with actual name) +├── mymodule/ # Python package: __init__.py (required), init.py, *.py +├── wwwroot/ # frontend: .ui / .dspy / .js / .css +├── models/ # {tablename}.json database table definitions +├── json/ # {alias}.json CRUD operation definitions +├── init/ # data.json / data.yaml initial data +├── scripts/ # supporting scripts (load_path.py RBAC registration) +├── skill/SKILL.md # MANDATORY agent-facing spec (data model, endpoints, pitfalls) +├── pyproject.toml # Python packaging +└── README.md # module documentation ``` ## Core Implementation Requirements ### 1. Module Initialization (init.py) -- **Primary purpose**: Register all module functions with ServerEnv instance so they can be called directly from .ui and .dspy files -- **NOT for route registration**: wwwroot files are automatically routed via `/{module_name}/filename.ext` -- **ServerEnv configuration is REQUIRED for all modules**: Every module must register its functions with ServerEnv in the `load_{modulename}()` function -- **CRITICAL: Export functions in __init__.py**: All async functions defined in `init.py` must be explicitly imported in the package's `__init__.py` file, otherwise Sage framework cannot find them and dspy calls will fail with `NameError: name 'xxx' is not defined` -**Pitfall: Triple-place function registration — adding or removing functions requires updating THREE files** - -When adding or removing a module function, THREE files must be updated in sync: -1. **`mymodule/mymodule.py`** (or wherever the function is defined) — the function implementation -2. **`mymodule/__init__.py`** — the import/export line -3. **`mymodule/init.py`** — the `env.xxx = xxx` registration in `load_{module}()` - -Forgetting any one causes runtime errors: -- Missing #2 → `ImportError` or `AttributeError` when init.py tries to import -- Missing #3 → `NameError: name 'xxx' is not defined` in .dspy/.ui templates -- Removing from #1 but not #2/#3 → `ImportError` at module load time - -**Cleanup checklist when removing a function:** -``` -grep -rn 'function_name' mymodule/ --include='*.py' -# Must appear in exactly: definition, __init__.py import, init.py env registration -# Remove from ALL three, then commit. -``` - -Example `__init__.py` (required): -```python -# mymodule/__init__.py -from .init import ( - create_user, - get_user_list, - update_user_profile, - # ... export all public API functions -) -``` - -- Keep init.py focused on ServerEnv registration and minimal initialization -- Include a `load_{modulename}()` function that registers all public functions with ServerEnv -- Functions registered with ServerEnv become directly callable from .ui templates and .dspy scripts - -Example structure: -```python -from ahserver.serverenv import ServerEnv - -# Module metadata and constants -MODULE_NAME = "mymodule" -MODULE_VERSION = "1.0.0" - -# Helper functions for web scripts -def get_data(): - """Get data - will be registered with ServerEnv""" - pass - -def save_data(data): - """Save data - will be registered with ServerEnv""" - pass - -def load_mymodule(): - """Register all functions with ServerEnv so they can be called from .ui/.dspy files""" - env = ServerEnv() - env.get_data = get_data - env.save_data = save_data - return True -``` - -**Pitfall: CRUD dspy wrappers use plural table names — register both singular and plural** - -The `xls2ui` tool generates CRUD wrapper `.dspy` files (e.g., `add_suppliers.dspy`) that reference functions with **plural** names matching the table name: `create_suppliers`, `update_suppliers`, `delete_suppliers`. But `init.py` typically defines functions with **singular** names (e.g., `create_supplier`). If only the singular name is registered, the CRUD UI will get `NoneType` errors. - -**Fix**: Register both forms in `load_{module}()`: -```python -env.create_supplier = create_supplier -env.create_suppliers = create_supplier # xls2ui uses plural -env.update_supplier = update_supplier -env.update_suppliers = update_supplier -env.delete_supplier = delete_supplier -env.delete_suppliers = delete_supplier -``` - -**Symptom**: CRUD "add" button returns 500 with `function not found` — the wrapper dspy calls `create_suppliers` but only `create_supplier` was registered. - -**Pitfall: Subagent Delegation for Module Creation — ALWAYS Validate After** - -When using `delegate_task` to create modules in parallel, subagents **consistently** produce errors even when given correct spec instructions (observed 2/3 failure rate): - -1. **File misplacement**: `init.py` and `__init__.py` placed at module root (`mymodule/init.py`) instead of package dir (`mymodule/mymodule/init.py`) -2. **Wrong model JSON format**: Uses `{"table":"...", "fields":{...}}` instead of `{"summary":[...], "fields":[...], "indexes":[...], "codes":[...]}` -3. **Wrong CRUD JSON format**: Uses `{"table":"...", "list":{...}}` instead of `{"tblname":"...", "params":{"browserfields":{...}, "editable":{...}}}` -4. **Hallucinated sqlor APIs**: Invents `sqlor.save/list/one/delete/insert/query` — these methods DO NOT EXIST. Only `sor.C/U/D/R/I/sqlExe` are valid. - -**Mandatory post-delegation validation** (see `references/bulk-module-creation-pattern.md` "Subagent Validation Checklist"): -- Move misplaced init.py/__init__.py to package dir -- Validate all model JSONs have `summary` key with array primary -- Validate all CRUD JSONs have `tblname` and `params.editable` -- Grep for hallucinated sqlor APIs and fix to sor.C/U/D pattern -- Run dspy audit (no imports, no print, no uuid) -- Run py_compile on all .py files - -**Pitfall: debug() in dspy does NOT include filename — always prefix manually** - -`debug()` calls in `.dspy` files log as `[sage][debug][:N]` — the framework injects dspy code into a `` context, so the file path is never shown. When tracing production issues across multiple dspy files, all debug output looks identical. - -**Fix**: Always prefix debug calls with the file name: - -```python -# WRONG — no way to tell which dspy this came from -debug(f'{params_kw=}') - -# CORRECT — immediately identifiable in logs -debug(f'product_category_create.dspy: START params_kw={dict(params_kw)}') -debug(f'get_product_category.dspy: ns keys={list(ns.keys())}') -``` - -This is essential when debugging multi-step flows (e.g. add → refresh) where several dspy files execute in sequence. Each debug line becomes self-identifying. - -**Pitfall: DSPY API handlers must forward ALL client params — never hardcode dispatch fields** - -When a `.dspy` API endpoint acts as a task dispatcher (submitting to a longtasks worker or downstream service), it MUST forward the client's parameters. Hardcoding dispatch fields like `task_type` silently ignores client intent: - -```python -# WRONG — hardcodes task_type, ignores client's task_type -payload = {'task_type': 'separate', 'task_id': task_id, 'audio_path': audio_path} -await longtasks.submit_task(payload) - -# CORRECT — reads task_type from request, forwards all params -task_type = params_kw.get('task_type', 'separate') -payload = {'task_type': task_type, 'task_id': task_id, 'audio_path': audio_path} -if params_kw.get('output_dir'): - payload['output_dir'] = params_kw['output_dir'] -await longtasks.submit_task(payload) -``` - -**Symptom**: Client sends `task_type: "separate_full"` but worker log shows `task_type: "separate"`. Worker always runs the default mode regardless of what client requested. Takes hours to trace because the worker code is correct — the DSPY is silently dropping the parameter. - -**Audit**: grep API DSPY files for hardcoded `'task_type'`, `'mode'`, or similar dispatch strings inside payload construction. - -**Pitfall: py_compile is INVALID for .dspy files** - -`.dspy` files are injected into an async function context at runtime — `return` at top level is correct. Running `py_compile` on `.dspy` files produces false positives (`'return' outside function`, `'await' outside function`). Only use `py_compile` for `.py` files (like `init.py`). For `.dspy` validation, use the dspy audit script (grep-based checks). - -**CRITICAL: dspy MUST use explicit `return` — ahserver wraps code in `async def`.** The dspy handler prepends `async def myfunc(request, **ns):` and awaits the function. Setting `result` as a bare expression without `return` causes the function to return `None`. See `references/dspy-execution-and-module-structure.md` for the full execution model, pre-loaded globals, Tree widget data format, and module symlink patterns. - -**Pitfall: Sage startup requires environment variables — cannot test locally without them** - -Sage's `app/sage.py` imports modules that read env vars at import time (e.g., `ALIPAY_PUB` for payment keys). Starting Sage locally without these vars causes `FileNotFoundError`. Use syntax checks (`py_compile` for .py files) and dspy audit scripts instead of full startup for local validation. - -**Pitfall: Raw SQL column names — ALWAYS verify against model.json, never guess** - -When writing raw SQL (`sor.sqlExe`) or ORM calls that reference table columns, ALWAYS verify the column names against the actual model definition (`models/{table}.json`), not from memory or similar tables. Guessing column names (e.g., `permcode` instead of `path`, `permname` instead of `name`) causes hard-to-debug runtime errors that require multiple fix attempts. - -**Check before writing SQL:** -```bash -python3 -c "import json; cols=[f['name'] for f in json.load(open('models/table.json'))['fields']]; print(cols)" -``` - -**Common wrong guesses → correct names in Sage:** -| Table | Wrong guess | Correct (from model) | -|-------|-----------|---------------------| -| permission | permcode, permname | path, name | -| users | created_date | created_at | -| organization | org_name | orgname | - - - -**Pitfall: `{{id}}` in uapi response templates resolves to Python built-in `id()`** - -In Sage uapi response templates, Jinja2 resolves `{{id}}` as Python's `id()` built-in function, not the upstream API's `id` field. This produces `` in the response instead of the actual task/request ID. - -**Fix**: Never use `{{id}}` in uapi response templates. Always map to a different field name: -```python -# WRONG — {{id}} resolves to Python id() function -'{"taskid":"{{id}}","status":"{{status}}"}' - -# CORRECT — map via request context or use a different variable -'{"taskid":"{{taskid}}","status":"{{status}}"}' -``` - -**Affected**: Any uapi where the upstream response has a field called `id`. Sage's Jinja2 context silently shadows it with the Python built-in. - -**Pitfall: uapiio `input_fields` MUST be a JSON array for bricks.js compatibility** - -The bricks.js `LlmIO` widget calls `this.input_fields.forEach(...)`. If `input_fields` is a JSON object `{"field": {...}}` instead of a JSON array `[{"name":"field", ...}]`, the browser throws: -``` -TypeError: this.input_fields.forEach is not a function -``` - -**Fix**: Always format uapiio `input_fields` as an array of objects: -```json -// WRONG — object format, bricks.js can't iterate -{"prompt":{"type":"string","required":true}} - -// CORRECT — array format with name/label/uitype -[{"name":"prompt","label":"用户输入","uitype":"text","required":true}] -``` - -**Reference**: Use existing KTV uapiio records (e.g., `ktv_asr_transcribe_io`) as the canonical format reference. - -**Pitfall: `json.dumps()` in uapi data templates causes `data=None`** - -When a uapi data template uses `{{json.dumps(prompt, ensure_ascii=False)}}`, the Jinja2 rendering can fail silently, resulting in the HTTP request body being `None`. This logs as `body=None` in the uapi debug output. - -**Fix**: Use plain string interpolation `"{{prompt}}"` instead of `json.dumps()`. The model name should be hardcoded to the upstream service ID (e.g., `"model":"hy-image-v3.0"`), not passed through `{{model}}`. - -```python -# WRONG — json.dumps can render to None -'{"model":"{{model}}","prompt":{{json.dumps(prompt,ensure_ascii=False)}}}' - -# CORRECT — plain string, hardcoded service model ID -'{"model":"hy-image-v3.0","prompt":"{{prompt}}"}' -``` - -**Model ID sourcing**: The model ID sent to the upstream must be the provider's actual service ID. Discover actual IDs by querying the provider's `/v3/models` endpoint (ARK) or `/v1/models` endpoint (tokenhub), then hardcode them in the uapi data template. - -**Pitfall: `return` inside `async with` → silent NoneType in .dspy files** - -The ahserver framework wraps .dspy code in `async def myfunc(request, **ns):` and awaits it. Returning from inside an `async with db.sqlorContext(...) as sor:` block causes the function to return `None`, producing `return data type error, `. - -**Fix**: Collect all results inside the `async with` block, then `return` AFTER the block exits: - -```python -# WRONG — return inside async with → NoneType -async with db.sqlorContext(dbname) as sor: - recs = await sor.R('table') - return json.dumps({'total': len(recs), 'rows': rows}) - -# CORRECT — collect inside, return outside -rows = [] -async with db.sqlorContext(dbname) as sor: - recs = await sor.R('table') - for r in (recs or []): - rows.append({'id': r.id, 'name': r.name}) -return {'total': len(rows), 'rows': rows} -``` - -**Symptom**: `Exception: /path/file.dspy return data type error, ` with no other traceback. Minimal test: `return {'ok': True}` inside async with also fails. - -**Pitfall: `sor.C()` silently drops records when `created_at` is missing** - -sqlor's `sor.C()` does NOT auto-set timestamp fields. If the model defines `created_at TIMESTAMP NOT NULL`, `sor.C` may succeed silently but the record never reaches the database. Always include `'created_at': curDateString()`: - -```python -ns = dict(params_kw) -ns['id'] = params_kw.get('id', getID()) -ns['created_at'] = curDateString() # REQUIRED — sor.C does not auto-set -ns['org_id'] = (await get_userorgid()) or '0' -async with db.sqlorContext(dbname) as sor: - await sor.C('table_name', ns) -``` - -**Symptom**: dspy returns `{"success": true}` but `SELECT` returns zero rows. - -**Pitfall: Bash heredoc `\$` escapes when writing dspy files via terminal** - -When using `cat > file.dspy << 'EOF'` in bash, backslash sequences like `\${pid}\$` intended as sqlor placeholders get written literally as `\${pid}\$` instead of `${pid}$`. This causes sqlor to fail silently because the placeholder pattern is unrecognized. - -**Fix**: Never escape `$` in heredocs for dspy files. Write `${pid}$` as-is: - -```bash -# WRONG — heredoc writes literal backslash -cat > file.dspy << 'EOF' -sql = 'WHERE id=\${pid}\$' -EOF - -# CORRECT — no escaping needed in single-quoted heredoc -cat > file.dspy << 'EOF' -sql = 'WHERE id=${pid}$' -EOF -``` - -**Verification**: `grep 'project_id' file.dspy | cat -A` — look for `\$` (shows as `\$`) which means the backslash was written literally. Correct output shows `${pid}$` with no backslash. - -**Pitfall: Do NOT name module functions `get_module_dbname`** - -In `init.py`, do NOT define a function named `get_module_dbname()` — this name is already provided as a global by ServerEnv and is used by .dspy scripts. Registering it via `env.get_module_dbname = ...` will overwrite the global. If you need a module-specific dbname helper, name it with a private prefix like `_get_dbname()`. - -**Pitfall: Do NOT name module functions `get_module_dbname`** - -**Key Points:** -- All functions that need to be accessible from .ui/.dspy files must be registered with ServerEnv -- The `load_{modulename}()` function is called by Sage during module loading -- After registration, functions can be called directly in .ui templates: `{% set data = get_data() %}` -- This pattern is used by all standard modules (rbac, appbase, etc.) +- Purpose: register ALL module functions with ServerEnv so .ui/.dspy can call them directly. NOT for route registration (wwwroot files auto-routed via `/{module_name}/filename.ext`). +- **CRITICAL: export functions in `__init__.py`** — all async functions defined in init.py MUST be imported in the package `__init__.py`; otherwise dspy calls fail `NameError: name 'xxx' is not defined`. +- **Pitfall: Triple-place function registration** — adding/removing a function requires updating THREE files in sync: ① implementation (`mymodule/mymodule.py`); ② `__init__.py` import line; ③ `init.py` `env.xxx = xxx` in `load_{module}()`. Missing ② → ImportError/AttributeError at init; missing ③ → NameError in .dspy/.ui; removed from ① only → ImportError at load. Cleanup check: `grep -rn 'function_name' mymodule/ --include='*.py'` must hit exactly those three places. +- **Pitfall: CRUD dspy wrappers use PLURAL table names** — xls2ui generates wrappers (`add_suppliers.dspy`) calling `create_suppliers/update_suppliers/delete_suppliers`, while init.py usually defines singular names (`create_supplier`). Register BOTH: `env.create_supplier = create_supplier; env.create_suppliers = create_supplier` (same for update/delete). Symptom: CRUD "add" returns 500 `function not found`. +- **Pitfall: Subagent delegation for module creation — ALWAYS validate after** (subagents consistently err even with correct specs; observed 2/3 failure rate). Consistent errors: ① init.py/__init__.py misplaced at module ROOT instead of package dir; ② model JSON `{"table":...,"fields":{...}}` instead of `{"summary":[...],"fields":[...],"indexes":[...],"codes":[...]}`; ③ CRUD JSON `{"table":...,"list":{...}}` instead of `{"tblname":...,"params":{"browserfields":{...},"editable":{...}}}`; ④ hallucinated sqlor APIs (`sqlor.save/list/one/delete/insert/query` DO NOT EXIST — only `sor.C/U/D/R/I/sqlExe`). Mandatory validation: move misplaced files to package dir; check model JSONs have `summary` (array primary); check CRUD JSONs have `tblname` + `params.editable`; grep for fake sqlor APIs → fix to sor.C/U/D; dspy audit (no imports/print/uuid); py_compile all .py. See `references/bulk-module-creation-pattern.md` "Subagent Validation Checklist". +- **Pitfall: `debug()` in .dspy has NO filename** — logs show `[sage][debug][:N]` (framework injects dspy code into a `` context). Always prefix manually: `debug(f'product_category_create.dspy: START params_kw={dict(params_kw)}')` — essential when several dspy run in sequence (add → refresh). +- **Pitfall: DSPY API handlers must forward ALL client params — never hardcode dispatch fields** — e.g. read `task_type = params_kw.get('task_type', 'separate')` and forward `output_dir` if present; do NOT build `{'task_type': 'separate', ...}`. Symptom: client sends `separate_full` but worker log shows `separate`; hours to trace because the worker is correct — the DSPY silently drops the param. Audit: grep API dspy for hardcoded `'task_type'`/`'mode'` dispatch strings inside payload construction. +- **Pitfall: py_compile is INVALID for .dspy files** — dspy is injected into an async function at runtime (top-level `return`/`await` are correct); py_compile gives false positives (`'return' outside function`). Use py_compile only for .py; validate .dspy with the dspy audit script (grep-based). +- **CRITICAL: dspy MUST use explicit `return`** — ahserver wraps code in `async def myfunc(request, **ns):` and awaits it; a bare `result` expression returns None. See `references/dspy-execution-and-module-structure.md` (execution model, pre-loaded globals, Tree widget data format, module symlink patterns). +- **Pitfall: Sage startup requires environment variables** (e.g. `ALIPAY_PUB`) read at import time in `app/sage.py` — cannot full-start locally (FileNotFoundError). Validate with py_compile (.py) + dspy audit instead of startup. +- **Pitfall: Raw SQL column names — ALWAYS verify against models/{table}.json, never guess.** Check first: `python3 -c "import json; cols=[f['name'] for f in json.load(open('models/table.json'))['fields']]; print(cols)"`. Known wrong guesses → correct: permission `permcode/permname` → `path/name`; users `created_date` → `created_at`; organization `org_name` → `orgname`. +- **Pitfall: `{{id}}` in uapi response templates resolves to Python built-in `id()`** — renders `` instead of the field. NEVER use `{{id}}`; map to a different name: `'{"taskid":"{{taskid}}","status":"{{status}}"}'`. Affects any uapi whose upstream response has an `id` field. +- **Pitfall: uapiio `input_fields` MUST be a JSON array** — bricks.js LlmIO calls `this.input_fields.forEach(...)`; an object `{"field":{...}}` throws `TypeError: this.input_fields.forEach is not a function`. Use array form: `[{"name":"prompt","label":"用户输入","uitype":"text","required":true}]`. Canonical reference: existing `ktv_asr_transcribe_io` record. +- **Pitfall: `json.dumps()` in uapi data templates → `data=None`** — `{{json.dumps(prompt, ensure_ascii=False)}}` can render silently to None (logs `body=None`). Use plain string interpolation `"{{prompt}}"` and HARDCODE the upstream model ID (`"model":"hy-image-v3.0"`), don't pass `{{model}}`. Source real model IDs from provider `/v3/models` (ARK) or `/v1/models` (tokenhub). +- **Pitfall: `return` inside `async with` → silent NoneType** — returning inside `async with db.sqlorContext(...) as sor:` returns None → `return data type error, `. Collect results inside the block, `return` AFTER it exits. Minimal repro: `return {'ok': True}` inside async with also fails. +- **Pitfall: `sor.C()` silently drops records when `created_at` missing** — sqlor does NOT auto-set timestamps; with `created_at TIMESTAMP NOT NULL` the insert silently vanishes. Always set `ns['created_at'] = curDateString()` (and `ns['org_id'] = (await get_userorgid()) or '0'`). Symptom: dspy returns `{"success": true}` but SELECT returns zero rows. +- **Pitfall: Bash heredoc `\$` escapes corrupt sqlor placeholders** — `cat > f.dspy << 'EOF'` with `\${pid}\$` writes the backslash literally; sqlor fails silently on the unrecognized placeholder. Never escape `$` in single-quoted heredocs — write `${pid}$` as-is. Verify: `grep 'project_id' file.dspy | cat -A` must show no `\$`. +- **Pitfall: Do NOT name module functions `get_module_dbname`** — that name is already a ServerEnv-provided global used by .dspy; registering it overwrites the global. Use a private prefix (`_get_dbname()`). ### 2. Frontend Development -- **Mandatory**: Use bricks-framework for all frontend interfaces -- Store all .ui files as **pure JSON format** (NOT HTML/CSS) in wwwroot/ -- See `references/bricks-ui-pitfalls.md` for id placement (widget level, not options), Button click event, Popup/Form patterns, and script actiontype Jinja2 limitations. -- **Automatic routing**: Files in wwwroot/ are automatically accessible via `/{module_name}/filename.ext` -- **No manual route registration needed**: The framework automatically serves .ui and .dspy files -- .ui files contain component definitions with widgettype, options, subwidgets, and binds properties -- Store CSS files in wwwroot/ for styling extensions -- Store JavaScript files in wwwroot/ for custom functionality and registerFunction extensions -- **ahserver auto-serves .css and .js files**: files placed in wwwroot/ are automatically discovered and injected by ahserver into the HTML response — **DO NOT** manually add `` or `