pipeline_core/skills_library/all/dspy-file-implementation-spec/SKILL.md

name	description	author	tags
dspy-file-implementation-spec	实现 .dspy 接口（wwwroot/api/*.dspy）时必读——exec-context 规则、返回格式、模块集成。不加载会产出不规范的 dspy（返回格式错、环境未注册）。	Hermes Agent	ahserver dspy backend web-development python

.dspy File Implementation Specification

Overview

.dspy files are controlled Python scripts executed by the ahserver web framework to provide dynamic API endpoints. They must follow strict conventions for security, performance, and framework compatibility.

Core Rules

1. No Import Statements

Never use import in .dspy files. The framework auto-provides module functions via load_{modulename}() and pre-loads common modules (datetime as the full module, json, os, sys) into the global context.

• ⚠️ from datetime import date WILL fail with an import error — use datetime.date.today().
• ✅ today = datetime.date.today().isoformat(); json.dumps({'k':'v'}); module functions like get_all_records() used bare.

2. Return, Not Print

Always return data to the client; ahserver handles JSON serialization. print() writes to stdout that ahserver ignores → return data type error, <class 'NoneType'>.

• ✅ return records (never print(json.dumps(result))).

3. ID Generation: uuid() in .dspy/.ui, getID() in .py

Both uuid() and getID() work in .dspy context (verified: llmage dspy files use getID() without import). uuid() returns shorter IDs, getID() returns 22-char IDs. In .py files (init.py, utils.py) you must from appPublic.uniqueID import getID.

4. Error Handling

• Array-returning endpoints (code components): try: ... return result except Exception: return [].
• Object-returning endpoints: try: ... return record except Exception: return {"error": str(e)}.

5. Input Validation & Safety

• Validate/sanitize params_kw inputs, e.g. if not record_id or not str(record_id).isdigit(): return {"error": "Invalid ID parameter"}.
• Never return sensitive fields (passwords, API keys) unless required and authorized.
• For expensive operations in production, add rate limiting.

Pre-Loaded Server-Env Functions (.dspy context — NO import needed)

Function	Description
password_encode(s) / password_decode(s)	Hash / decode a password using the app's configured key
remember_user(userid, username, userorgid)	Set session user (login)
forget_user()	Clear session user (logout)
get_user() / get_username() / get_userorgid() / get_userinfo()	Current user id / display name / org id / full user object
get_session() / session_getvalue(key) / session_setvalue(key, value)	Session access
get_module_dbname(modulename)	Get DB name for a module
get_sor_context(env, modulename)	Async context manager for cross-module DB access
DBPools()	Database connection pool (current module's DB)
params_kw	Dict of request params — query string + POST body (incl. application/json) merged; nested JSON preserved as dict/list. The ONLY way to access request data — there is NO http_request variable.
request	The ahserver Request object (auto-injected)
json, datetime, time, os	Pre-loaded modules (os MAY be available — verify if needed)
uuid / getID	ID generation (both work)
DictObject	appPublic.dictObject — available directly
partial	functools.partial — available directly
FileStorage	ahserver.filestorage — available directly
curDateString / timestampstr	appPublic.timeUtils date/time string helpers
get_config_value(key)	Get config value
exception, error, debug, info, warning, critical	Logging functions — all available
format_exc	traceback.format_exc() — full traceback string (pre-loaded, do NOT import traceback)

Verified (llmage cleanup 2026-07-01): all 31 dspy files worked after removing imports of: json, datetime, getID (appPublic.uniqueID), debug (appPublic.log), curDateString/timestampstr (appPublic.timeUtils), get_sor_context (sqlor.dbpools), time, DictObject (appPublic.dictObject), partial (functools), FileStorage (ahserver.filestorage), os.

Endpoint Patterns

Code Component Data Endpoints

File: /wwwroot/entity_name/list/index.dspy. Must return an array of {value, text} (value as string) — [] on error/no data.

try:
    result = [{"value": str(r.get('id')), "text": r.get('name', f"Record {r.get('id')}")} for r in get_all_records()]
    return result
except Exception:
    return []

Single Record / Action Endpoints

• /wwwroot/entity_name/get/index.dspy: record_id = params_kw.get('id'); missing id → {"error": "ID parameter required"}; else record or {"error": str(e)}.
• /wwwroot/entity_name/test/index.dspy: return {"status": "success"/"error", "message": ...}.

Login Endpoint

File: /wwwroot/login.dspy. Key points:

• passwd = password_encode(password) before comparison; rzt = await check_user_password(request, username, passwd) (RBAC auth).
• On success: dbname = get_module_dbname('rbac'), async with DBPools().sqlorContext(dbname) as sor: → await sor.sqlExe("SELECT id, username, name, orgid FROM users WHERE username=${username}$", {'username': username}); then await remember_user(user.id, user.username, getattr(user, 'orgid', '') or '').
• Use remember_user() to create session — NOT user_login() (requires explicit import, fails in .dspy).
• Always return a string via json.dumps(..., ensure_ascii=False), never None.

CRUD List API Pattern (sqlor-based)

Return format: {'success': bool, 'rows': [...], 'total': int} via json.dumps(result, ensure_ascii=False, default=str).

• Use params_kw.get() for pagination (page, rows, sort), ${param}$ syntax everywhere.
• Do separate count + data queries; ns = {'page': int(...), 'rows': int(...), 'sort': ...}.
• Merge ns dicts with query_ns = dict(list(ns.items()) + list(where_ns.items())) — {**ns, **sql_ns} FAILS.
• sqlExe with page/rows in ns returns {'total': N, 'rows': [...]} dict; without them returns a list of row objects.
• Convert rows: [dict(r) if hasattr(r, 'keys') else r for r in rows].
• CRITICAL: all SELECT columns must exactly match the actual DB schema — verify with DESCRIBE table_name before writing queries (DDL files may differ from deployed schema).

DataViewer CRUD Endpoint

Create/update/delete endpoints for DataViewer editable forms must return Message widget JSON as a STRING, not raw data:

result = {'widgettype': 'Message', 'options': {'title': 'Error', 'message': 'Invalid request'}}
try:
    ...
    result = {'widgettype': 'Message', 'options': {'title': 'Success', 'message': 'Created successfully', 'type': 'success'}}
except Exception as e:
    result['options'] = {'title': 'Error', 'message': f'Failed: {str(e)}', 'type': 'error'}
return json.dumps(result, ensure_ascii=False)

CRITICAL: the return value MUST be a string (json.dumps()); any path ending without return → return data type error, <class 'NoneType'>.

DataViewer Editable Config in .ui

Configure in the DataViewer's options.editable block:

{
  "widgettype": "DataViewer",
  "options": {
    "data_url": "/main/module/api/list.dspy",
    "editable": {
      "new_data_url": "/main/module/api/create.dspy",
      "update_data_url": "/main/module/api/update.dspy",
      "delete_data_url": "/main/module/api/delete.dspy",
      "form_cheight": 8,
      "fields": [{"name": "field_name", "label": "Label", "uitype": "text", "required": true}]
    }
  }
}

new_data_url/update_data_url = form submission URLs; delete_data_url = POST URL sending {params: row_data}.

Cross-Module Database Access — get_sor_context is the ONLY pattern

• REQUIRED: env = request._run_ns; async with get_sor_context(env, "module_name") as sor: then await sor.R('table', {'filter': 'value'}). It resolves module → DB via the module_dbname config (Sage: module "tenant" → sage DB; pipeline-app: → pipeline DB), configured per deployment.
• ❌ NEVER hardcode DB names (async with db.sqlorContext("pipeline") breaks cross-deployment portability — the most common cross-module DSPY error).
• ❌ NEVER use DBPools() + sqlorContext() for cross-module access (that's for the current module's DB only).
• ❌ NEVER use ServerEnv() in .dspy — all server-env functions are injected as globals.
• If a cross-module function is registered via load_{modulename}() (e.g. create_user_apikey from dapi), call it directly: create_user_apikey(sor, dappid, user_id, user_orgid).

Batch Lookups with $or

users = await sor.R('users', {'$or': [{'id': uid} for uid in user_ids]}). Validate the ID list is non-empty first; chunk lists >100 items to avoid query complexity limits.

Safe Attribute Access on SQLor Rows

SQLor returns row objects that may or may not support dict access. Use getattr(user, 'orgid', '') or '' (handles missing attribute AND None). For dict access use dict(r) if hasattr(r, 'keys') else r.

DSPY Code Review Checklist

Syntax & Security

• No imports — module DSPY files must have zero import statements. All needed names (json, datetime, get_sor_context, DBPools, params_kw, request, uuid, time, os, DictObject, FileStorage, logging functions) are pre-loaded.
• No forbidden patterns — no eval(), exec(), __import__(), os.system(), subprocess, pickle.loads().
• Valid Python AST — bare ast.parse() rejects top-level await/async with ("await outside async function"); wrap first: wrapped = 'async def __c__(params_kw, request, uid, org_id, json, DBPools, get_user, get_userorgid, get_module_dbname, getID, debug, sor, params_kw=None):\n' + '\n'.join(' ' + l if l.strip() else l for l in src.split('\n')) then ast.parse(wrapped) (add injected names the file uses to the wrapper signature). Mandatory after patching triple-quoted prompt constants — a stray """ silently closes the string and dumps following prose as code; only ast.parse exposes it (caught live 2026-08 in cockpit_chat.dspy).
• All branches return — missing return → return data type error, <class 'NoneType'>.

SQL & Database

• Parameterized queries — ${param}$ syntax, never f-string interpolation or %s in SQL strings.
• Decimal / SUM aggregate safety — MySQL SUM() returns Decimal; wrap with int() (e.g. int(r.total_size or 0)) or pass default=str in json.dumps().
• Cross-module access — get_sor_context(env, 'module'), not DBPools().sqlorContext(dbname).
• sqlExe return type awareness — no page/rows in ns → list of row objects (r.field attrs); with page/rows → {'total': N, 'rows': [...]} dict.
• Error handling — at least try/except around DB ops with a fallback return.

Code Quality (KISS/DRY)

• Sibling file consistency — inconsistent return format (raw dict vs json.dumps()), divergent helper signatures, or different API patterns across .dspy files in one directory are red flags.
• DRY — no duplicated helpers — size formatters (fmt_size, fmt), date formatters, SQL builders duplicated across files → extract.
• No hardcoded config values — storage limits, API URLs, timeouts come from config.
• f-string safety — avoid f-strings in dict returns (see Pitfall: f-string braces); use 'prefix: ' + str(var).
• No print() — use return (see Core Rule 2).

Return Format

• Consistent return style per directory — either raw dict return {...} or json.dumps({...}), same everywhere.
• DataViewer CRUD endpoints — return Message widget JSON, not raw data.
• Code component endpoints — return [{value, text}] array.
• JSON validity — Decimal, datetime, bytes types break json.dumps without default=str.

Testing and Validation

• Test endpoints directly: http://localhost:8000/app-name/entity_name/list/.
• Format expectations: code components → array of {value, text}; DataViewer → array of full record objects; Forms → single record or success/error object.
• Test error scenarios: missing params, invalid IDs, DB failures.

Integration with Bricks Framework (.ui)

• Reference endpoints: {"uitype": "code", "data_url": "/app-name/entity_name/list/"}.
• Pass params via query string: {"data_url": "/app-name/entity_name/get/?id={{selectedRow.id}}"}.

Common Pitfalls (deduped, all known)

1. print() instead of return — stdout is ignored → return data type error, <class 'NoneType'>. (Real case: top_models.dspy fixed by return json.dumps(models, ensure_ascii=False, default=str).)
2. Import statements — violates the ahserver security model; everything in the Server-Env table is pre-loaded. If a module function is needed, export it via load_{modulename}() in init.py.
3. Jinja2 .ui files cannot execute Python — they render JSON only; no DB queries/async/complex logic. (llmusage_ioinfo_display.ui used {% set sor = db.sqlorContext() %} → NameError: name 'db' is not defined; convert to .dspy.)
4. sqlPaging() is slow for large tables (3-4s; wraps SQL in select count(*) from (...)) — separate count + data queries instead.
5. Extract reusable DB ops to async functions in module utils.py — async def get_record_by_id(...) using get_sor_context; call from .dspy via from module.utils import get_record_by_id (note: see Pitfall 24 re import rules — module-utils imports are used in some modules, ragserver forbade them).
6. FileStorage requires realPath() for file I/O — fs.realPath(webpath) converts webpath → filesystem path before aiofiles.open(...).
7. Implicit None return — EVERY branch must end with return result; a bare expression is local to the wrapped async function. Multi-branch pattern: set result = {...} in each if/elif/else, then a single return result at module level, OUTSIDE all branches. Even result = {"text": "hello"} returns None without explicit return result.
8. Debug NoneType errors at the error location — open the .dspy FIRST; do NOT start by adding broad try/except wrappers in Python functions or changing DB/SQL. except Exception: return [] masks real errors. Check file format (JSON {"python": {...}} vs Python script) and return paths.
9. JSON-format vs Python-script-format DSPY — JSON format ({"python": {"import": "...", "call": "..."}}) handles None returns differently than script format. If one DSPY in a module uses JSON format while others use script format, it's likely the bug. This is the single most common dspy error: result set in branches but final return result forgotten.
10. Wrong data format — code components need {value, text} arrays; DataViewer needs full record arrays.
11. Missing error handling — causes 500s instead of graceful degradation.
12. Returning wrapper objects unnecessarily — most components expect direct data.
13. SQL column mismatch with DDL — SELECT columns MUST exactly match the deployed schema; verify with DESCRIBE table_name.
14. CGI-style .dspy forbidden — never os.environ, sys.stdin, os.read(0, ...), print(), asyncio.new_event_loop(). ahserver auto-parses ALL request data (query string + POST body, incl. JSON application/json) into params_kw; JSON bodies preserved as nested dict/list.
15. DataViewer CRUD returning raw JSON — must return Message widget JSON.
16. Dict merge {**a, **b} fails — use dict(list(a.items()) + list(b.items())).
17. sqlExe return type depends on ns — WITH page/rows → dict {'total': N, 'rows': [...]} (don't iterate as list); WITHOUT → list of row objects (don't do ret['key']).
18. Row objects need safe conversion — dict(r) if hasattr(r, 'keys') else r.
19. Sort column must exist in table — default 'id' may not always be available.
20. API file location — list API .dspy files go in wwwroot/api/ (e.g. wwwroot/api/customers_list.dspy); .ui files go directly in wwwroot/.
21. Session expiration during testing — cookie sessions expire after session_max_time (default 3600s); re-login via /main/login.dspy?username=xxx&password=xxx on 401.
22. Connection pool dirty reads (multiserver) — aiomysql.connect() defaults to autocommit=False; sqlorContext only commits writes, so a reused pooled connection can keep a stale REPEATABLE READ snapshot → "get reads deleted record". Fix: in mysqlor.enter(), await self.conn.commit() to end lingering transactions before reuse. (Symptom: high-frequency stale reads in multi-instance deployments.)
23. Filter NaN/null/empty before MySQL INSERT/UPDATE — bricks UiFloat widgets via urlwidget + datawidget: "self" may send NaN, null, or '' → MySQL OperationalError: nan can not be used with MySQL. Sanitize ALL numeric params before float() or sor.C/U: s = str(v).strip().lower(); if s in ('', 'nan', 'none', 'null'): v = None.
24. SQL parameter syntax — use ${param}$, NOT %(param)s ("format requires a mapping" errors).
25. Optional DATE fields — MySQL DATE columns reject ''; convert empty form values to None: params_kw.get('sign_date', '').strip() or None.
26. Safe row attribute access — getattr(row, 'field', '') or '', not row.field/row['field'].
27. CRITICAL: ServerEnv() forbidden in .dspy AND in Python helpers — all functions are injected globals; env = ServerEnv() / getattr(env, 'func_name', None) / getConfig(); db.databases = config.databases are all wrong. For Python functions in init.py called from dspy: env = request._run_ns (bare ServerEnv() has no request binding — get_user(), get_userorgid() return None → 500 'NoneType' object is not callable). Correct: user_id = await env.get_user().
28. Bare function calls from load_X() registrations can be None in dspy — ServerEnv-singleton merge into the exec namespace can fail silently → 'NoneType' object is not callable. VERIFIED DECISION (ragserver, 2026-07-29): (a) env.func = func in init → request._run_ns.func always None; (b) from rag.pipeline import func → blocked (imports not allowed); (c) inline all logic directly in the DSPY is the ONLY reliable approach — use only pre-loaded globals; import installed venv packages (PyPDF2, docx, pptx, openpyxl, aiohttp, base64) inline at point of use (venv packages, not custom modules). DSPY = self-contained, zero custom imports. (Contrast: discount module verified from discount.init import bind_customer, set_promote_discount + await bind_customer(request, bind_params) works.) Diagnosis: minimal test dspy result = {'text': str(type(bind_customer))} — <class 'NoneType'> means the name isn't in the namespace.
29. CRITICAL: f-string braces inside dict returns cause exec() parse error — exec() misreads } as closing the outer dict → SyntaxError: '{' was never closed. ❌ return {"timeout": 5, "message": f"处理失败: {e}"} → ✅ return {"timeout": 5, "message": "处理失败: " + str(e)}. Also affects exception(f'{var=}') — use concatenation for debug/exception calls too.
30. sqlExe rows are SimpleNamespace-like — attribute access only (r.id); dict access (r['id']) raises TypeError: 'SimpleNamespace' object is not subscriptable, often SILENTLY swallowed by try/except → empty dropdowns / "undefined" in UI. Use getattr(r, 'id', '').

Complex Logic: Move to Python, DSPY as Thin Wrapper

When a .dspy needs module-internal classes/functions not registered on ServerEnv (e.g. EmailClient, PROVIDERS), the DSPY hits NameError. Never add imports to the DSPY. Instead: (1) add logic as a provider-class method (e.g. TransferGateway.check_transfer()), (2) register on ServerEnv (env.PROVIDERS = PROVIDERS), (3) DSPY becomes a thin wrapper:

provider = env.PROVIDERS.get('transfer')
title, msg = await provider.check_transfer(tcode, env)
return {"widgettype": "Message", "options": {"title": title, "message": msg}}

This also avoids the f-string brace issue (Pitfall 29) — Python methods can use f-strings freely; only the DSPY wrapper uses concatenation.

add_startup Blocks Server — Use Manually Triggered Actions

add_startup(coro) awaits the coroutine during startup; an infinite while True loop blocks the server indefinitely. Never use it with infinite loops/long polling. Trigger manually (button → DSPY endpoint) or spawn non-blocking tasks via asyncio.create_task() inside the startup callback.

How DSPY Execution Works (ahserver wraps in async function)

ahserver wraps the code in async def myfunc(request, **ns): then exec(txt, lenv, lenv), return await func(request, **lenv) (baseProcessor.py ~line 234). Consequences:

• async with, await, async for DO work inside .dspy.
• You MUST use explicit return — the function's return value is what reaches the caller.
• A bare expression (e.g. result on the last line) inside an async with block is local to the function → returns None. ❌ bare result → ✅ return [...] inside the block, return [] after it.

Async/Await — Fully Supported in DSPY

VERIFIED (2026-07-29, ragserver): async with, await, async for all work (the earlier prohibition was incorrect). E.g. async with get_sor_context(env, 'rag') as sor: / file_data = await request.read() / async with aiohttp.ClientSession() as s: r = await s.post(...).

• Symptom 500 'NoneType' object is not callable is almost always a ServerEnv registration failure (Pitfall 28), NOT an async/sync problem.
• PITFALL: params_kw unavailable in some DSPY contexts (standalone page endpoints like /discount/promote.dspy) — use request._run_ns.params_kw.get('code', '') instead.
• PITFALL: binds with actiontype: "script" causes 500 in DSPY output — the server-side JSON parser may evaluate the script string as Python. Keep binds in static .ui templates, never in DSPY widget JSON.
• Async functions can be exported via load_discount() (env.generate_promo_qr = generate_promo_qr) and awaited from the DSPY (await func_name(request, params_kw)) — DSPY stays a thin 2-line wrapper. (Note: this contradicts Pitfall 28's ragserver finding — ServerEnv registration propagation varies by module; verify with the type() diagnosis.)

CRUD Wrapper Pattern (Legitimate Exception)

When init.py registers CRUD functions via load_{module}() (env.create_tablename = create_tablename), the wwwroot/api/{table}_create.dspy / {table}_update.dspy / {table}_delete.dspy files become thin wrappers that delegate to those functions. These wrappers may use ServerEnv() and print() — a legitimate exception to the no-ServerEnv rule. CRITICAL: json is pre-loaded in ALL dspy contexts — do NOT import json (redundant, causes pre-commit audit failures). The only allowed import is from ahserver.serverenv import ServerEnv:

from ahserver.serverenv import ServerEnv
env = ServerEnv()
create_func = getattr(env, 'create_tablename', None)
if create_func is None:
    print(json.dumps({"status": "error", "message": "create_tablename function not found"}))
else:
    result = await create_func(request, params_kw)
    print(result)

Applies ONLY to wwwroot/api/{table}_create|update|delete.dspy wrappers delegating to init.py-registered CRUD functions. Business-logic .dspy files (queries, calculations, cross-module ops) must follow the standard pattern (no imports, no ServerEnv, use return).

Module Deployment Workflow

CRITICAL: never edit code directly on test/production servers.

1. Edit in local repo (~/repos/<module>/) → 2. git add + git commit + git push → 3. On test server: git pull in the module dir → 4. If the server has no SSH key for git, scp changed files individually.

Module directory structure (Sage — modules live under Sage's pkgs/, NOT pipeline-app's):

/d/apitest/sage/
  pkgs/module_name/          ← git repo (code)
    wwwroot/                 ← symlinked from ../../wwwroot/module_name
    module_name/             ← Python package (copied to site-packages)
  wwwroot/module_name -> ../pkgs/module_name/wwwroot
  py3/lib/python3.10/site-packages/module_name/

Python code is copied to site-packages/ for the Sage venv; wwwroot/ is symlinked from Sage's main wwwroot/.

Schema inspection without direct DB access: create a temp debug .dspy at wwwroot/api/debug_tables.dspy querying information_schema.COLUMNS (SELECT COLUMN_NAME, COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='dbname' AND TABLE_NAME='table_name' with ns = {'page': 1, 'rows': 50, 'sort': 'COLUMN_NAME'}), test via curl, delete after use.

Best Practices Summary

• ✅ Use return for all data responses; never import; handle exceptions gracefully.
• ✅ Return component-appropriate formats; validate all inputs.
• ✅ Keep .dspy files focused/minimal; only create them when standard CRUD endpoints are insufficient.
• ✅ Consistent naming (/list/, /get/, /test/, etc.).
• ✅ getattr(row, 'field', '') or '' for SQLor rows; $or in sor.R for batch ID lookups.
• ✅ Bare get_module_dbname('module') for cross-module DB access (no ServerEnv() wrapper).

Linked References

• references/user-sync-pattern.md — Cross-module user sync API pattern
• references/dirty-apikey-record-pattern.md — Orphan downapikey records
• references/accounting-table-architecture.md — Accounting table schema
• references/sage-deploy-test-server.md — Sage module deployment workflow
• references/pipeline-app-setup.md — Pipeline-app config, debugging, and KTV setup
• references/sage-crontab-etl-pattern.md — Cron DSPY endpoints + build.sh crontab + j2_ stat cards
• references/cross-table-column-pitfalls.md — Column name mismatches across Sage tables (userorgid vs orgid) + catelogid length + GROUP BY ambiguity
• sqlor-database-module skill references/dapi-table-architecture.md — Full dapi module table structure