---
name: crud-definition-spec
version: 1.0.0
description: 定义 CRUD（json/*.json，根键 tblname+params）时必读——CRUD 定义格式、list/tree 两种视图、同表多逻辑才加 alias（目录=tblname，单逻辑不加）。不加载会产出不符合规范的 CRUD json、alias 滥用。表结构定义用 database-table-definition-spec。
trigger_conditions:
  - User needs to create or modify CRUD definition files in JSON format
  - Task involves generating CRUD configurations for the json directory
  - Working with sqlor-database-module CRUD operations
  - Need to determine between list vs tree view based on table relationships
---

# CRUD Definition Specification

## Overview
Standardized JSON format for CRUD operations integrating bricks-framework (frontend) and sqlor-database-module (backend). The framework auto-selects list vs tree view from table structure. Files live in the module's `json/` directory and are generated into `wwwroot/<table>/` UIs + dspy endpoints via `xls2ddl.xls2crud`.

## View Type Determination
- **Tree View**: table has a self-referencing foreign key (one field points to another record's id in the same table)
- **List View**: all other tables (no hierarchical relationships)

## Common Root Properties (Both View Types)
```json
{
    "tblname": "table_name",        // Required: actual table name
    "alias": "optional_alias",      // Optional: multiple CRUD interfaces for same table
    "title": "Display Title",       // Optional: defaults to table title
    "params": { ... }               // Required: view-specific parameters
}
```

## List View CRUD Specification
### Complete Structure
```json
{
    "tblname": "table_name",
    "alias": "optional_alias",
    "title": "Display Title",
    "params": {
        "sortby": ["field1 desc", "field2"],
        "logined_userorgid": "org_id_field",
        "logined_userid": "user_id_field",
        "data_filter": {
            "AND": [
                {"field": "field1", "op": "LIKE", "var": "filter_var1"},
                {"field": "field2", "op": "=", "var": "filter_var2"}
            ]
        },
        "confidential_fields": ["field1", "field2"],
        "editor": {
            "binds": [{
                "wid": "source_field_id",     // Required: source widget ID (field name)
                "event": "changed",           // Required: event type (typically "changed")
                "actiontype": "script",       // Required: action type ("script" for JS)
                "target": "target_field_id",  // Required: target widget ID
                "script": "// JS code"        // Required: JS script content
            }]
        },
        "browserfields": {
            "exclouded": ["id"],              // Optional: fields excluded from display
            "alters": {
                "field_name": {
                    "uitype": "code",         // Required: "code" = dropdown/select
                    "data": [{"value": "v1", "text": "Display Text"}],  // Required for uitype="code" (static)
                    // OR dynamic: "dataurl": "api/endpoint", "datamethod": "GET", "dataparams": {"param": "value"}
                }
            }
        },
        "editexclouded": ["readonly_field"],  // Optional: fields excluded from edit form
        "subtables": [{                       // Optional: foreign key relationships
            "field": "foreign_key_field",     // Required: FK field name
            "title": "Subtable Title",        // Optional: defaults to subtable title
            "url": "{{entire_url(subtable_alias)}}",  // Required when alias defined
            "subtable": "related_table_name"  // Required: related table name
        }]
    }
}
```

## Tree View CRUD Specification
### Complete Structure
```json
{
    "tblname": "hierarchical_table",
    "alias": "optional_alias",
    "uitype": "tree",                          // Required: must be "tree" for tree view
    "title": "Display Title",
    "params": {
        "idField": "id",                       // Required: node ID field
        "textField": "display_field",          // Required: node display text field
        "sortby": ["field1 desc", "field2"],   // Optional: sort fields for tree nodes
        "confidential_fields": ["field1", "field2"],  // Optional: sensitive fields
        "browserfields": {"alters": {}},       // Optional: field attribute modifications
        "logined_userorgid": "org_id_field",   // Optional: org filtering field
        "logined_userid": "user_id_field",     // Optional: user filtering field
        "editable": true,                      // Required: true=editable, false=read-only
        "edit_exclouded_fields": ["system_field"],  // Optional: excluded from editing
        "parentField": "parent_id",            // Required: parent node reference field
        "subtables": [...]                     // Optional: same structure as list view
    }
}
```

## Filter/Search Integration (data_filter)
`data_filter` in `params` follows the `sqlor/filter.py` DBFilter JSON format:
```json
{"tblname": "llm", "params": {"data_filter": {"AND": [
    {"field": "model", "op": "LIKE", "var": "model_input"},
    {"field": "ppid", "op": "=", "var": "ppid_input"},
    {"field": "status", "op": "=", "const": "1"}
]}}}
```
**Frontend behavior**: With `data_filter` present, DataViewer adds a "搜索" toolbar button → PopupWindow with a Form. Fields are generated from the filter tree: each `var` becomes a form input; fields with `browserfields.alters[field].uitype == "code"` render as dropdowns. Customize via `filter_labels: {"var_name": "显示名"}` (labels), `filter_label` (button text), `filter_title` (popup title). **No inline search form is rendered** — filter UI is exclusively toolbar-triggered. Form submit sends `data_filter` (JSON string) + each `var` value as URL params → backend `.dspy` uses `DBFilter.gen(ns)` for the SQL WHERE clause.

**Key rules:**
- `var`: parameter name — value from user input in the popup form
- `const`: hardcoded value — no user input, does NOT generate a form field
- `op` supported: `=`, `!=`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `LIKE`, `NOT LIKE`, `IS NULL`, `IS NOT NULL`
- Logical operators: `AND` (array, length ≥ 2), `OR` (array, length ≥ 2), `NOT` (single dict); OR/AND nestable
- `const` conditions are included in the sent `data_filter` JSON but generate no form inputs
- Empty filter values are excluded from request params (not sent as empty strings)

## File Management Requirements
- All CRUD definition files **must** be stored in the `json/` directory of the module
- Each table gets one or more JSON files (multiple if using aliases)
- Naming: `{table_name}.json` or `{alias}.json` — e.g. `users` → `json/users.json`; alias `user_admin` → `json/user_admin.json`

## Key Implementation Notes
- **Field exclusion**: `browserfields.exclouded` hides fields in read-only/list view; `editexclouded` hides in edit forms (list view); `edit_exclouded_fields` hides in edit forms (tree view)
- **Dynamic data loading**: static `data` array (value/text pairs) | dynamic `dataurl` + `datamethod` + `dataparams` | cross-module data via `references/cross-module-dataurl.md` (multi-DB query pattern)
- **Event binding**: only in list view `editor.binds`, uses JavaScript for dynamic form behavior (e.g. cascading dropdowns); `wid` = source field, `target` = destination field
- **Security**: always specify `confidential_fields` for sensitive data; use `logined_userorgid`/`logined_userid` for data isolation; set `editable: false` for read-only tree views

## Integration Requirements
- Frontend: `bricks-framework`; backend: `sqlor-database-module`; references table definitions in `models/`
- Follows module structure from `module-development-spec` skill
- Each `data_url` / `editable` URL must have a matching `.dspy` endpoint — see `references/api-endpoint-patterns.md`
- CRUD files are generated from JSON via `xls2ddl.xls2crud` — see `references/xls2crud-generation.md`

## Validation Checklist
- [ ] View type correctly chosen (tree vs list based on table relationships)
- [ ] Required fields present for chosen view type
- [ ] `tblname` exactly matches a table defined in table definition
- [ ] `params` dict exists and is non-empty
- [ ] `editable` paragraph exists with `new_data_url`, `update_data_url`, `delete_data_url`
- [ ] `data_url` exists and points to a valid `.dspy` list endpoint
- [ ] Every field in `browserfields.exclouded` exists in the model's field list
- [ ] Every field in `browserfields.alters` keys exists in the model's field list
- [ ] Every field in `editexclouded` / `edit_exclouded_fields` exists in the model's field list
- [ ] All NOT NULL DEFAULT columns that aren't user-editable are in `editexclouded` (prevents "cannot be null" on submit)
- [ ] `alters` entries use `uitype: "code"` with `dataurl` (endpoint returns plain `[{value,text}]` array) or `data` array (static)
- [ ] **`alters` entries with `valueField`/`textField` MUST also have `uitype: "code"`** (else text mapping silently ignored — see Pitfall 33)
- [ ] `subtables[].url` uses `{{entire_url('../alias')}}` with `../` prefix, no `wwwroot` in path
- [ ] `editor.binds[].actiontype` is one of: urlwidget, method, script, registerfunction, event
- [ ] `entire_url()` arguments are quoted strings
- [ ] No forbidden root keys: `tablename` (use `tblname`), `grid`, `form`, `name`, `type`, `components`
- [ ] **No Jinja2 control blocks in CRUD JSON** (`{% if %}`, `{% for %}`). `{{entire_url(...)}}` in strings is OK. See `references/crud-json-rules.md`.
- [ ] File stored in correct `json/` directory with `{table_name}.json` naming
- [ ] All referenced fields exist in table definition (`models/` directory)
- [ ] Security fields properly configured (`confidential_fields` + `browserfields.exclouded`)
- [ ] Subtable refs valid: `field` and `subtable` keys present, `subtable` matches an existing table in `models/`, and a corresponding wwwroot directory or CRUD config exists
- [ ] Every `data_url` / `editable` URL has a matching `.dspy` endpoint file in `wwwroot/api/`
- [ ] If `data_filter` present, corresponding `.dspy` list endpoint uses `DBFilter` to parse it
- [ ] **CRUD endpoint audit**: Run `scripts/verify-crud-endpoints.py` from `~/repos/` to check all create/update/delete `.dspy` files for json.dumps wrapping, wrong return format, and sor.U argument count. Fix all failures before commit.

## Forbidden Root Keys (DO NOT USE)
The following keys do not exist in the CRUD spec and will cause failures:
- `tablename` → use `tblname`
- `grid` → does not exist; CRUD files do NOT support `fields`/`joins`/`select_fields` (no SQL joins, no cross-table fields) — use `browserfields.exclouded` + `alters`
- `form` → does not exist — use `editexclouded` + `alters` with uitype
- `name`, `type`, `components` → not CRUD keys; such files are `.ui` files in `wwwroot/`, not CRUD JSON (see Pitfall 9)

## Common Pitfalls

### Pitfall 1: Root key is `tblname`, not `tablename`
`"tablename": "users"` is invalid — must be `"tblname": "users"`.

### Pitfall 2: CRUD files reference ONLY the base table
No SQL joins, `select_fields`, or cross-table field references. All fields in `browserfields.exclouded`, `editexclouded`, and `alters` must exist in the table definition (`models/`) for the table in `tblname`. Example: for `financial_vouchers`, reference only fields like `contract_id`, `voucher_number`, `amount` — never `contract_number` (that field is in the `contract` table).

### Pitfall 3: Dropdown fields use `alters` with `uitype: "code"`
Dropdowns go in `browserfields.alters`, never a `form` section.
- Static: `"alters": {"status": {"uitype": "code", "data": [{"value": "1", "text": "Active"}, {"value": "0", "text": "Inactive"}]}}`
- Dynamic: `"providerid": {"uitype": "code", "dataurl": "{{entire_url('../api/get_organizations.dspy')}}"}`

Rules:
- `dataurl` must use `{{entire_url('...')}}` with a quoted string
- Endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object; on error or empty data return `[]`
- **Prefer `appcodes`/`appcodes_kv` (model `codes` definitions) over inline `data` arrays** for fixed options — see `references/appcodes-pattern.md`
- `data_field` is **deprecated** (old nested-response pattern `{"data": {"organizations": [...]}}`) — DO NOT use; wrapped formats make the dropdown silently not render
- `valueField`/`textField` are **NOT deprecated** — required when the data source returns keys other than `value`/`text` (see Pitfall 34)

### Pitfall 4: Field hiding uses `exclouded`/`editexclouded`, not field-level `"hidden": true`
No `{"name": "org_id", "widget": "hidden"}`. Use `"editexclouded": ["org_id"]` (hides in edit form) and `"browserfields": {"exclouded": ["org_id"]}` (hides in list view).

### Pitfall 5: Always load this skill BEFORE creating/modifying CRUD files
Never guess the CRUD format — the user has zero tolerance for guessed/improvised formats. If unsure about any property, load this skill and follow the examples verbatim.

### Pitfall 6: Confidential fields should be hidden in browser view
Sensitive fields (API keys, passwords, tokens) must be in BOTH `confidential_fields` (server-side redaction) AND `browserfields.exclouded` (removed from the grid).

### Pitfall 7: `editable` paragraph required for ALL CRUD files (even list-only views)
Every CRUD JSON file — including list-only views — MUST include `editable` with `new_data_url`, `update_data_url`, `delete_data_url` (optionally `get_data_url`). Without it the framework cannot process form submissions. URLs use `{{entire_url('../api/xxx.dspy')}}` format with `../` prefix:
```json
"params": {"editable": {
    "new_data_url": "{{entire_url('../api/table_create.dspy')}}",
    "update_data_url": "{{entire_url('../api/table_update.dspy')}}",
    "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}"
}}
```
It must be an **object** — never the string `"default"` (see Pitfall 41).

### Pitfall 8: Field references MUST match model definitions exactly
All names in `browserfields.exclouded`, `browserfields.alters` keys, and `editexclouded` must be exact matches to fields in the table's model JSON (`models/`). Even a one-character difference fails. Common mismatches:

| Wrong Field | Correct Field |
|-------------|---------------|
| `org_id` (not in model) | Remove it |
| `sales_stage` | `current_stage` |
| `source` | `source_type` |
| `is_active` | `is_won_stage` / `is_lost_stage` |
| `changed_by` | `changed_by_id` / `changed_by_name` |

### Pitfall 9: Non-CRUD files must not be placed in `json/` directory
Files with custom structures (`{"name": ..., "title": ..., "type": "page", "components": [...]}`) are NOT CRUD definitions and cause framework failures. They belong as `.ui` files in `wwwroot/`.

### Pitfall 10: Validate ALL json/ files when touching any file in the directory
When the task involves any CRUD file modification, scan EVERY `.json` file in `json/` against this spec — not just the files directly involved. Zero tolerance for non-CRUD files left in or added to `json/`.

### Pitfall 11: ID values must use `appPublic.uniqueID.getID()`, not `uuid.uuid4()`
Database `id` columns are typically VARCHAR(32); `uuid.uuid4().replace('-', '')` may exceed the column length. Always use:
```python
from appPublic.uniqueID import getID
new_id = getID()
```
Applies to both `.dspy` API files and Python backend code.

### Pitfall 12: Hand-written `get_*_list.dspy` SHADOWS framework auto-generated list endpoints
When `json/{table}_list.json` exists, the CRUD framework auto-generates the list endpoint. A hand-written `wwwroot/api/get_{table}_list.dspy` shadows it → **500 errors** (filters on nonexistent fields), **403 errors** (bypasses RBAC and `logined_userorgid` handling), **silent data leakage** (misses `confidential_fields` redaction).
**Rule**: if `json/{table}_list.json` exists, do NOT create `get_{table}_list.dspy`. Only write hand-written dspy for custom business endpoints (create, update, delete, client-specific actions).
**Detect**: list endpoint 500/403 → (1) does `json/{alias}.json` exist? (2) does `wwwroot/api/get_{table}_list.dspy` also exist? (3) if both → DELETE the hand-written dspy.

### Pitfall 13: `subtables[].subtable` must reference an existing table with accessible UI
The `subtable` value needs: (1) a table definition in `models/`, (2) a corresponding wwwroot directory (or properly configured CRUD). A phantom reference (e.g. `"subtable": "llmtype"` with no model/wwwroot) breaks the UI silently — the tab renders but shows no data.
**CRITICAL — explicit `url` when default path lacks RBAC permissions**: the framework's default path `/module/subtable_name` may not be registered in RBAC → subtab returns 401. Always set an explicit `url` to a `.ui` file that already exists:
```json
"subtables": [{"field": "llmid", "title": "能力映射",
    "url": "{{entire_url('./llm_api_map_manage.ui')}}", "subtable": "llm_api_map"}]
```
Verify: `find wwwroot -type d` (CRUD dir for subtable), `ls models/<subtable>.json`, `grep -r subtable_name wwwroot` (existing `.ui` files). **Default: always set `url` explicitly to a concrete `.ui` file** rather than relying on framework path generation.

### Pitfall 14: NEVER replace CRUD auto-generated endpoints with custom scripts
Auto-generated `get_{table}.dspy` handles RBAC, `logined_userorgid`, `confidential_fields` redaction, and DBFilter parsing correctly; custom scripts often violate `.dspy` conventions and create maintenance burden. **Wrong**: creating `wwwroot/api/llm_list.dspy` to replace `get_llm.dspy` just to add `_text` fields. **Correct**: keep the auto-generated endpoint and fix the `dataurl` API to return `[{field_name, field_name_text}]` instead. Only propose custom scripts for a genuine special requirement the framework cannot handle — and **discuss the approach first**.
Validate configs: `python references/validate_crud.py <module>/json/ --model-dir <module>/models/` (forbidden keys, missing required keys, nonexistent field refs, improper alters syntax). Module-wide audit: follow `references/crud-audit-procedure.md`.

### Pitfall 15: NEVER manually add `data_url` to CRUD JSON
The framework auto-generates `get_{table}.dspy`; adding `"data_url": "{{entire_url('../api/get_llm.dspy')}}"` overrides it with a non-existent path → 500. Omit `data_url` entirely.
**Exception**: only add `data_url` for a genuinely custom list endpoint whose `.dspy` file actually exists. For `_text` FK display, create `get_search_{fieldname}.dspy` (Pitfall 20) and reference it in `alters[field].dataurl` — do NOT modify the auto-generated list endpoint.

### Pitfall 16: Generated CRUD directories are NOT committed to git
`xls2ddl.xls2crud` generates `wwwroot/<table>/` (index.ui, get/add/update/delete .dspy) — these are auto-generated artifacts and must be gitignored (one `.gitignore` entry per table). The `json/` CRUD definitions ARE committed (they are the source). See `references/xls2crud-generation.md`.

### Pitfall 17: Custom `data_url` requires xls2ddl template support
For CRUD JSON `data_url` overrides to take effect, the xls2ddl template (`data_browser_tmpl`) must use `{% if data_url %}` (not `{% if get_data_url %}`). Fixed in xls2ddl commit `9f9a60a`. If the generated index.ui ignores `data_url`, update xls2ddl.

### Pitfall 18: Code-type (uitype: "code") field debugging — never blame bricks first
If a code-type field shows `undefined`, raw IDs, or wrong values, the problem is in the **data layer** (API response format, dataurl path, backend query) — NOT bricks (all code fields share the same component; a framework bug would hit all of them).
- Filter dropdown `undefined` → check dataurl returns `[{value, text}]` (ONLY correct format); check the path resolves; check the endpoint isn't swallowing errors into `[]`
- Raw IDs in grid cells → the dataurl API must return `[{value, text}]` via SQL aliases:
```python
orgs = await sor.sqlExe("select id as value, orgname as text from organization order by orgname", {})
return orgs
```
Do NOT edit the auto-generated `wwwroot/<table>/get_<table>.dspy` (regenerated from templates; it doesn't join reference tables). For grid `_text` resolution use the custom LEFT JOIN list endpoint — see Pitfall 47.

### Pitfall 19: data_filter dropdown must have empty/default option — FIXED
Fixed in bricks commit `f8f02c6` (`get_filter_fields()`): DataViewer auto-injects `{value: '', text: ''}` as the first option for all code-type filter fields (both `data` and `dataurl` sources), unless data already contains an empty/null/undefined entry. No backend changes needed. Old symptom (if unfixed): user selects a value and cannot reset to "show all".

### Pitfall 20: codes fields need dedicated `get_search_{fieldname}.dspy` for filter dropdowns
When a model's `codes` section defines a foreign key (e.g. `providerid` → `organization`), the `browserfields.alters` dataurl should point to a dedicated `get_search_{fieldname}.dspy` (single-purpose, includes the "全部" fallback), not the generic list endpoint:
```python
result = [{'value': '', 'text': '全部'}]
try:
    async with get_sor_context(request._run_ns, 'rbac') as sor:
        orgs = await sor.sqlExe("select id as value, orgname as text from organization order by orgname", {})
        return json.dumps([{'value': '', 'text': '全部'}] + list(orgs), ensure_ascii=False)
except Exception as e:
    debug(f'get_search_providerid error: {e}')
return json.dumps(result, ensure_ascii=False)
```
Key rules:
- Return `[{value, text}]` — SQL aliases `id as value, name as text`
- **CRITICAL: on success prepend 全部** via `[{'value': '', 'text': '全部'}] + list(orgs)` — do NOT just `return orgs` (drops the 全部 option)
- On error: return fallback with only 全部; no imports allowed (json, get_sor_context, debug are pre-loaded)
- Name: `get_search_{fieldname}.dspy` in `wwwroot/api/`; register in `load_path.py`
- **CRITICAL: also register in DB `permission` + `rolepermission` tables** — `load_path.py` alone is NOT sufficient for new `api/*.dspy` endpoints. After deploying, run on the target server:
```sql
INSERT INTO permission (id, path) VALUES (REPLACE(UUID(),'-',''), '/module/api/endpoint.dspy');
INSERT INTO rolepermission (id, roleid, permid)
  SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/endpoint.dspy';
```
Path MUST be `/module/api/xxx.dspy` (files in `wwwroot/api/` are served at `/module/api/`). Without DB registration → `403 Forbidden` even if load_path.py is correct. Use role `'logined'` for endpoints any authenticated user can call.
**Architecture note**: edit-form dropdowns use `alters.dataurl`; filter dropdowns use model `codes` → `get_code.dspy`. Both must be updated when changing a field's data source. See `references/filter-vs-edit-dropdown.md`.

### Pitfall 21: NOT NULL DEFAULT columns MUST be in `editexclouded`
If a column has `NOT NULL DEFAULT <value>` (e.g. `login_fail_count SMALLINT NOT NULL DEFAULT 0`, `created_at TIMESTAMP NOT NULL DEFAULT current_timestamp()`) and users should NOT edit it, it MUST be in `editexclouded`. Otherwise the edit form renders an empty input and submits NULL → `(1048, "Column 'xxx' cannot be null")`.
```json
"editexclouded": ["id", "created_at", "login_fail_count", "last_login", "last_login_fail"]
```
Common culprits: `DEFAULT current_timestamp()` timestamps, `DEFAULT 0` counters, `DEFAULT '0'` status columns.

### Pitfall 22: `record_toolbar` pattern for state-change action buttons
Per-row action buttons (enable/disable, approve/reject, activate/deactivate):
```json
"params": {"record_toolbar": [{
    "label": "启用", "actiontype": "dspy",
    "url": "/module/table/enable_record.dspy",
    "options": {"icon": "check", "cwidth": 16, "cheight": 9}
}]}
```
dspy handler pattern: check `params_kw.get('id')` (return `{"widgettype":"Error",...}` if missing); then `db = DBPools(); async with db.sqlorContext(get_module_dbname('module_name')) as sor: await sor.U('table_name', {'id': params_kw.id, 'status_field': 'new_value'})` (dict with `id` + fields — do NOT pass a 3rd argument); return `{"widgettype":"Message",...}`.
Key rules: each button needs its own `.dspy` in `wwwroot/`; dspy receives `params_kw.id`; register all paths in `load_path.py`; use `cwidth`/`cheight` in options (not fixed px).
**Global toolbar buttons** (`toolbar.tools` + `binds`): see `references/toolbar-tools-binds.md` — sit above the list (unlike per-row `record_toolbar`), require `selected_row: true` + a `binds` entry with `wid: "self"`; covers urlwidget→PopupWindow, params_mapping, `${id}$` placeholder, DSPY return format, 403→load_path→restart flow.

### Pitfall 23: dspy files must NOT have import statements — use pre-loaded modules only
Every `.dspy` file is injected into a pre-built async function context. Pre-loaded (NEVER import):
- From ahserver `y_env`: `debug`, `exception`, `error`, `info`, `warning`, `critical`; `get_user`, `get_username`, `get_userorgid`, `get_userinfo`; `entire_url`, `i18n`, `redirect`, `clientinfo`, `terminalType`
- Globals (injected at compile time): `json`, `datetime` (datetime/date/timedelta), `time`, `DictObject`, `DBPools`, `get_sor_context`, `getID`, `curDateString`, `timestampstr`, `FileStorage`, `partial`, `params_kw`, `format_exc`
- **Only allowed import**: `from sqlor.filter import DBFilter`
Common violations (all WRONG): `import json`, `import time`, `import datetime`, `from appPublic.uniqueID import getID`, `from appPublic.log import debug`, `from appPublic.dictObject import DictObject`, `from appPublic.timeUtils import curDateString, timestampstr`, `from sqlor.dbpools import get_sor_context, DBPools`, `from functools import partial`, `from ahserver.filestorage import FileStorage`.
If a module-internal function is needed (e.g. `from llmage.utils import get_llmusage_by_id`), export it via the module's `load_XXX()` in `init.py` (`env.get_llmusage_by_id = get_llmusage_by_id`), then dspy calls it directly.
Audit before every commit touching .dspy: `grep -rn '^import \|^from ' wwwroot/ --include='*.dspy' | grep -v 'sqlor.filter'` — must return empty; delete any matching import line.

### Pitfall 24: Tabular edit sends `_text` suffix fields — MUST strip before sor.U/sor.C
Tabular's edit form collects ALL field data including `_text` display columns (`user_status_text`, `orgid_text`, `sync_from_text`) that are NOT real DB columns; passing them to `sor.U`/`sor.C` fails silently or skips the update. In EVERY add/update `.dspy` after `ns = params_kw.copy()`:
```python
ns = params_kw.copy()
for k, v in ns.items():
    if v == 'NaN' or v == 'null':
        ns[k] = None
# remove _text suffix fields sent by Tabular (not real DB columns)
for k in list(ns.keys()):
    if k.endswith('_text'):
        ns.pop(k, None)
```
Affects ALL `add_*.dspy`/`update_*.dspy` generated by xls2crud, plus any custom form handler receiving Tabular/Form code-type fields. Detection: grep for `_text` in params_kw before sor.C/U calls.

### Pitfall 25: Delegate-pattern endpoints (return result from helper) are valid — do NOT flag as format errors
A `_create.dspy` / `_update.dspy` / `_delete.dspy` that just calls a helper and returns its result (`result = await create_marketing(request, params_kw); return result`) is VALID — the helper (in `init.py`) returns the correct widgettype format. Do NOT flag as `no_widgettype` errors. Only flag endpoints that construct their own return value (e.g. `{'success': True, ...}`) without widgettype. Detect: last line `return result` / `return json.loads(result)` + a `result = await helper(...)` line → delegate pattern, skip format checking.

### Pitfall 26: Model name variants (dot vs hyphen) must be covered in model_mappings
LLM API responses may use dotted versions (`doubao-seedance-2.0`) while pricing YAML entries use hyphens (`doubao-seedance-2-0`); `model_mappings` that only map full version suffixes miss the dot-variants → `{config_data=..., mismatched}` / `没有找到合适的定价`. Fix — map all dot-variants explicitly:
```yaml
model_mappings:
  doubao-seedance-2.0: doubao-seedance-2-0
  doubao-seedance-2.0-fast: doubao-seedance-2-0-fast
  doubao-seedance-2-0-260128: doubao-seedance-2-0
```
Detection: compare the incoming `model` field against pricing filter values; if they differ only by `.` vs `-` in version numbers, add the mapping. Cross-skill: also affects `pricing-data-format`.

### Pitfall 27: Tabular sends request params via `data_params`, NOT `params`
bricks DataViewer reads `this.opts.data_params` as the default request parameters — NOT `this.opts.params` (which is why a Tabular gets no `discountid` and returns empty data). **Wrong**: `"params": {"discountid": "{{params_kw.discountid}}"}`. **Correct**:
```json
"data_url": "...dspy",
"data_params": {"discountid": "{{params_kw.discountid}}"}
```
Applies to ALL hand-written Tabular widgets, including subtable pages; CRUD auto-generated files handle this correctly via the template.

### Pitfall 28: data_filter conflicts with logined_userorgid — FIXED in xls2ddl
Fixed in xls2ddl commit `ebd4b4a`: the generated `get_<table>.dspy` now injects `logined_userorgid`/`logined_userid` conditions into `filterjson` before DBFilter processes it (appending `{'field': '<orgfield>', 'op': '=', 'var': '__logined_orgid__'}` + `ns['__logined_orgid__'] = userorgid`; same for userid). Ensures both work together. Old workaround (manual ownerid filter in dspy / const in data_filter) no longer needed. After updating xls2ddl, regenerate affected modules.

### Pitfall 29: Subtables auto-population — use `field` + `mapping` to pass parent values
For a subtable's add form to receive the parent record's ID (e.g. supplier contract needs `supplier_id`): (1) add the field to the subtable's `editexclouded` so users don't edit it; (2) carry the value via `new_data_url`:
```json
"editexclouded": ["id", "resellerid", "supplier_id"],
"editable": {"new_data_url": "{{entire_url('../api/create.dspy')}}?supplier_id={{params_kw.get('supplier_id','')}}"}
```
The xls2crud `params_mapping.mapping` sends `parent.id → subtable.field` to the subtable page as URL params; the add form's `new_data_url` carries it to the dspy.

### Pitfall 30: xls2ddl `json.dumps(true)` generates Python-invalid `true` — MUST wrap in `json.loads()`
Generated dspy fails `NameError: name 'true' is not defined` (JSON booleans like `"not_null": true` are valid JSON but not Python). Fix in xls2ddl `tmpls.py` — **CRITICAL: the `json.loads()` call MUST have single quotes around the Jinja2 expression** (without quotes it receives a dict → `TypeError: the JSON object must be str, bytes or bytearray, not dict`):
```python
# WRONG:  tblfields = {{json.dumps(fields, ensure_ascii=False)}}
# WRONG:  tblfields = json.loads({{json.dumps(fields, ensure_ascii=False)}})   # dict, not string
# CORRECT:
tblfields = json.loads('{{json.dumps(fields, ensure_ascii=False)}}')
filterjson = json.loads('{{json.dumps(data_filter, ensure_ascii=False)}}')
ns['sort'] = json.loads('{{json.dumps(sortby)}}')
```
Apply ONLY to Python-context lines (tblfields, filterjson, sort arrays). JSON-context lines (browserfields, toolbar, binds) use `true`/`false` correctly and must NOT be wrapped. Commit: xls2ddl `af6006f`.

### Pitfall 31: Post-insert re-query uses `primary`, not `pkey` — xls2ddl template bug
`OperationalError: (1054, "Unknown column 'None' in 'WHERE'")` with `SELECT * FROM <table> WHERE None = %s` after adding a record (xls2ddl `fc91486` added a post-insert re-query `... WHERE {{summary[0].pkey}} = ${id}$`; model JSONs use `primary` (array), not `pkey` → Jinja2 renders `None`). Fix: `{{summary[0].pkey}}` → `{{summary[0].primary[0]}}` in xls2ddl `tmpls.py`, then regenerate all modules.

### Pitfall 32: logined_userorgid generates `WHERE None = %s` — FIXED in xls2ddl
Same `(1054, "Unknown column 'None' in 'WHERE'")` symptom when `logined_userorgid` is configured but the user's orgid is not set. Fix: update xls2ddl to commit `ebd4b4a` or later, then regenerate ALL affected modules:
```bash
cd ~/repos/xls2ddl && git pull
cd ~/repos/<module> && PYTHONPATH=~/repos/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot <mod> json/*.json
```

### Pitfall 33: `valueField`/`textField` in alters MUST also have `uitype: "code"`
Without `uitype: "code"` the framework treats the field as plain text and **silently discards the textField mapping** — grid shows raw IDs even though the API returns `{fieldname}_text` columns:
```json
// WRONG: {"llmid": {"valueField": "llmid", "textField": "llmid_text"}}
// CORRECT:
{"llmid": {"uitype": "code", "valueField": "llmid", "textField": "llmid_text"}}
```
Applies to both hand-written `.ui` files and CRUD JSON `browserfields.alters`. See Pitfall 34 for the full valueField/textField pattern.

### Pitfall 34: `valueField`/`textField` in alters apply to BOTH filter form AND edit form
`browserfields.alters` is a **single configuration** shared by the list grid, the filter/search form, and the add/edit form. Use valueField/textField when: the `get_search_*.dspy` returns `{fieldname, fieldname_text}` keys (not `{value, text}`), the list endpoint returns `{fieldname}_text` columns, and you need the stored value key to be the actual field name:
```json
"providerid": {
    "uitype": "code",
    "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}",
    "valueField": "providerid",
    "textField": "providerid_text"
}
```
The dspy MUST return those exact keys: `select id as providerid, orgname as providerid_text from organization order by orgname`, and the "全部" fallback must use the same keys (`{'providerid': '', 'providerid_text': '全部'}`).
Key rules: valueField/textField must exactly match the endpoint's returned keys; the same pair is used in filter form AND edit form (they cannot differ); if the dspy returns `{value, text}`, do NOT set valueField/textField (framework defaults). Symptom if wrong: filter dropdown shows `undefined`/raw IDs; edit form sends wrong values; filter selection doesn't match stored data.

### Pitfall 35: MUST regenerate index.ui on each server after modifying CRUD JSON alters
`wwwroot/<table>/` (index.ui) is auto-generated by `xls2crud` and **gitignored** (Pitfall 16) — `git pull` on a server leaves stale UI. After ANY change to `browserfields.alters` (dataurl/valueField/textField/uitype/data), `browserfields.exclouded`, `editexclouded`, `data_filter`, `filter_labels`, `filter_title`, `subtables`, `record_toolbar`, `editor.binds`, or model definitions in `models/`, re-run on EACH environment:
```bash
cd /path/to/module
PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot <modulename> json/<table>.json
```

### Pitfall 36: CRUD page not triggering data request — diagnose CRUD spec FIRST, not permissions
When a CRUD list page loads but does NOT trigger the `get_<table>.dspy` request, the root cause is the CRUD spec or generated template — NOT RBAC/permissions. Diagnostic order: (1) generated `index.ui` has a `data_url` pointing to the right endpoint; (2) CRUD JSON has required structure (tblname, params, editable); (3) `get_<table>.dspy` exists in the CRUD directory; (4) diff against a working module (e.g. llmage/llm, pricing); (5) only then check RBAC/load_path.py.
Common causes (frequency order): stale index.ui (not regenerated), data_url missing/incorrect in generated index.ui, CRUD JSON missing required keys, outdated xls2ddl template. **Anti-pattern**: assume permissions first, shuffle PATHS_ANY/PATHS_LOGINED, or blame bricks before checking the CRUD config.

### Pitfall 37: `data_url` vs `get_data_url` in generated index.ui — two different mechanisms
1. **`data_url`** (outside `editable` block) — initial page load by Tabular/DataViewer. Defaults to `./get_<table>.dspy`; overridden if CRUD JSON has a `data_url` key.
2. **`get_data_url`** (inside `editable` block, optional) — overrides the data URL with extra parameters (e.g. `?pagerows=50`). Only generated when the CRUD JSON explicitly defines it:
```json
"editable": {"get_data_url": "{{entire_url('get_llmusage.dspy')}}?pagerows=50", "new_data_url": "...", ...}
```
Most CRUD pages only need the auto `data_url`. Tabular uses `data_url` on initial render, then `get_data_url` for subsequent refreshes if defined. If neither exists in generated index.ui, the table renders empty with no network request.

### Pitfall 38: `data_filter` MUST use DBFilter tree structure, NOT `{"fields": [...]}`
A flat `"fields": [{"field": ..., "title": ..., "uitype": "code"}]` silently fails (search does nothing / popup doesn't appear). Correct — DBFilter tree with op/var:
```json
"data_filter": {"AND": [
    {"field": "supplier_org_id", "op": "=", "var": "supplier_org_id"},
    {"field": "resource_type", "op": "=", "var": "resource_type"}
]}
```
Add a `"filter_labels"` object for Chinese labels.

### Pitfall 39: NEVER directly edit auto-generated .ui files — always modify source configs
Files in `wwwroot/<table>/` (especially index.ui) are auto-generated by `xls2crud` — never edit directly, even in production (e.g. no `sed -i` on production index.ui). Correct approach: (1) find the source config — `json/<table>.json` for CRUD-generated UIs, `wwwroot/<custom_name>.ui` for hand-written UIs; (2) modify the source; (3) regenerate (CRUD only, see Pitfall 35 command); (4) deploy.
How to tell if auto-generated: `json/<table>.json` exists → yes; directory contains `index.ui` + `get_<table>.dspy` → yes; files outside `wwwroot/<table>/` (e.g. `wwwroot/custom_feature.ui`) → hand-written, editable directly.

### Pitfall 40: CRUD list performance — FOUR critical optimizations required
Symptom: list page 3+ seconds. **File location**: never edit the gitignored, regenerated `wwwroot/<table>/get_<table>.dspy` — write `wwwroot/api/<table>_list.dspy` and point `editable.get_data_url` at it (committed, survives regeneration). Apply optimizations in order (each 2-5x; together 20-40x):
1. **`sqlPaging` wraps queries in subqueries (slowest)** → separate count + data queries: `select count(*) as cnt ...` then `select col1, col2 ... limit {rows_per_page} offset {(page-1)*rows_per_page}` with `page = int(ns.get('page', 1))`, `rows_per_page = int(ns.get('rows', ns.get('pagerows', 50)))`
2. **`default_filterjson` generates LIKE for ALL fields incl. large TEXT (full table scan)** → exclude TEXT columns: `filter_fields = [f['name'] for f in ori_fields if f['name'] not in ('usages', 'ioinfo')]`
3. **`SELECT *` pulls large TEXT/BLOB (off-page storage I/O)** → explicit column list
4. **DBFilter + ArgsConvert framework overhead** → bypass entirely with raw SQL + manual WHERE on 5-7 common filter fields (`conditions = ['1=1']`, append `'field=${var}$'` + `ns['var'] = value` only when value present), using `DBPools()` + `get_module_dbname('module_name')`; return `{'success': True, 'total': total, 'rows': rows, 'page': page, 'page_size': rows_per_page}`; wrap in try/except with `debug(format_exc())` and `{'success': False, ...}` fallback.
Real-world (llmage/llmusage, 17 columns): sqlPaging + all-fields filter + SELECT * → 8-12s; after #1: 3-4s; after #2: 1-2s; after #3: 0.8-1.5s; after #4: 0.2-0.5s.

### Pitfall 41: `"editable": "default"` string causes xls2ui serialization error
`"editable": "default"` (string instead of object) → `build.sh`/`xls2ui` fails with `TypeError: Object of type builtin_function_or_method is not JSON serializable` at `json.dumps(binds, ...)`. Fix: replace with the full object containing `get_data_url`, `new_data_url`, `update_data_url`, `delete_data_url` (the `"default"` shorthand is unsupported). This is a case of Pitfall 7 focused on *format* — must be an object, not a string.

### Pitfall 42: Field title renaming — modify `models/<table>.json`, NOT CRUD/UI files
To rename display titles (e.g. "用户id" → "username"), change `fields[].title` in `models/<table>.json`. The model `title` is the single source of truth, propagated to list grid headers, filter form labels, edit form labels, and add form labels. Do NOT edit titles in `json/<table>.json` browserfields (overrides only), generated `wwwroot/<table>/index.ui` (build artifact), or `wwwroot/api/get_<table>.dspy` (titles aren't in the query layer). After renaming, regenerate CRUD files. Real example (llmage/llmusage 2026-06-25, commit `13c123c`): userorgid "用户机构"→"orgname", ownerid "模型机构"→"orgname", userid "用户id"→"username", llmid "模型id"→"model".

### Pitfall 43: `logined_userorgid` and `logined_userid` go in `params`, NOT `browserfields`
Putting them inside `browserfields` crashes `xls2ui` with `JSONDecodeError` or places them wrongly in generated files. They are first-class `params` keys, siblings of `sortby`, `data_filter`, `editable`:
```json
"params": {
    "logined_userorgid": "customerid",
    "browserfields": {"exclouded": ["id"]}
}
```

### Pitfall 44: `editable.new_data_url` was NEVER read by xls2ddl template — fixed in fb613d0
CRUD JSON with `params.editable.new_data_url` set, but the generated index.ui still uses the default `add_<table>.dspy` (custom create dspy never called). Root cause: the template checked `{% if new_data_url %}` at top level, but after `desc.update(crud_data.params.copy())` the key is nested in `desc.editable`; Jinja2 resolves it to `None` → `{% else %}` always runs. Fix: xls2ddl commit `fb613d0`+ — template checks `{% if (editable and editable.new_data_url) or new_data_url %}` (same for `delete_data_url`, `update_data_url`). Was masked while pre-gitignore generated dspy files existed on servers. Regenerate after updating xls2ddl.

### Pitfall 45: filter_fields generated without inline data from browserfields.alters
Search popup crashes with `TypeError: Cannot read properties of undefined (reading 'length')` at `bricks.UiCode.build_options` when a data_filter field has inline `data` only in `browserfields.alters` (e.g. `{"uitype":"code","data":[...]}`) but no corresponding model `codes` entry — filter_fields gets a code header with `data: undefined`. Fix: xls2ddl commit `faa571f`+ (merges alters `data`/`dataurl`/`valueField`/`textField` into filter_fields, applies alters `uitype` override). Regenerate affected CRUD pages.

### Pitfall 46: CRUD JSON files must be PURE JSON — NO Jinja2 control-flow blocks
`json/` files are parsed as pure JSON by `xls2ddl.xls2crud`; `{% if %}`, `{% for %}`, `{% endif %}` cause `JSONDecodeError` at parse time. Use fixed values instead (e.g. hardcode `"width": "40%"` instead of `{% if params_kw._is_mobile %}`). `{{entire_url('...')}}` placeholders inside JSON string values ARE acceptable — handled during URL generation. This is distinct from `.ui` template files which DO support Jinja2. See `references/crud-json-rules.md`.

### Pitfall 47: Custom list endpoint for code resolution — LEFT JOIN reference tables for `_text` grid display
Symptom: form/filter dropdowns show names (via `uitype: "code"` + dataurl) but the list grid still shows raw IDs. Root cause: the auto-generated `get_<table>.dspy` queries only the base table — no reference-table joins. `uitype:"code"` + dataurl only controls form/filter dropdowns, NOT grid cells; **inline `data` arrays for status/type fields also do NOT auto-resolve for grid cells**. Pitfalls 14/15 don't cover this — the auto-generated list never LEFT JOINs, and a custom list endpoint is the only way to get grid code resolution.
**Approach A — COALESCE (recommended, the only reliable one)**: replace raw IDs with display names directly in SQL output — guarantees grid rendering regardless of bricks `_text` detection:
```python
sql = '''select a.id, a.domain,
    COALESCE(b.orgname, a.resellerid) as resellerid,
    COALESCE(c.orgname, a.orgid) as orgid,
    CASE a.status WHEN 'active' THEN '启用' WHEN 'inactive' THEN '停用' ELSE a.status END as status
from (select * from tenant_domain where 1=1 [[filterstr]]) a
left join (select id, orgname from organization) b on a.resellerid = b.id
left join (select id, orgname from organization) c on a.orgid = c.id'''
```
Filter WHERE applies on the inner query with raw IDs before COALESCE. Con: the edit form must use a separate `_get.dspy` returning raw IDs.
**Approach B — `_text` suffix**: return `resellerid_text`/`orgid_text` columns via LEFT JOIN; bricks Tabular MAY auto-detect `_text` columns but detection is unreliable (2026-07-10 testing: did NOT render) — fall back to Approach A.
**Placement (two valid locations)**: (1) `wwwroot/{alias}/get_{alias}.dspy` — CRUD framework auto-discovers; no `get_data_url` needed; directory may be gitignored (use `git add -f`); takes precedence over `get_data_url`; (2) `wwwroot/api/<table>_list.dspy` + `editable.get_data_url` — requires xls2ddl ≥ fb613d0 (Pitfall 44).
**Key rules**: LEFT JOIN alias MUST match the field name; never edit the gitignored `wwwroot/<table>/get_<table>.dspy`; handle status/type fields in SQL with CASE WHEN; new dspy endpoints need BOTH `permission` + `rolepermission` (`role 'logined'`) SQL rows (see Pitfall 20).
**CRITICAL deployment pitfalls (2026-07-10):**
1. **Server restart required for CRUD directory dspys**: custom `get_{alias}.dspy` files in CRUD directories are cached at server STARTUP — hot-reload applies to `wwwroot/api/` but NOT to auto-generated CRUD directories. After deploying, restart Sage (`./stop.sh && ./start.sh`).
2. **`default_filterjson` trap — ns field pollution**: any field set in `ns` before `default_filterjson(fields, ns)` becomes an implicit filter. NEVER set business fields in `ns` — only framework vars (`__logined_orgid__`, `userorgid`). Example: `ns['resellerid'] = userorgid` leaked into default_filterjson and filtered the list to 1 row.
3. **Approach A (COALESCE) is the only reliable option** — Approach B's `_text` columns did not render in practice.