--- name: crud-definition-spec version: 1.0.0 description: Standardized specification for defining CRUD operations using JSON format with support for both list and tree view interfaces, integrated with bricks-framework frontend components. trigger_conditions: - User needs to create or modify CRUD definition files in JSON format - Task involves generating CRUD configurations for the json directory - Working with sqlor-database-module CRUD operations - Need to determine between list vs tree view based on table relationships --- # CRUD Definition Specification ## Overview This skill defines the standardized JSON format for CRUD (Create, Read, Update, Delete) operations that integrate with the bricks-framework frontend and sqlor-database-module backend. The framework automatically selects between list view and tree view based on table structure. ## View Type Determination - **Tree View**: Use when table has a self-referencing foreign key (parent-child relationship where one field points to another record's id in the same table) - **List View**: Use for all other tables without hierarchical relationships ## Common Root Properties (Both View Types) ```json { "tblname": "table_name", // Required: Actual table name "alias": "optional_alias", // Optional: Used to create multiple CRUD interfaces for same table "title": "Display Title", // Optional: If omitted, uses table title from table definition "params": { ... } // Required: View-specific parameters } ``` ## List View CRUD Specification ### Complete Structure ```json { "tblname": "table_name", "alias": "optional_alias", "title": "Display Title", "params": { "sortby": ["field1 desc", "field2"], "logined_userorgid": "org_id_field", "logined_userid": "user_id_field", "data_filter": { "AND": [ {"field": "field1", "op": "LIKE", "var": "filter_var1"}, {"field": "field2", "op": "=", "var": "filter_var2"} ] }, "confidential_fields": ["field1", "field2"], "editor": { "binds": [ { "wid": "source_field_id", // Required: Source widget ID (field name) "event": "changed", // Required: Event type (typically "changed") "actiontype": "script", // Required: Action type ("script" for JS) "target": "target_field_id", // Required: Target widget ID "script": "// JavaScript code" // Required: JS script content } ] }, "browserfields": { "exclouded": ["id"], // Optional: Fields to exclude from display "alters": { "field_name": { "uitype": "code", // Required: UI type ("code" for dropdown/select) // OR use dataurl approach: // "dataurl": "api/endpoint", // "datamethod": "GET", // "dataparams": {"param": "value"}, "data": [ // Required when uitype="code": Option data { "value": "v1", // Required: Actual stored value "text": "Display Text" // Required: Display text for option } ] } } }, "editexclouded": ["readonly_field"], // Optional: Fields excluded from edit forms "subtables": [ // Optional: Foreign key relationships { "field": "foreign_key_field", // Required: Foreign key field name "title": "Subtable Title", // Optional: Uses subtable title if omitted "url": "{{entire_url(subtable_alias)}}", // Required when alias defined "subtable": "related_table_name" // Required: Related table name } ] } } ``` ## Tree View CRUD Specification ### Complete Structure ```json { "tblname": "hierarchical_table", "alias": "optional_alias", "uitype": "tree", // Required: Must be "tree" for tree view "title": "Display Title", "params": { "idField": "id", // Required: Node ID field (typically "id") "textField": "display_field", // Required: Field used for node display text "sortby": ["field1 desc", "field2"], // Optional: Sort fields for tree nodes "confidential_fields": ["field1", "field2"], // Optional: Sensitive field names "browserfields": { "alters": {} // Optional: Field attribute modifications }, "logined_userorgid": "org_id_field", // Optional: Organization filtering field "logined_userid": "user_id_field", // Optional: User filtering field "editable": true, // Required: true=editable, false=read-only "edit_exclouded_fields": ["system_field"], // Optional: Fields excluded from editing "parentField": "parent_id", // Required: Field containing parent node reference "subtables": [ // Optional: Foreign key relationships { "field": "foreign_key_field", "title": "Subtable Title", "url": "{{entire_url(subtable_alias)}}", "subtable": "related_table_name" } ] } } ``` ### Filter/Search Integration (data_filter) CRUD definitions can include a `data_filter` field in `params` to enable search/filter UI. The filter definition follows the `sqlor/filter.py` DBFilter JSON format: ```json { "tblname": "llm", "params": { "data_filter": { "AND": [ {"field": "model", "op": "LIKE", "var": "model_input"}, {"field": "ppid", "op": "=", "var": "ppid_input"}, {"field": "status", "op": "=", "const": "1"} ] } } } ``` **Frontend behavior (bricks-framework DataViewer):** When a CRUD JSON has `data_filter` in `params`, the DataViewer automatically adds a "搜索" button to the toolbar. Clicking it opens a `PopupWindow` containing a `Form` widget. The form fields are dynamically generated from the `data_filter` definition: - Each `var` in the filter tree becomes a form input field - Fields with `browserfields.alters[field].uitype == "code"` render as dropdowns - Custom labels via `filter_labels: { "var_name": "显示名" }` in params - Custom button text via `filter_label: "自定义按钮名"` in params - Custom popup title via `filter_title: "自定义标题"` in params - **No inline search form is rendered** — the filter UI is exclusively triggered by the toolbar button Form submit collects user values → sends `data_filter` (JSON string) + each `var` value as URL params → backend `.dspy` uses `DBFilter.gen(ns)` for SQL WHERE clause. **Key rules:** - `var`: parameter name — value comes from user input in the popup form - `const`: hardcoded value — does not require user input, does NOT generate a form field - `op`: SQL operator — supported: `=`, `!=`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `LIKE`, `NOT LIKE`, `IS NULL`, `IS NOT NULL` - Logical operators: `AND` (array, length ≥ 2), `OR` (array, length ≥ 2), `NOT` (single dict) - OR/AND can be nested - `const` conditions are included in the sent `data_filter` JSON but do not generate form inputs - Empty filter values are excluded from the request params (not sent as empty strings) ## File Management Requirements ### Storage Location - All CRUD definition files **must** be stored in the `json/` directory of the module - Each table gets one or more JSON files (multiple if using aliases) ### Naming Convention - Filename format: `{table_name}.json` or `{alias}.json` - Examples: - Table `users` → `json/users.json` - Alias `user_admin` for users table → `json/user_admin.json` ## Key Implementation Notes ### Field Exclusion Patterns - **browserfields.exclouded**: Hides fields in read-only/list view - **editexclouded**: Hides fields in edit forms (list view) - **edit_exclouded_fields**: Hides fields in edit forms (tree view) ### Dynamic Data Loading For dropdown/select fields, you can either: 1. **Static data**: Use `data` array with value/text pairs 2. **Dynamic data**: Use `dataurl`, `datamethod`, and `dataparams` properties 3. **Cross-module data**: When options come from another module's database, see `references/cross-module-dataurl.md` for the multi-DB query pattern ### Event Binding - Only supported in list view editor.binds - Uses JavaScript for dynamic form behavior (e.g., cascading dropdowns) - `wid` = source field, `target` = destination field ### Security Considerations - Always specify `confidential_fields` for sensitive data - Use `logined_userorgid` and `logined_userid` for proper data isolation - Set `editable: false` for read-only tree views when appropriate ## Integration Requirements - Works with `bricks-framework` for frontend rendering - Integrates with `sqlor-database-module` for backend operations - References table definitions from `models/` directory - Follows module structure defined in `module-development-spec` skill - Each `data_url` / `editable` URL in CRUD JSON must have a matching `.dspy` endpoint file — see `references/api-endpoint-patterns.md` - CRUD files are generated from JSON definitions via `xls2ddl.xls2crud` — see `references/xls2crud-generation.md` ## Validation Checklist - [ ] View type correctly chosen (tree vs list based on table relationships) - [ ] Required fields present for chosen view type - [ ] `tblname` value exactly matches a table defined in table definition - [ ] `params` dict exists and is non-empty - [ ] `editable` paragraph exists with `new_data_url`, `update_data_url`, `delete_data_url` - [ ] `data_url` exists and points to a valid `.dspy` list endpoint - [ ] Every field in `browserfields.exclouded` exists in the model's field list - [ ] Every field in `browserfields.alters` keys exists in the model's field list - [ ] Every field in `editexclouded` / `edit_exclouded_fields` exists in the model's field list - [ ] All NOT NULL DEFAULT columns that aren't user-editable are in `editexclouded` (prevents "cannot be null" errors on form submit) - [ ] `alters` entries use `uitype: "code"` with `dataurl` (endpoint returns plain `[{value,text}]` array) or `data` array (static) - [ ] **`alters` entries with `valueField`/`textField` MUST also have `uitype: "code"`** (without it, text mapping is silently ignored — see Pitfall 35) - [ ] `subtables[].url` uses `{{entire_url('../alias')}}` with `../` prefix, no `wwwroot` in path - [ ] `editor.binds[].actiontype` is one of: urlwidget, method, script, registerfunction, event - [ ] `entire_url()` arguments are quoted strings - [ ] No forbidden root keys: `tablename` (use `tblname`), `grid`, `form`, `name`, `type`, `components` - [ ] **No Jinja2 control blocks in CRUD JSON** (`{% if %}`, `{% for %}`). `{{entire_url(...)}}` in strings is OK. See `references/crud-json-rules.md`. - [ ] File stored in correct `json/` directory with `{table_name}.json` naming - All referenced fields exist in table definition (`models/` directory) - Security fields properly configured (`confidential_fields` + `browserfields.exclouded`) - Subtable references are valid: `field` and `subtable` keys present, `subtable` value matches an existing table in `models/`, and a corresponding wwwroot directory or CRUD config exists for it - Every `data_url` / `editable` URL has a matching `.dspy` endpoint file in `wwwroot/api/` - If `data_filter` present, corresponding `.dspy` list endpoint uses `DBFilter` to parse it - **CRUD endpoint audit**: Run `scripts/verify-crud-endpoints.py` from `~/repos/` to check all create/update/delete `.dspy` files for json.dumps wrapping, wrong return format, and sor.U argument count. Fix all failures before commit. ## Common Pitfalls ### WRONG Format Patterns (DO NOT USE) The following patterns are **not** part of the CRUD spec and will cause failures: ```json // WRONG - these keys do not exist in the spec { "tablename": "...", // Should be "tblname" "grid": { // "grid" key does not exist "fields": [...], // Use browserfields.exclouded + alters instead "joins": [...], // CRUD files do NOT support SQL joins "select_fields": [...] // Cross-table fields are not allowed }, "form": { // "form" key does not exist "fields": [ { "widget": "text" } // Use editexclouded + alters with uitype instead ] } } ``` ### Pitfall 1: Root key is `tblname`, not `tablename` - **Wrong**: `"tablename": "users"` - **Correct**: `"tblname": "users"` ### Pitfall 2: CRUD files reference ONLY the base table CRUD definition files do NOT support SQL joins, select_fields, or cross-table field references. All fields referenced in `browserfields.exclouded`, `editexclouded`, and `alters` must exist in the table definition (`models/` directory) for the table specified in `tblname`. - **Wrong**: Referencing `contract_number` when `tblname` is `financial_vouchers` (that field is in the `contract` table) - **Correct**: Only reference fields that exist in `financial_vouchers` table definition (e.g., `contract_id`, `voucher_number`, `amount`) ### Pitfall 3: Dropdown fields use `alters` with `uitype: "code"` Dropdown/select fields must be defined in `browserfields.alters`, not in a `form` section: **Static data (inline options):** ```json "params": {"browserfields": {"alters": {"status": { "uitype": "code", "data": [{"value": "1", "text": "Active"}, {"value": "0", "text": "Inactive"}] }}}} ``` **Dynamic data (API endpoint):** ```json "params": {"browserfields": {"alters": {"providerid": { "uitype": "code", "dataurl": "{{entire_url('../api/get_organizations.dspy')}}" }}}} ``` - `dataurl` — API endpoint URL (must use `{{entire_url('...')}}` with quoted string) - The endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object - **Prefer appcodes over inline `data` arrays** — fixed options should go into `appcodes`/`appcodes_kv` via model `codes` definitions. See `references/appcodes-pattern.md` for the full pattern. - The endpoint **must return a plain JSON array** `[{value, text}, ...]` — no wrapping object - On error or empty data, return `[]` - `data_field` is **deprecated** — it was part of an older nested-response pattern (`{"data": {"organizations": [...]}}`). Do not use it. - `valueField` and `textField` are **NOT deprecated** — they are required when the data source returns keys other than `value`/`text`. See Pitfall 26 for details. **⚠️ Deprecated nested-response pattern (DO NOT USE):** ```json // WRONG — data_field is deprecated (nested response wrapping) "providerid": { "uitype": "code", "dataurl": "...", "data_field": "organizations" // DEPRECATED — do not use } ``` The endpoint must NOT return `{"success": true, "data": {"organizations": [...]}}`. Bricks Form's UiCode parses the response directly as an array. Wrapped formats cause the dropdown to silently not render. ### Pitfall 4: Field hiding uses `exclouded`/`editexclouded`, not field-level `"hidden": true` - **Wrong**: `{"name": "org_id", "widget": "hidden"}` - **Correct**: `"editexclouded": ["org_id"]` (hides in edit form), `"browserfields": {"exclouded": ["org_id"]}` (hides in list view) ### Pitfall 5: Always load this skill BEFORE creating/modifying CRUD files Never guess the CRUD format. Load `crud-definition-spec` first, then follow the structure exactly. The user has zero tolerance for guessed or improvised formats — they expect strict adherence to the spec with production-ready output, not experimental or guessed formats. If you're unsure about any property name or structure, load this skill and follow the examples verbatim. ### Pitfall 6: Confidential fields should be hidden in browser view Sensitive fields like API keys, passwords, or secret tokens should be listed in both `confidential_fields` and `browserfields.exclouded`. The `confidential_fields` array triggers server-side redaction, while `exclouded` removes them from the browser grid entirely. ### Pitfall 7: Every CRUD JSON file MUST have an `editable` paragraph Every CRUD JSON definition file — including list-only views — must include the `editable` paragraph with `new_data_url`, `update_data_url`, and `delete_data_url`. Without it, the framework cannot process form submissions. ```json "params": { "editable": { "new_data_url": "{{entire_url('../api/table_create.dspy')}}", "update_data_url": "{{entire_url('../api/table_update.dspy')}}", "delete_data_url": "{{entire_url('../api/table_delete.dspy')}}" } } ``` ### Pitfall 8: Field references must match model definitions exactly All field names in `browserfields.exclouded`, `browserfields.alters`, and `editexclouded` must exactly match field names defined in the table's model JSON (`models/` directory). Common mismatches: | Wrong Field | Correct Field | |-------------|---------------| | `org_id` (not in model) | Remove it | | `sales_stage` | `current_stage` | | `source` | `source_type` | | `is_active` | `is_won_stage` / `is_lost_stage` | | `changed_by` | `changed_by_id` / `changed_by_name` | ### Pitfall 9: Non-CRUD format files must not be placed in `json/` directory Files using custom structures like `{"name": "...", "title": "...", "type": "page", "components": [...]}` are NOT CRUD definitions and will cause framework failures. These should be `.ui` files in `wwwroot/`, not JSON files in `json/`. ### Pitfall 9: Validate ALL json/ files when touching any file in the directory The user has zero tolerance for non-CRUD files being left in or added to the `json/` directory. When the task involves any CRUD file modification, you MUST scan every `.json` file in that directory against this spec — not just the files directly involved in the task. ### Pitfall 10: ALL field references MUST exactly match model field names Every field name used in `browserfields.exclouded`, `browserfields.alters`, and `editexclouded` MUST be an exact match to a field defined in the table's model JSON (`models/` directory). Even a one-character difference will cause failures. - **Wrong**: `alters` key `"sales_stage"` when the model field is `"current_stage"` - **Wrong**: `exclouded` includes `"org_id"` when that field doesn't exist in the model - **Wrong**: `alters` key `"is_active"` when the model has `"is_won_stage"` and `"is_lost_stage"` but no `"is_active"` - **Correct**: Use only names that appear in the model's `fields` array ### Pitfall 11: ID values must use `appPublic.uniqueID.getID()`, not `uuid.uuid4()` Database `id` columns are typically VARCHAR(32). `uuid.uuid4().replace('-', '')` produces a 32-char hex string that may exceed the column length depending on the database encoding. Always use: ```python from appPublic.uniqueID import getID new_id = getID() ``` This applies to both `.dspy` API files and Python backend code. ### Pitfall 14: Hand-written `get_*_list.dspy` SHADOWS framework auto-generated list endpoints When a module has CRUD JSON definitions in `json/` (e.g., `rl_vendor_config_list.json`), the Sage CRUD framework **automatically generates** list endpoints. If you also have a hand-written `wwwroot/api/get_{table}_list.dspy`, it **shadows** (overrides) the framework-generated one. This causes: - **500 errors**: hand-written code may apply filters (e.g., `org_id`) on fields that don't exist in the table - **403 errors**: hand-written files bypass the framework's RBAC and `logined_userorgid` handling - **Silent data leakage**: hand-written code misses `confidential_fields` redaction **Rule**: If `json/{table}_list.json` exists, do NOT create `wwwroot/api/get_{table}_list.dspy`. The framework handles list queries from the JSON definition. Only create hand-written `.dspy` for custom business logic endpoints (create, update, delete, client-specific actions). **How to detect**: If a list endpoint returns 500 or 403, check: 1. Does `json/{alias}.json` exist for this table? 2. Does `wwwroot/api/get_{table}_list.dspy` also exist? 3. If both exist → DELETE the hand-written dspy, the framework auto-generates it ### Pitfall 15: `editable` section is required for ALL CRUD files (even list-only views) Every CRUD JSON file in the `json/` directory MUST have an `editable` paragraph with `new_data_url`, `update_data_url`, and `delete_data_url`. This is not optional — even files that only display lists need it, because the framework expects it for form submission handling. The URLs must use `{{entire_url('../api/xxx.dspy')}}` format with `../` prefix. ### Pitfall 13: `subtables[].subtable` must reference an existing table with accessible UI The `subtables` section references a related table's CRUD UI. The value of `subtable` must be a real table name that has: 1. A table definition file in `models/` directory 2. A corresponding wwwroot directory (or at minimum, the CRUD is properly configured) A phantom reference to a non-existent table (e.g., `"subtable": "llmtype"` where `llmtype` has no model or wwwroot) causes the UI to break silently — the subtable tab renders but shows no data or errors. **CRITICAL: Explicit `url` when default path lacks RBAC permissions.** The framework generates a default path like `/module/subtable_name` for subtable UIs. If that path is NOT registered in RBAC permissions, the subtab will return 401. Solution: add an explicit `url` in the subtables entry pointing to a `.ui` file that already exists in wwwroot: ```json "subtables": [ { "field": "llmid", "title": "能力映射", "url": "{{entire_url('./llm_api_map_manage.ui')}}", "subtable": "llm_api_map" } ] ``` This bypasses the framework's auto-generated path and uses a known-permitted route. Always check that the referenced `.ui` file actually exists before setting the URL. **How to verify**: 1. `find /path/to/module/wwwroot -type d` — check if a CRUD directory exists for the subtable 2. `ls /path/to/module/models/.json` — confirm table definition exists 3. `grep -r "subtable_name" /path/to/module/wwwroot` — check for existing `.ui` files 4. If no CRUD directory exists but a standalone `.ui` file does (e.g., `xxx_manage.ui`), use `"url": "{{entire_url('./xxx_manage.ui')}}"` instead of relying on auto-generated path 5. **Default**: always set `url` explicitly to a concrete `.ui` file rather than depending on framework path generation — this avoids RBAC 401 errors entirely ### Pitfall 19: NEVER replace CRUD auto-generated endpoints with custom scripts The CRUD framework automatically generates `get_{table}.dspy` endpoints from JSON definitions. These are **base framework functionality** — stable, tested, and maintained. Do NOT replace them with hand-written scripts unless there is a genuine special requirement. **Wrong approach**: Creating `wwwroot/api/llm_list.dspy` to replace `get_llm.dspy` just to add `_text` fields. **Correct approach**: Keep using the CRUD auto-generated `get_llm.dspy`, and fix the `dataurl` API to return `[{field_name, field_name_text}]` format instead. **Why this matters**: - Base framework endpoints handle RBAC, `logined_userorgid`, `confidential_fields` redaction, and DBFilter parsing correctly - Custom scripts often violate `.dspy` file conventions (imports, dict access patterns, etc.) - Replacing stable framework code with custom scripts creates maintenance burden — "today this way, tomorrow that way" makes the system unmaintainable **When to propose custom scripts**: Only when there is a genuine special requirement that the CRUD framework cannot handle. Even then, **discuss the approach first** before implementing — get confirmation that the deviation is necessary and the proposed solution is acceptable. ```bash python references/validate_crud.py /json/ --model-dir /models/ ``` It checks for forbidden keys, missing required keys, field references that don't exist in the model, and improper alters syntax. ### Systematic Module Audit When auditing an entire module's CRUD configs (e.g., "check all supplychain CRUD"), follow `references/crud-audit-procedure.md` — it has a step-by-step checklist for cross-referencing json/*.json against models/*.json, verifying alters coverage, editexclouded completeness, dataurl existence, and data_filter format correctness. ### Pitfall 16: Generated CRUD directories are NOT committed to git When `xls2ddl.xls2crud` generates `wwwroot//` directories (containing index.ui, get/add/update/delete .dspy), these are auto-generated artifacts. They must NOT be committed to git. Add them to the module's `.gitignore`: ``` # CRUD definition directories (auto-generated by Sage platform) wwwroot/llm/ wwwroot/llm_api_map/ # ... one entry per generated table ``` The `json/` CRUD definitions ARE committed (they are the source). The `wwwroot/
/` directories are regenerated from `json/` + `models/` via `xls2ddl.xls2crud`. See `references/xls2crud-generation.md` for the full generation workflow. ### Pitfall 17: Custom `data_url` in CRUD JSON requires xls2ddl template support If a CRUD JSON specifies `"data_url": "{{entire_url('../api/custom_list.dspy')}}"` to override the default `get_
.dspy`, the xls2ddl template (`data_browser_tmpl`) must use `{% if data_url %}` (not `{% if get_data_url %}`). As of xls2ddl commit 9f9a60a this is fixed. If you see generated index.ui ignoring `data_url`, update xls2ddl. ### Pitfall 18: Code-type (uitype: "code") field debugging — never blame bricks first **Core principle**: If a code-type field shows `undefined`, raw IDs, or wrong values, the problem is in the **data layer** (API response format, dataurl path, backend query) — NOT in bricks framework. All code-type inputs use the same bricks component; if it were a framework bug, ALL code fields would be affected, not just one specific field. **Symptom: `undefined` in filter dropdown options** - Check: does the `dataurl` endpoint return `[{value, text}]`? This is the ONLY correct format. Any other key names will fail. - Check: is the `dataurl` path correct? Relative paths like `../api/xxx.dspy` may resolve differently than expected. - Check: does the endpoint throw silently (returns `[]` on error)? Look at the dspy code for try/except that swallows errors. **Symptom: raw IDs in grid cells (not human-readable names)** The **dataurl API** (the endpoint specified in `alters[field].dataurl`) must return `[{value, text}]` format using SQL aliases: ```python # CORRECT - use SQL aliases for value/text orgs = await sor.sqlExe( "select id as value, orgname as text from organization order by orgname", {} ) return orgs # DictObject serializes correctly ``` The bricks framework uses `value` for the ID and `text` for display in both filter dropdowns and grid cells. **DO NOT** edit the auto-generated `wwwroot/
/get_
.dspy` directly — it's regenerated from templates. If the grid still shows raw IDs after fixing the `dataurl` API, the auto-generated list endpoint is not doing code resolution (it doesn't join reference tables). See Pitfall 46 for the custom list endpoint pattern with explicit LEFT JOINs. ### Pitfall 20: data_filter dropdown must have empty/default option — FIXED **Status**: Fixed in bricks commit `f8f02c6` (dataviewer.js `get_filter_fields()`). **What changed**: Bricks now auto-injects `{value: '', text: ''}` as first option for all code-type filter fields, unless data already contains an empty/null/undefined value entry. This covers both `data` (static) and `dataurl` (dynamic) sources. **Previously**: The `dataurl` endpoint had to manually add empty option. Now the framework handles it. No backend changes needed. **Symptom (if unfixed)**: User opens filter form, selects a value, cannot reset to "show all". ### Pitfall 22: codes fields need dedicated `get_search_{fieldname}.dspy` for filter dropdowns **Critical**: The success path MUST prepend the "全部" option to query results. Common mistake: returning `orgs` directly without the empty-value option. Correct pattern: ```python try: async with get_sor_context(request._run_ns, 'rbac') as sor: orgs = await sor.sqlExe( "select id as value, orgname as text from organization order by orgname", {} ) return json.dumps([{'value': '', 'text': '全部'}] + list(orgs), ensure_ascii=False) except Exception as e: debug(f'get_search_providerid error: {e}') return json.dumps([{'value': '', 'text': '全部'}], ensure_ascii=False) ``` When a model's `codes` section defines a foreign key (e.g., `providerid` → `organization`), the CRUD `browserfields.alters` dataurl should point to a **dedicated search script** named `get_search_{fieldname}.dspy`, not the generic list endpoint. **Pattern:** ```python # wwwroot/api/get_search_providerid.dspy result = [{'value': '', 'text': '全部'}] try: async with get_sor_context(request._run_ns, 'rbac') as sor: orgs = await sor.sqlExe( "select id as value, orgname as text from organization order by orgname", {} ) # CRITICAL: prepend 全部 to results — do NOT just "return orgs" return json.dumps([{'value': '', 'text': '全部'}] + list(orgs), ensure_ascii=False) except Exception as e: debug(f'get_search_providerid error: {e}') return json.dumps(result, ensure_ascii=False) ``` **Key rules:** - Return format: `[{value, text}]` — SQL aliases `id as value, name as text` - On success: **prepend 全部 to query results** via `[{'value': '', 'text': '全部'}] + list(orgs)` — do NOT just `return orgs` (this drops the 全部 option, a common mistake) - On error: return fallback `result` with only "全部" option - Name convention: `get_search_{fieldname}.dspy` in `wwwroot/api/` - Register in `load_path.py` - **CRITICAL: Also register in DB `permission` + `rolepermission` tables** — `load_path.py` alone is NOT sufficient for new `api/*.dspy` endpoints. After deploying, run SQL on the target server: ```sql INSERT INTO permission (id, path) VALUES (REPLACE(UUID(),'-',''), '/module/api/endpoint.dspy'); INSERT INTO rolepermission (id, roleid, permid) SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/endpoint.dspy'; ``` The path MUST use `/module/api/xxx.dspy` (not `/module/xxx.dspy`) — files in `wwwroot/api/` are served at `/module/api/`. Without this DB registration, the endpoint returns `403 Forbidden` even if `load_path.py` is correct. The role should be `'logined'` for endpoints that any authenticated user can call. - No imports allowed (json, get_sor_context, debug are pre-loaded) **Why dedicated scripts:** Generic list endpoints may serve multiple purposes with different formats. Search scripts are single-purpose and include the "全部" option as fallback. **Architecture note**: Edit-form dropdowns use `alters.dataurl`; filter dropdowns use model `codes` → `get_code.dspy`. Both must be updated when changing a field's data source. See `references/filter-vs-edit-dropdown.md`. ### Pitfall 24: NOT NULL DEFAULT columns MUST be in `editexclouded` ### Pitfall 38: `return json.dumps(result)` in hand-written dspy causes double-serialization — tree/list won't update **Symptom**: After adding/updating a record via CRUD form (especially tree view), the operation appears to succeed (no error), but the new/changed record does NOT appear in the UI. Refreshing the page shows the data IS in the database — it was inserted but the tree/list didn't refresh. **Root cause**: Hand-written create/update/delete `.dspy` files using `return json.dumps(result, ensure_ascii=False)` instead of `return result`. The dspy framework already JSON-serializes the return value. Double-serialization produces a JSON string instead of an object, which the bricks frontend cannot parse (expects `{widgettype: "Message", ...}`, receives `'{"widgettype": "Message", ...}'`). **Wrong**: ```python # WRONG — dspy framework auto-serializes; this double-serializes return json.dumps(result, ensure_ascii=False) ``` **Correct**: ```python # CORRECT — framework handles serialization return result ``` **Also check**: Remove `import json` at the top of hand-written dspy files — `json` is pre-loaded and importing it triggers dspy compliance warnings. **Audit command**: ```bash grep -rn "json.dumps" wwwroot/api/ --include='*.dspy' ``` **Applicability**: Hand-written `.dspy` files in `wwwroot/api/`. Auto-generated CRUD dspy files (in `wwwroot/
/`) use `return r` correctly via the template — only hand-written files have this issue. ### Pitfall 24 (original): NOT NULL DEFAULT columns MUST be in `editexclouded` When a table column has `NOT NULL DEFAULT ` (e.g., `login_fail_count SMALLINT NOT NULL DEFAULT 0`, `created_at TIMESTAMP NOT NULL DEFAULT current_timestamp()`), and the CRUD form should NOT let users edit it, it **MUST** be listed in `editexclouded`. If omitted, the edit form renders an empty input for that field, and on submit the framework sends `NULL` for it — causing `(1048, "Column 'xxx' cannot be null")` errors. **Symptom**: Adding a new record via CRUD form fails with `Column 'login_fail_count' cannot be null` even though the DB column has a DEFAULT value. **Fix**: Add ALL non-user-editable NOT NULL columns to `editexclouded`: ```json "editexclouded": ["id", "created_at", "login_fail_count", "last_login", "last_login_fail"] ``` **Common culprits**: timestamp columns with `DEFAULT current_timestamp()`, counter columns with `DEFAULT 0`, status columns with `DEFAULT '0'`. ### Pitfall 25: `record_toolbar` pattern for state-change action buttons CRUD tables support per-row action buttons via `record_toolbar` in `params`. Use this for enable/disable, approve/reject, activate/deactivate, or any state-change operation on individual records. **Structure:** ```json "params": { "record_toolbar": [ { "label": "启用", "actiontype": "dspy", "url": "/module/table/enable_record.dspy", "options": { "icon": "check", "cwidth": 16, "cheight": 9 } }, { "label": "禁用", "actiontype": "dspy", "url": "/module/table/disable_record.dspy", "options": { "icon": "block", "cwidth": 16, "cheight": 9 } } ] } ``` **dspy handler pattern** (enable_user.dspy / disable_user.dspy): ```python if not params_kw.get('id'): return {"widgettype":"Error","options":{"title":"Error","message":"no record selected","cwidth":16,"cheight":9,"timeout":3}} dbname = get_module_dbname('module_name') db = DBPools() async with db.sqlorContext(dbname) as sor: await sor.U('table_name', {'id': params_kw.id, 'status_field': 'new_value'}) return {"widgettype":"Message","options":{"title":"Success","message":"record updated","cwidth":16,"cheight":9,"timeout":3}} ``` **Key rules:** - Each button needs its own `.dspy` file in `wwwroot/` - The dspy receives `params_kw.id` (the selected row's ID) - Use `sor.U()` with a dict containing `id` + fields to update — do NOT pass a 3rd argument - Register all dspy paths in `load_path.py` - Use `cwidth`/`cheight` in options (not fixed px) **Global toolbar buttons (`toolbar.tools` + `binds`):** See `references/toolbar-tools-binds.md` for the top-level toolbar pattern — unlike `record_toolbar` (per-row), these sit above the list and require `selected_row: true` + a `binds` entry with `wid: "self"`. Covers urlwidget→PopupWindow, params_mapping, `${id}$` placeholder, DSPY return format, and the 403→load_path→restart flow. ### Pitfall 23: dspy files must NOT have import statements — use pre-loaded modules only **Every `.dspy` file is injected into a pre-built async function context.** The following are already imported and available without any `import` statement: **From ahserver `y_env` (processorResource.py):** - `debug`, `exception`, `error`, `info`, `warning`, `critical` (from appPublic.log) - `get_user`, `get_username`, `get_userorgid`, `get_userinfo` - `entire_url`, `i18n`, `redirect`, `clientinfo`, `terminalType` **Pre-loaded globals (injected at dspy compile time):** - `json` (json.dumps, json.loads) - `datetime` (datetime, date, timedelta) - `time` (time.time, time.sleep) - `DictObject` (from appPublic.dictObject) - `DBPools`, `get_sor_context` (from sqlor.dbpools) - `getID` (from appPublic.uniqueID) - `curDateString`, `timestampstr` (from appPublic.timeUtils) - `FileStorage` (from ahserver.filestorage) - `partial` (from functools) - `params_kw` (request params), `format_exc` (traceback) **Only allowed import**: `from sqlor.filter import DBFilter` (not pre-loaded). **Common violations:** ```python # WRONG — all of these are pre-loaded, NEVER import them import json import time import datetime from appPublic.uniqueID import getID from appPublic.log import debug from appPublic.dictObject import DictObject from appPublic.timeUtils import curDateString, timestampstr from sqlor.dbpools import get_sor_context, DBPools from functools import partial from ahserver.filestorage import FileStorage ``` **If a module-internal function is needed** (e.g. `from llmage.utils import get_llmusage_by_id`), export it via the module's `load_XXX()` function in `init.py` instead: ```python # llmage/init.py from .utils import get_llmusage_by_id def load_llmage(): env = ServerEnv() env.get_llmusage_by_id = get_llmusage_by_id ``` Then dspy files call it directly: `record = await get_llmusage_by_id(usage_id)`. **Audit command** (run before every commit touching .dspy files): ```bash grep -rn '^import \|^from ' wwwroot/ --include='*.dspy' | grep -v 'sqlor.filter' ``` Must return empty. Any match is a violation — delete the import line. ### Pitfall 19: Never manually add `data_url` to CRUD JSON — framework auto-generates endpoints ### Pitfall 37: Tabular edit sends `_text` suffix fields — MUST strip before sor.U/sor.C **Symptom**: Editing a row with code-type fields (e.g., `user_status`, `orgid`) saves successfully but the changed value does not persist. Or: the update silently fails with no error message. **Root cause**: Tabular's edit form collects ALL field data including `_text` suffix display columns (e.g., `user_status_text`, `orgid_text`, `sync_from_text`). These `_text` fields are NOT real DB columns. When `sor.U('table', ns)` receives them in the data dict, some sqlor implementations may fail silently or skip the update. **Fix**: In EVERY add and update `.dspy`, add cleanup after `ns = params_kw.copy()`: ```python ns = params_kw.copy() for k,v in ns.items(): if v == 'NaN' or v == 'null': ns[k] = None # remove _text suffix fields sent by Tabular (not real DB columns) for k in list(ns.keys()): if k.endswith('_text'): ns.pop(k, None) ``` **Affected files**: ALL `add_*.dspy` and `update_*.dspy` files generated by xls2crud. **Also fix in hand-written dspy files**: Any custom form submit handler that receives data from a Tabular or Form with code-type fields. **Detection**: Grep for `_text` in params_kw before sor.C/U calls. If `_text` fields are present and not stripped, the update will fail. ### Pitfall 47: Delegate-pattern endpoints (return result from helper) are valid — do NOT flag as format errors **Symptom**: A `_create.dspy` / `_update.dspy` / `_delete.dspy` file contains only a few lines that call a helper function and return its result: ```python result = await create_marketing(request, params_kw) return result ``` **Rule**: These are valid. The helper function (defined in `init.py` or elsewhere) is responsible for returning the correct widgettype format. Do NOT flag these as `no_widgettype` errors. Only flag endpoints that construct their own return value (e.g., `return {'success': True, ...}`) without widgettype. **How to detect**: If the last line is `return result` or `return json.loads(result)` and the file also has `result = await helper(...)`, it's a delegate pattern — skip format checking. ### Pitfall 48: Model name variants (dot vs hyphen) must be covered in model_mappings **Cross-skill**: This pitfall affects `pricing-data-format` as well. **Symptom**: Pricing engine logs `{config_data=..., mismatched}` and raises `没有找到合适的定价` even though the pricing YAML looks correct. **Root cause**: LLM API responses use dotted version numbers (e.g., `doubao-seedance-2.0`) while pricing YAML entries use hyphenated versions (e.g., `doubao-seedance-2-0`). The `model_mappings` in the YAML only maps full version suffixes (e.g., `doubao-seedance-2-0-260128` → `doubao-seedance-2-0`) but does NOT map dot-variants. **Fix**: Add explicit mappings for all dot-variants: ```yaml model_mappings: doubao-seedance-2.0: doubao-seedance-2-0 doubao-seedance-2.0-fast: doubao-seedance-2-0-fast doubao-seedance-2-0-260128: doubao-seedance-2-0 ``` **Detection**: Compare the `model` field in incoming usage data against the `model` filter values in pricing entries. If they differ only by `.` vs `-` in version numbers, add the mapping. **Proper fix workflow (do NOT edit generated files):** Since generated CRUD dspy files must never be edited directly (Pitfall 28), the correct approach is: 1. Create custom dspy files in `wwwroot/api/` (e.g., `add_user.dspy`, `update_user.dspy`) 2. Copy the generated dspy logic, add `_text` cleanup + any other fixes 3. Point the CRUD JSON to the custom files: ```json { "tblname": "users", "params": { "new_data_url": "{{entire_url('/module/api/add_user.dspy')}}", "update_data_url": "{{entire_url('/module/api/update_user.dspy')}}", ... } } ``` 4. Register the new paths in `sage/load_path.py` 5. Regenerate: `xls2crud -m models -o wwwroot json/
.json` The generated `index.ui` will pick up the custom URLs automatically. The generated dspy files remain untouched and can be regenerated at any time without losing fixes. ### Pitfall 36: Tabular sends request params via `data_params`, NOT `params` **Symptom**: Tabular loads data but request parameters (e.g., `discountid`, parent record ID) are not sent to the data_url endpoint. The dspy receives no params and returns empty data. **Root cause**: bricks `DataViewer` (line 8-14) reads `this.opts.data_params` as the default request parameters — NOT `this.opts.params`: ```javascript this.loader = new bricks.PageDataLoader({ url:this.opts.data_url || this.opts.url, params:this.opts.data_params, // ← data_params, not params ... }); ``` **Wrong**: ```json "data_url": "...dspy", "params": { "discountid": "{{params_kw.discountid}}" } ``` **Correct**: ```json "data_url": "...dspy", "data_params": { "discountid": "{{params_kw.discountid}}" } ``` This applies to ALL hand-written Tabular widgets, including subtable pages. The CRUD auto-generated files handle this correctly via the template — only hand-written Tabular UIs need this fix. ### Pitfall 19 (original): Never manually add `data_url` to CRUD JSON — framework auto-generates endpoints The CRUD framework automatically generates `get_{table}.dspy` endpoints from JSON definitions in the `json/` directory. Do NOT add `"data_url": "{{entire_url('../api/get_{table}.dspy')}}"` to the CRUD JSON — this overrides the auto-generated path and points to a non-existent file, causing 500 errors. **Wrong:** ```json { "tblname": "llm", "params": { "data_url": "{{entire_url('../api/get_llm.dspy')}}", // WRONG — file doesn't exist ... } } ``` **Correct:** Omit `data_url` entirely. The framework uses the auto-generated `wwwroot/{table}/get_{table}.dspy`. **Exception:** Only add `data_url` when you have a genuinely custom list endpoint with special logic that the framework cannot handle. Even then, ensure the target `.dspy` file actually exists. **Related:** If you need to add `_text` fields for foreign key display, create a `get_search_{fieldname}.dspy` endpoint (see Pitfall 22) and reference it in `alters[field].dataurl`. Do NOT try to modify the auto-generated list endpoint. ### Pitfall 21: data_filter conflicts with logined_userorgid — FIXED in xls2ddl **Status**: Fixed in xls2ddl commit `ebd4b4a` (tmplspy `get_data_tmpl`). **What changed**: The generated `get_
.dspy` now injects `logined_userorgid`/`logined_userid` conditions into the `filterjson` object before DBFilter processes it. This ensures both work together correctly. **Fix mechanism** (in template): ```python # After default_filterjson fallback, before DBFilter: {% if logined_userorgid or logined_userid %} if filterjson: if not isinstance(filterjson, dict) or 'AND' not in filterjson: filterjson = {'AND': [filterjson] if filterjson else []} {% if logined_userorgid %} filterjson['AND'].append({'field': '{{logined_userorgid}}', 'op': '=', 'var': '__logined_orgid__'}) ns['__logined_orgid__'] = userorgid {% endif %} {% if logined_userid %} filterjson['AND'].append({'field': '{{logined_userid}}', 'op': '=', 'var': '__logined_uid__'}) ns['__logined_uid__'] = userid {% endif %} {% endif %} ``` **Deployment**: After updating xls2ddl, regenerate affected modules: ```bash cd && PYTHONPATH= python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` **Previously**: Workaround was to manually add ownerid filter in dspy or use const in data_filter. No longer needed. ### Pitfall 49: Subtables auto-population — use `field` + `mapping` to pass parent values When a subtable's add form needs the parent record's ID auto-populated (e.g., supplier contract needs `supplier_id`), the subtable definition's `field` already handles filtering. For the add form to receive the value: 1. **Add the field to `editexclouded`** in the subtable's CRUD JSON — so users don't see/manually edit it 2. **Modify `new_data_url`** to carry the value: `new_data_url: "...dspy?supplier_id={{params_kw.get('supplier_id','')}}"` The xls2crud `params_mapping.mapping` sends `parent.id → subtable.field` to the subtable page as URL params. The add form's `new_data_url` then carries it to the dspy. Example: ```json // Parent CRUD JSON (suppliers_list.json) "subtables": [{ "field": "supplier_id", // parent's id → subtable's supplier_id "title": "供应商合同", "url": "{{entire_url('../supply_contracts_list')}}", "subtable": "supply_contracts" }] // Subtable CRUD JSON (supply_contracts_list.json) "editexclouded": ["id", "resellerid", "supplier_id", ...], "editable": { "new_data_url": "{{entire_url('../api/create.dspy')}}?supplier_id={{params_kw.get('supplier_id','')}}" } ``` ### Pitfall 40: xls2ddl `json.dumps(true)` generates Python-invalid `true` — MUST wrap in `json.loads()` **Symptom**: Generated `.dspy` files fail with `NameError: name 'true' is not defined. Did you mean: 'True'?`. The auto-generated `update_
.dspy` contains `"not_null": true` (JSON boolean) instead of `"not_null": True` (Python). **Root cause**: xls2ddl templates use `{{json.dumps(fields, ensure_ascii=False)}}` to inline Python data structures. JSON's `true`/`false` are valid JSON but not valid Python literals. When the generated code runs in Python context, `NameError` occurs. **Fix in xls2ddl `tmpls.py`** — wrap Python-context `json.dumps()` in `json.loads()` to convert JSON booleans back to Python. **CRITICAL: The `json.loads()` call MUST have single quotes around the Jinja2 expression.** Jinja2's `{{json.dumps(...)}}` embeds the result as Python literal (not a string), so without quotes `json.loads()` receives a dict and crashes with `TypeError: the JSON object must be str, bytes or bytearray, not dict`: ```python # BEFORE (broken — JSON booleans in Python code): tblfields = {{json.dumps(fields, ensure_ascii=False)}} filterjson = {{json.dumps(data_filter, ensure_ascii=False)}} # ALSO BROKEN — json.loads receives dict, not string: tblfields = json.loads({{json.dumps(fields, ensure_ascii=False)}}) # CORRECT — quotes make it a string, json.loads parses true→True: tblfields = json.loads('{{json.dumps(fields, ensure_ascii=False)}}') filterjson = json.loads('{{json.dumps(data_filter, ensure_ascii=False)}}') ns['sort'] = json.loads('{{json.dumps(sortby)}}') ``` **Where to apply**: Only Python-context lines (tblfields, filterjson, sort arrays). JSON-context lines (browserfields, toolbar, binds) use `true`/`false` correctly and must NOT be wrapped. **Commit reference**: xls2ddl `af6006f`. **Symptom**: After adding a record via CRUD-generated `new_*.dspy` / `add_*.dspy`, the operation fails with: ``` OperationalError: (1054, "Unknown column 'None' in 'WHERE'") markedSQL='SELECT * FROM
WHERE None = %s' ``` The error occurs in `sor.sqlExe()` inside the generated dspy, on the post-insert re-query line. Affects ALL tables across ALL modules. **Root cause**: In xls2ddl `tmpls.py` `data_new_tmpl`, commit `fc91486` added a post-insert re-query: ```python _new_rows = await sor.sqlExe("SELECT * FROM {{summary[0].name}} WHERE {{summary[0].pkey}} = ${id}$", {'id': id}) ``` But model JSONs use `primary` (array), not `pkey`. Jinja2 resolves to Python `None`, rendering `${None}$` → `WHERE None = %s`. **Fix**: `{{summary[0].pkey}}` → `{{summary[0].primary[0]}}` in xls2ddl `tmpls.py`. Then regenerate all modules. ### Pitfall 36: logined_userorgid generates `WHERE None = %s` — xls2ddl template bug **Symptom**: `OperationalError: (1054, "Unknown column 'None' in 'WHERE'")` with `SELECT * FROM table WHERE None = %s`. The auto-generated `get_*.dspy` substitutes Python `None` for the `logined_userorgid` field name when the user's orgid is not set. **Fix**: Update xls2ddl to commit `ebd4b4a` or later (fixes `get_data_tmpl` to properly inject `logined_userorgid`/`logined_userid` conditions into `filterjson`). Then regenerate all affected CRUD files: ```bash cd ~/repos/xls2ddl && git pull cd ~/repos/ && PYTHONPATH=~/repos/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/*.json ``` **Note**: Affects ALL modules using `logined_userorgid` or `logined_userid` in their CRUD JSON. Must regenerate every affected module. **Symptom**: Data API correctly returns `{fieldname}_text` columns (e.g., `llmid_text`, `userid_text`), CRUD alters has `valueField`/`textField` configured, but the list grid still shows raw IDs instead of human-readable names. **Root cause**: The `alters` entry is missing `"uitype": "code"`. Without it, the Tabular/DataViewer treats the field as plain text and ignores the textField mapping entirely. **Wrong**: ```json "llmid": { "valueField": "llmid", "textField": "llmid_text" } ``` **Correct**: ```json "llmid": { "uitype": "code", "valueField": "llmid", "textField": "llmid_text" } ``` **Rule**: Whenever you use `valueField`/`textField` in `alters`, you MUST also include `"uitype": "code"`. The framework only activates the textField lookup for code-type fields. Without `uitype`, it renders the raw field value (the ID) and silently discards the text mapping. **Related**: This applies to both hand-written `.ui` files and CRUD JSON `browserfields.alters`. See Pitfall 26 for the full valueField/textField pattern. ### Pitfall 26: `valueField`/`textField` in alters apply to BOTH filter form AND edit form CRUD JSON `browserfields.alters` is a **single configuration** shared by the list grid, the filter/search form, and the add/edit form. When you set `valueField`/`textField` on a code-type field, it affects ALL three contexts. **When to use valueField/textField:** - The `get_search_*.dspy` returns `{fieldname, fieldname_text}` keys (not `{value, text}`) - The main list endpoint also returns `{fieldname}_text` columns for grid display - You need the stored value key to be the actual field name (e.g., `providerid`) rather than generic `value` **Example — code field with custom valueField/textField:** ```json "alters": { "providerid": { "uitype": "code", "dataurl": "{{entire_url('../api/get_search_providerid.dspy')}}", "valueField": "providerid", "textField": "providerid_text" } } ``` The `get_search_providerid.dspy` MUST return data with those exact keys: ```python # CORRECT — keys match valueField/textField rows = await sor.sqlExe( "select id as providerid, orgname as providerid_text from organization order by orgname", {} ) return json.dumps([{'providerid': '', 'providerid_text': '全部'}] + list(rows), ensure_ascii=False) ``` **Key rules:** - `valueField`/`textField` must exactly match the keys returned by the `dataurl` endpoint - The same `valueField`/`textField` is used in the filter form AND the edit form — they cannot differ - If the dspy returns `{value, text}` (the simple pattern), do NOT set valueField/textField — the framework defaults to those - The "全部" fallback option must also use the same keys (not `{value: '', text: '全部'}` when valueField is `providerid`) **Symptom if wrong:** Filter dropdown shows `undefined` or raw IDs; edit form sends wrong values; filter selection doesn't match stored data. ### Pitfall 27: MUST regenerate index.ui on each server after modifying CRUD JSON alters The `wwwroot/
/` directory (containing `index.ui`) is auto-generated by `xls2crud` and is **gitignored** (see Pitfall 16). When you modify CRUD JSON `alters` (adding/changing `valueField`, `textField`, `dataurl`, `uitype`, etc.), the change only takes effect after re-running `xls2crud` on **each deployment environment**. **Common mistake:** Update JSON, commit and push, then `git pull` on the server — but `index.ui` is NOT in git, so the server still has the stale auto-generated UI. **Fix — run on each server after pull:** ```bash cd /path/to/module PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` **Triggers that require regeneration:** - Any change to `browserfields.alters` (dataurl, valueField, textField, uitype, data) - Any change to `browserfields.exclouded` or `editexclouded` - Any change to `data_filter`, `filter_labels`, `filter_title` - Any change to `subtables` - Any change to `record_toolbar` - Any change to `editor.binds` - Any change to model definitions in `models/` directory ### Pitfall 30: CRUD page not triggering data request — diagnose CRUD spec FIRST, not permissions **User correction**: When a CRUD list page loads but does NOT trigger the `get_
.dspy` data request, the root cause is in the CRUD specification or generated template — NOT in RBAC/permissions. Do NOT waste time checking `load_path.py` PATHS_ANY vs PATHS_LOGINED first. **Correct diagnostic order:** 1. **Check generated `index.ui`** — does it have a `data_url` field pointing to the correct endpoint? 2. **Check CRUD JSON** — does it have the required structure (tblname, params, editable)? 3. **Check `get_
.dspy`** — does the file exist in the CRUD directory? 4. **Compare with a working module** — diff against llmage/llm or pricing which are known to work 5. **Only then** check RBAC/load_path.py if all above are correct **Common causes (in order of frequency):** - CRUD directory not regenerated after JSON changes (stale index.ui) - `data_url` missing or incorrect in generated index.ui - CRUD JSON missing required keys (editable, browserfields) - Template version outdated (xls2ddl needs update) **Anti-pattern (DO NOT):** - Immediately assume it's a permissions issue - Move paths between PATHS_ANY and PATHS_LOGINED as first diagnostic step - Blame the bricks framework before checking the CRUD config ### Pitfall 31: `data_url` vs `get_data_url` in generated index.ui — two different mechanisms The xls2crud template (`data_browser_tmpl` in `tmpls.py`) generates TWO data-loading URLs with different purposes: 1. **`data_url`** (outside `editable` block) — used by the Tabular/DataViewer component for **initial page load**. Points to `get_
.dspy` by default: ``` data_url: "{{entire_url('./get_suppliers.dspy')}}" ``` - If CRUD JSON has a `data_url` key, that value is used instead (custom endpoint) - If CRUD JSON omits `data_url`, the template defaults to `./get_
.dspy` 2. **`get_data_url`** (inside `editable` block, optional) — used to **override** the data URL with extra parameters (e.g., `?pagerows=50`). Only generated when the CRUD JSON explicitly defines it: ```json "editable": { "get_data_url": "{{entire_url('get_llmusage.dspy')}}?pagerows=50", "new_data_url": "...", "update_data_url": "...", "delete_data_url": "..." } ``` **Key rules:** - Most CRUD pages only need `data_url` (auto-generated) — no `get_data_url` needed - `get_data_url` is for special cases (pagination overrides, custom query params) - The Tabular component uses `data_url` on initial render, then `get_data_url` for subsequent refreshes if defined - If neither exists in the generated index.ui, the table renders empty with no network request ### Pitfall 29: `data_filter` MUST use DBFilter tree structure, NOT `{"fields": [...]}` A common mistake is using a flat `fields` array instead of the required `AND`/`OR` tree: ```json // WRONG — this format does NOT work, search will silently fail "data_filter": { "fields": [ {"field": "supplier_org_id", "title": "供应商", "uitype": "code"}, {"field": "resource_type", "title": "资源类型", "uitype": "code"} ] } // CORRECT — DBFilter tree with op/var "data_filter": { "AND": [ {"field": "supplier_org_id", "op": "=", "var": "supplier_org_id"}, {"field": "resource_type", "op": "=", "var": "resource_type"} ] } ``` **Symptom**: Search button renders but filtering does nothing, or search popup doesn't appear at all. **Fix**: Replace `"fields": [...]` with `"AND": [{"field": "...", "op": "...", "var": "..."}]` and add a `"filter_labels"` object for Chinese labels. ### Pitfall 28: NEVER directly edit auto-generated .ui files — always modify source configs **Critical architectural rule**: Files in `wwwroot/
/` (especially `index.ui`) are auto-generated by `xls2crud` from source configurations. **Never edit them directly**, even in production environments. **Wrong approach** (causes user frustration): ```bash # WRONG — directly editing production file sudo sed -i 's/old_pattern/new_pattern/g' /d/apitest/sage/wwwroot/discount/discount_setting/index.ui ``` **Correct approach**: 1. Find the source configuration: - For CRUD-generated UIs: `json/
.json` (CRUD config) - For hand-written UIs: `wwwroot/.ui` (directly editable) 2. Modify the source config 3. Regenerate (if CRUD): ```bash cd /path/to/module PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` 4. Deploy the regenerated files **How to determine if a file is auto-generated:** - Check if `json/
.json` exists → yes = auto-generated - Check `wwwroot/
/` directory structure (has `index.ui`, `get_
.dspy`, etc.) → yes = auto-generated - Files outside `wwwroot/
/` directories (e.g., `wwwroot/custom_feature.ui`) → hand-written, can edit directly ### Pitfall 34: CRUD list performance — FOUR critical optimizations required **Symptom**: CRUD list page loads extremely slowly, queries take 3+ seconds, users complain about unacceptable performance even after initial fixes. **CRITICAL: Correct file location for custom list endpoints** The auto-generated `wwwroot/
/get_
.dspy` is **gitignored** and regenerated by `xls2crud`. DO NOT edit it — your changes will be lost on next regeneration. **Correct approach**: Write a custom list endpoint in `wwwroot/api/
_list.dspy` and point the CRUD JSON's `editable.get_data_url` to it: ```json // json/
.json "params": { "editable": { "get_data_url": "{{entire_url('../api/
_list.dspy')}}?pagerows=50", "new_data_url": "...", "update_data_url": "...", "delete_data_url": "..." } } ``` This custom endpoint is committed to git and persists across regenerations. **Root causes (in order of impact):** **1. `sqlPaging` wraps queries in subqueries (slowest)** The `sor.sqlPaging(sql, ns)` function wraps the data query in a subquery for count, which is extremely slow for large tables: ```python # SLOW — sqlPaging generates: SELECT * FROM (SELECT ... WHERE ...) AS t r = await sor.sqlPaging(sql, ns) ``` **Fast pattern — separate count and data queries:** ```python # Separate count query count_sql = f'select count(*) as cnt from tablename {where_clause}' count_recs = await sor.sqlExe(count_sql, ns) total = count_recs[0].cnt if count_recs else 0 # Separate data query with LIMIT/OFFSET page = int(ns.get('page', 1)) rows_per_page = int(ns.get('rows', ns.get('pagerows', 50))) offset = (page - 1) * rows_per_page data_sql = f'''select col1, col2, ... from tablename {where_clause} order by {ns.get('sort', 'use_time desc')} limit {rows_per_page} offset {offset}''' rows = await sor.sqlExe(data_sql, ns) return {'total': total, 'rows': rows if rows else []} ``` **2. `default_filterjson` generates LIKE filters for TEXT fields (full table scan)** When no filter is provided, `default_filterjson(fields, ns)` generates LIKE conditions for ALL fields, including large TEXT columns: ```python # SLOW — generates LIKE filters for TEXT fields filterjson = default_filterjson(fields, ns) ``` **Fast pattern — exclude TEXT fields from filter generation:** ```python filter_fields = [f['name'] for f in ori_fields if f['name'] not in ('usages', 'ioinfo')] filterjson = default_filterjson(filter_fields, ns) ``` **3. SELECT * includes large TEXT/BLOB columns (I/O overhead)** ```python # SLOW — fetches 100+ KB per row from off-page storage sql = "SELECT * FROM tablename WHERE ..." # FAST — explicit column list sql = '''select id, col1, col2, ... from tablename where 1=1''' ``` **4. DBFilter + ArgsConvert framework overhead (bypass entirely for max performance)** When the above three optimizations still leave the query slow (2+ seconds), completely bypass the DBFilter/ArgsConvert framework and write raw SQL: ```python # In wwwroot/api/
_list.dspy — no imports needed result = {'success': False, 'rows': [], 'total': 0, 'page': 1, 'page_size': 50} try: page = int(params_kw.get('page', 1)) rows_per_page = int(params_kw.get('rows', params_kw.get('pagerows', 50))) offset = (page - 1) * rows_per_page sort_field = params_kw.get('sort', 'use_time desc') # Manually build WHERE conditions (only common filter fields) conditions = ['1=1'] ns = {} llmid = params_kw.get('llmid') if llmid: conditions.append('llmid=${llmid}$') ns['llmid'] = llmid status = params_kw.get('status') if status: conditions.append('status=${status}$') ns['status'] = status # ... add other common filter fields as needed ... where = 'WHERE ' + ' AND '.join(conditions) # List fields (exclude large TEXT columns) select_fields = 'id, col1, col2, ...' db = DBPools() dbname = get_module_dbname('module_name') async with db.sqlorContext(dbname) as sor: count_recs = await sor.sqlExe(f'SELECT count(*) as cnt FROM tablename {where}', ns) total = count_recs[0].cnt if count_recs else 0 rows = await sor.sqlExe( f'SELECT {select_fields} FROM tablename {where} ORDER BY {sort_field} LIMIT {rows_per_page} OFFSET {offset}', ns ) result['success'] = True result['total'] = total result['rows'] = rows if rows else [] result['page'] = page result['page_size'] = rows_per_page except Exception as e: debug(f'
_list error: {format_exc()}') result['error'] = str(e) return json.dumps(result, ensure_ascii=False, default=str) ``` **Why bypassing DBFilter matters:** - DBFilter generates filter conditions for ALL fields in `ori_fields`, even when the user didn't provide values - ArgsConvert template processing adds overhead for every request - Manual WHERE clause building with only 5-7 common filter fields is 3-5x faster than full framework processing **Performance comparison (real-world example — llmage/llmusage, 17 columns):** - Original: `sqlPaging` + `default_filterjson` on all fields + `SELECT *` → 8-12 seconds - After fix #1 (separate queries): 3-4 seconds - After fix #2 (exclude TEXT from filter): 1-2 seconds - After fix #3 (explicit columns): 0.8-1.5 seconds - After fix #4 (bypass DBFilter entirely): 0.2-0.5 seconds - Total: 20-40x improvement **How to identify the problem:** 1. Check if `get_
.dspy` uses `sqlPaging` → replace with separate queries 2. Check if `default_filterjson` includes TEXT/BLOB fields → exclude them 3. Check if query uses `SELECT *` → replace with explicit column list 4. If still slow (>1s), bypass DBFilter entirely in custom `api/
_list.dspy` **Rule**: When a CRUD list query is slow, apply optimizations in order: (1) separate count/data queries, (2) exclude TEXT fields from filter, (3) explicit column list, (4) bypass DBFilter entirely. Each provides 2-5x improvement; together they provide 20-40x improvement. ### Pitfall 33: `"editable": "default"` string causes xls2ui serialization error **Symptom**: Running `build.sh` (which calls `xls2ui`) fails with: ``` TypeError: Object of type builtin_function_or_method is not JSON serializable ``` at `json.dumps(binds, indent=4, ensure_ascii=False)`. **Root cause**: The CRUD JSON has `"editable": "default"` as a **string** instead of an object. The framework internally tries to generate binds configuration from this shorthand, and some fields end up being set to function references instead of strings, causing JSON serialization to fail. **Wrong**: ```json { "tblname": "pricing_program", "params": { "editable": "default", // WRONG — causes serialization error "sortby": "name" } } ``` **Correct**: ```json { "tblname": "pricing_program", "params": { "editable": { "get_data_url": "{{entire_url('get_pricing_program.dspy')}}", "new_data_url": "{{entire_url('../api/pricing_program_create.dspy')}}", "update_data_url": "{{entire_url('../api/pricing_program_update.dspy')}}", "delete_data_url": "{{entire_url('../api/pricing_program_delete.dspy')}}" }, "sortby": "name" } } ``` **Fix**: Replace `"editable": "default"` with the full object containing all four URL keys (`get_data_url`, `new_data_url`, `update_data_url`, `delete_data_url`). The `"default"` shorthand is not supported by the framework. **Related**: This is a specific case of Pitfall 7 and Pitfall 15 (editable section required). The difference is that those pitfalls focus on the *presence* of editable; this pitfall focuses on the *format* — it must be an object, not a string. ### Pitfall 32: Field title renaming in model JSON affects ALL CRUD views When the user asks to rename field display titles (e.g., "用户id" → "username", "模型机构" → "orgname"), modify the `title` field in `models/
.json`, NOT in the CRUD JSON or generated .ui files. **Why**: The model's `fields[].title` is the single source of truth for column headers. CRUD auto-generation reads from the model and propagates titles to: - List grid column headers - Filter form labels - Edit form field labels - Add form field labels **Correct approach**: ```json // models/llmusage.json { "name": "userid", "title": "username", // Changed from "用户id" "type": "str", "length": 32 } ``` **Wrong approaches**: - Editing `json/
.json` browserfields — titles there are for overrides, not primary labels - Editing generated `wwwroot/
/index.ui` — this is a build artifact - Editing `wwwroot/api/get_
.dspy` — titles are not in the query layer **After renaming**: Regenerate CRUD files if needed: ```bash cd /path/to/module PYTHONPATH=/path/to/xls2ddl python3 -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` **Real-world example** (llmage/llmusage, 2026-06-25): - userorgid: "用户机构" → "orgname" - ownerid: "模型机构" → "orgname" - userid: "用户id" → "username" - llmid: "模型id" → "model" - Commit: `13c123c` ### Pitfall 43: `logined_userorgid` and `logined_userid` go in `params`, NOT `browserfields` **Symptom**: After adding `logined_userorgid` to a CRUD JSON, `xls2ui` crashes with `JSONDecodeError`, or the field appears in the wrong location in generated files. **Wrong** (inside `browserfields`): ```json { "tblname": "payment_log", "params": { "browserfields": { "exclouded": ["id"], "logined_userorgid": "customerid" // WRONG — should be at params level } } } ``` **Correct** (at `params` level, sibling to `browserfields`): ```json { "tblname": "payment_log", "params": { "logined_userorgid": "customerid", "browserfields": { "exclouded": ["id"] } } } ``` `logined_userorgid` and `logined_userid` are first-class `params` keys alongside `sortby`, `data_filter`, `editable` — NOT `browserfields` children. ### Pitfall 45: `editable.new_data_url` in `params.editable` was NEVER read by xls2ddl template — fixed in fb613d0 **Symptom**: CRUD JSON has `"params": {"editable": {"new_data_url": "{{entire_url('../api/custom_create.dspy')}}"}}` but the generated `index.ui` still uses the default `add_
.dspy` URL. The custom create dspy is never called. **Root cause**: The xls2ddl template (`data_browser_tmpl` in `tmpls.py`) checks `{% if new_data_url %}` at the TOP LEVEL of the template context. But after `desc.update(crud_data.params.copy())`, `new_data_url` is nested inside `desc.editable` (a dict), NOT at `desc.new_data_url`. Jinja2 resolves `new_data_url` → `desc.new_data_url` → `None` (via `DictObject.__getattr__` which returns None for missing keys), so the `{% else %}` branch always runs. **Fix**: Update xls2ddl to commit `fb613d0` or later — template now checks `{% if (editable and editable.new_data_url) or new_data_url %}`, falling back through nested editable, then top-level, then default. Same for `delete_data_url` and `update_data_url`. **Why this was masked**: In practice, the auto-generated `add_
.dspy` files existed on servers from BEFORE they were gitignored. So even though the template used the default URL, the file was present and the add flow worked. After gitignoring the generated directories, the default dspy files were no longer deployed → the old bug surfaced. **Regeneration required**: ```bash cd ~/repos/xls2ddl && git pull cd ~/repos/ ~/repos/xls2ddl/py3/bin/python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` ### Pitfall 44: filter_fields generated without inline data from browserfields.alters **Symptom**: CRUD list page loads but the search/filter form's code-type dropdowns (e.g., `is_external`, `status`) crash with `TypeError: Cannot read properties of undefined (reading 'length')` at `bricks.UiCode.build_options`. The filter popup doesn't appear at all. **Root cause**: xls2ddl's `build_filter_field_list()` only merges `data` from model-level `codes` definitions, NOT from `browserfields.alters` inline `data` arrays. Fields like `is_external` with `{"uitype":"code","data":[{"value":"1","text":"外部供应商"}]}` in alters — but no corresponding model `codes` entry — get a `uitype:"code"` header in filter_fields with `data: undefined`. **Fix**: Update xls2ddl to commit `faa571f` or later (merges alters `data`/`dataurl`/`valueField`/`textField` into filter_fields, and applies alters `uitype` override). **Regeneration required**: ```bash cd ~/repos/xls2ddl && git pull cd ~/repos/ ~/repos/xls2ddl/py3/bin/python -m xls2ddl.xls2crud -m models -o wwwroot json/
.json ``` **Affected**: All CRUD pages where `data_filter` includes fields that have inline `data` only in `browserfields.alters` (no model `codes` entry). ### Pitfall 42: CRUD JSON files must be PURE JSON — NO Jinja2 control-flow blocks **Hard rule**: CRUD JSON files in `json/` are consumed by `xls2ddl.xls2crud`, which parses them as pure JSON. Jinja2 control-flow blocks (`{% if %}`, `{% for %}`, `{% endif %}`) cause `JSONDecodeError` at parse time. **Wrong** (causes `json.decoder.JSONDecodeError` at xls2ui runtime): ```json { "binds": [{ "popup_options": { {% if params_kw._is_mobile %} "width": "100%", {% else %} "width": "40%", {% endif %} "archor": "cc" } }] } ``` **Correct** — use fixed values: ```json { "binds": [{ "popup_options": { "width": "40%", "height": "80%", "archor": "cc" } }] } ``` `{{entire_url('...')}}` placeholders inside JSON string values are acceptable — they're handled by the framework during URL generation. But `{% %}` Jinja2 control flow is never allowed in CRUD JSON files. This is distinct from `.ui` template files which DO support Jinja2. ### Pitfall 50: Custom list endpoint for code resolution — LEFT JOIN reference tables for `_text` grid display **Symptom**: Form/filter dropdowns show orgname correctly (via `uitype: "code"` + `dataurl`), but the list grid still shows raw IDs. The `dataurl` endpoint returns correct `[{value, text}]` / `[{fieldname, fieldname_text}]`. **Root cause**: The auto-generated `get_
.dspy` queries only the base table (`SELECT * FROM table WHERE ...`). It does NOT join reference tables to produce `_text` columns for grid cells. `uitype: "code"` + `dataurl` only controls dropdown rendering in forms/filters — it does NOT extend to grid cell display. **Inline `data` arrays for status/type fields also do NOT auto-resolve for list grid cells.** **Why Pitfall 18 and Pitfall 19 don't cover this**: Pitfall 18 says "DO NOT...the CRUD auto-generated list handles everything" — it doesn't. The auto-generated list never LEFT JOINs lookup tables. Pitfall 19 says "Never manually add data_url" — but custom list endpoints are the only way to get code resolution in grid cells. **Fix — TWO approaches (choose one):** **Approach A — COALESCE (recommended, more reliable):** Replace raw IDs with display names directly in SQL output. Guarantees grid renders names regardless of bricks `_text` detection behavior. ```python sql = '''select a.id, a.domain, COALESCE(b.orgname, a.resellerid) as resellerid, COALESCE(c.orgname, a.orgid) as orgid, CASE a.status WHEN 'active' THEN '启用' WHEN 'inactive' THEN '停用' ELSE a.status END as status, a.created_at, a.updated_at from (select * from tenant_domain where 1=1 [[filterstr]]) a left join (select id, orgname from organization) b on a.resellerid = b.id left join (select id, orgname from organization) c on a.orgid = c.id''' ``` Pros: Guaranteed. Filter WHERE applies on inner query with raw IDs before COALESCE. Cons: Edit form must use separate `_get.dspy` returning raw IDs. **Approach B — `_text` suffix:** Return extra `resellerid_text`, `orgid_text` columns via LEFT JOIN. Bricks Tabular MAY auto-detect `_text` columns — but detection is unreliable (in our 2026-07-10 testing, did NOT render). If it doesn't work, fall back to Approach A. **Placement — TWO valid locations:** 1. `wwwroot/{alias}/get_{alias}.dspy` — CRUD framework auto-discovers. No `get_data_url` needed. BUT this directory may be gitignored (use `git add -f`). 2. `wwwroot/api/
_list.dspy` + `editable.get_data_url` — requires xls2ddl template support (Pitfall 45). **RBAC registration (required):** ```sql INSERT INTO permission (id,path) VALUES (REPLACE(UUID(),'-',''),'/module/api/.dspy'); INSERT INTO rolepermission (id,roleid,permid) SELECT REPLACE(UUID(),'-',''), 'logined', id FROM permission WHERE path='/module/api/.dspy'; ``` **Key rules:** - LEFT JOIN alias MUST match the field name (e.g., `id as resellerid` → `on a.resellerid = b.id`) - DO NOT edit `wwwroot/
/get_
.dspy` — it's gitignored and regenerated - **Inline `data` arrays in CRUD JSON alters do NOT resolve for list grid cells** — handle status/type fields in SQL with CASE WHEN - New dspy endpoints need BOTH `permission` table entry AND `rolepermission` entry (role `logined`) - The auto-discovery path `wwwroot/{alias}/get_{alias}.dspy` takes precedence over `get_data_url` **CRITICAL deployment pitfalls (2026-07-10):** 1. **Server restart required for CRUD directory dspys**: The CRUD framework caches custom `get_{alias}.dspy` files at server STARTUP. Hot-reload applies to `wwwroot/api/` but NOT to auto-generated CRUD directories. After deploying a new `get_{alias}.dspy`, restart Sage (`./stop.sh && ./start.sh`). 2. **`default_filterjson` trap — ns field pollution**: Any field set in `ns` before `default_filterjson(fields, ns)` becomes an implicit filter. NEVER set business fields in `ns` — only set framework vars (`__logined_orgid__`, `userorgid`). Example: `ns['resellerid'] = userorgid` leaked into default_filterjson and filtered the list to 1 row. 3. **Approach A (COALESCE) is the only reliable option**: In practice, bricks Tabular did NOT render Approach B's `_text` columns. COALESCE replaces values directly in the output column — guaranteed to work. 4. **Git-ignored directory**: `wwwroot/{alias}/` is in `.gitignore`. Custom dspys require `git add -f` to track.