--- name: database-table-definition-spec version: 1.0.0 description: 定义业务表结构（models/*.json 四段式 summary/fields/indexes/codes）时必读——字段类型/约束/索引/编码引用的标准格式。不加载会产出不规范表定义（缺审计/软删除字段、金额类型用错）。设计库整体架构时用 database-design。 trigger_conditions: - User needs to create or modify database table definitions in JSON format - Task involves generating table definition files for the models directory - Working with sqlor-database-module table specifications --- # Database Table Definition Specification ## Overview This skill defines the standardized JSON format for database table definitions used with the sqlor-database-module framework. Table definitions are database-agnostic — the actual type mapping to each database (MySQL, PostgreSQL, Oracle, SQL Server, SQLite, etc.) is handled by sqlor's DDL templates. ## JSON Structure Specification ### Root Object The table definition is a JSON object with four main sections: ```json { "summary": [...], "fields": [...], "indexes": [...], "codes": [...] } ``` ### Summary Section (Required - Exactly One Record) ```json "summary": [ { "name": "table_name", // Required: Actual table name "title": "Table Title", // Required: Human-readable title "primary": ["id"], // Required: Array of primary key field names. Always ["id"] for single-key tables "catelog": "entity|relation|dimession|indication" // Optional: Table category } ] ``` **Critical**: `primary` must be an **array** (e.g., `["id"]`), NOT a string. The DDL template uses `','.join(summary[0].primary)` which would produce `"i,d"` if given the string `"id"`. ### Fields Section (Required - One or More Records) ```json "fields": [ { "name": "field_name", // Required: Field name "title": "Field Title", // Required: Human-readable title (rendered as COMMENT in DDL) "type": "str", // Required: Abstract type (see Supported Types below) "length": 32, // Required for str/char/float/double/ddouble: positive integer "dec": 2, // Required for float/double/ddouble: positive integer "nullable": "yes|no", // Optional: "no" renders as NOT NULL, omitted means nullable "default": "default_value" // Optional: Default value } ] ``` **Supported Abstract Types:** These types are database-agnostic. The actual SQL type mapping is defined in sqlor's DDL templates (`sqlor/ddl_template_mysql.py`, `sqlor/ddl_template_postgresql.py`, etc.). | Abstract Type | Description | length | dec | |--------------|-------------|--------|-----| | `str` | String/text | Required (>0) | No | | `char` | Fixed-length string | Required (>0) | No | | `short` | Small integer | No | No | | `int` | Integer | No | No | | `long` | Big integer | No | No | | `float` | Float number | Required (>0) | Required (>0) | | `double` | Double number | Required (>0) | Required (>0) | | `ddouble` | Double-double precision | Required (>0) | Required (>0) | | `decimal` | Decimal/fixed-point (alias for double) | Required (>0) | Required (>0) | | `date` | Date | No | No | | `time` | Time | No | No | | `datetime` | Date and time | No | No | | `timestamp` | Timestamp | No | No | | `text` | Long text | No | No | | `bin` | Binary data | No | No | **Pitfalls:** - **date vs timestamp — UI control depends on type**: The CRUD framework generates different form controls based on field type. `type: "timestamp"` renders a date+time picker; `type: "date"` renders a date-only picker. If the business requirement is date-only (e.g., registration date, business date), use `date` — using `timestamp` will show time components in the UI that confuse users. Example: `created_at` for "注册日期" should be `date`, not `timestamp`, while `last_login` (exact login time) should remain `timestamp`. - **NEVER use string format for length/dec**: Do NOT write `"length": "15,2"` for decimal fields. The `length` and `dec` must be separate integer keys: `"length": 15, "dec": 2`. Using a string like `"15,2"` will cause the DDL generator to produce invalid SQL (e.g., `DECIMAL('15,2')` instead of `DECIMAL(15,2)`). This was a recurring bug across financial_management and other modules. - **Do NOT omit length/dec for float/double/ddouble**: These types MUST have both `length` and `dec` as positive integers. Omitting them causes the DDL generator to produce `FLOAT` or `DECIMAL()` without precision, which fails in MySQL. Use `length: 5, dec: 2` for temperature-like values (0.00-1.00 range), `length: 15, dec: 2` for monetary amounts. **Rules:** - `id` field must use `str` type with `length: 32` (or larger if needed) - For types `str`, `char`, `float`, `double`, `ddouble`: `length` must be >0 integer - For types `float`, `double`, `ddouble`: `dec` must be >0 integer - **Do NOT use database-native types** like `varchar(64)`, `decimal(15,2)` in the `type` field. Use abstract types with `length` and `dec` parameters. - The mapping from abstract types to database-specific types is in sqlor's DDL templates (e.g., `sqlor/ddl_template_mysql.py`) ### Indexes Section (Optional) ```json "indexes": [ { "name": "idx_unique_name", // Required: Unique index name per table "idxtype": "unique|index", // Required: Index type ("unique" or "index") "idxfields": ["field1", "field2"] // Required: Array of field names (MUST be array) } ] ``` **Critical**: `idxfields` must be an **array** (e.g., `["customer_id"]`), NOT a string. Do NOT use `fields` or `columns` as the key name — only `idxfields` is recognized by the DDL template. ### Codes Section (Optional) ```json "codes": [ { "field": "target_field_name", // Required: Field that will have coded values "table": "source_table_name", // Required: Source table for lookup values "valuefield": "source_value_field", // Required: Field containing actual values "textfield": "source_display_field", // Required: Field containing display text "cond": "where_condition" // Optional: Filter condition for source data } ] ``` **CRITICAL PITFALL: appcodes_kv cond MUST use `parentid=`, NEVER `id=`** When `table` is `appcodes_kv` (the most common codes source), the `cond` field MUST filter by `parentid`, NOT `id`: ```json // ✅ CORRECT — parentid matches the dict group key {"field": "status", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='pipeline_status'"} // ❌ WRONG — id is the row primary key, not the group key; returns 0 or 1 row {"field": "status", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "id='pipeline_status'"} ``` The `parentid` field in `appcodes_kv` links items to their parent group (defined in `appcodes` table). Using `id=` silently produces empty dropdowns. This mistake has occurred repeatedly — always verify `cond` uses `parentid=` when referencing `appcodes_kv`. **CRITICAL PITFALL: appcodes and appcodes_kv data MUST be inserted together** The `appcodes` and `appcodes_kv` tables form a parent-child relationship. When adding dictionary data, you MUST insert records into BOTH tables — never only `appcodes_kv`: ```sql -- Step 1: Add parent record to appcodes INSERT INTO `appcodes` (`id`, `name`, `hierarchy_flg`) VALUES ('user_status', '用户状态', '0') ON DUPLICATE KEY UPDATE name=VALUES(name); -- Step 2: Add child records to appcodes_kv INSERT INTO `appcodes_kv` (`id`, `parentid`, `k`, `v`) VALUES ('abc12345', 'user_status', '0', '可用'), ('def67890', 'user_status', '1', '禁用') ON DUPLICATE KEY UPDATE v=VALUES(v); ``` **appcodes ID naming convention**: The `appcodes.id` field (which becomes `appcodes_kv.parentid`) MUST be a meaningful string under 31 characters. Use descriptive names like `user_status`, `pipeline_status`, `order_type` — not random IDs or UUIDs. This ID is referenced in `codes[].cond` across all modules, so readability matters. **Table structure reference:** - `appcodes` (parent): `id VARCHAR(32) PK`, `name VARCHAR(255)`, `hierarchy_flg VARCHAR(1)` — `hierarchy_flg='0'` for flat lists, `'1'` for hierarchical - `appcodes_kv` (child): `id VARCHAR(32) PK`, `parentid VARCHAR(32)`, `k VARCHAR(32)`, `v VARCHAR(255)` — unique on `(parentid, k)` **Two types of codes entries:** 1. **Dictionary codes** (table=`appcodes_kv`): Use `parentid=` cond, `valuefield: "k"`, `textfield: "v"`. Data comes from `init/data.json` Format B. 2. **Foreign key codes** (table=other table): Use `valuefield: "id"`, `textfield: "display_field"`, no cond needed. References another module's table for dropdown population. **CRITICAL PITFALL: `codes.table` MUST NOT use `module.table` dot notation** The `codes[].table` value is consumed by TWO code paths: - **Edit form dropdowns** (`get_code_desc` → `alter_field`): `c.table` builds `params.table`, then `alter_field` overrides `dataurl` with the CRUD JSON `alters` value. Dot notation here is harmless. - **Filter/search dropdowns** (`build_filter_field_list`): `c.table` is passed directly as `params.table` to the bricks client → bricks calls `get_code.dspy?table=` → `get_code.dspy` runs `SELECT ... FROM `. The dot is NOT resolved — it becomes invalid SQL `FROM module.table`. **Symptom**: CRUD filter dropdowns 500 error. Sage log shows MySQL error: `SELECT command denied to user ... for table module.table`. **Wrong**: ```json {"field": "providerid", "table": "supplychain.suppliers", "valuefield": "id", "textfield": "supplier_name"} ``` **Correct** — plain table name; `dbname` already routes to the right database: ```json {"field": "providerid", "table": "suppliers", "valuefield": "id", "textfield": "supplier_name"} ``` **CRITICAL PITFALL: Duplicate codes entries cause SQL `Duplicate column name` errors** When the `codes` array has two entries for the same `field`, the xls2crud template generates two LEFT JOINs on the same code table with identical column aliases (e.g., both produce `status_text`), producing SQL error `OperationalError(1060): Duplicate column name 'status_text'`. The CRUD list page returns 500. **Symptom**: List page returns 500 with `(1060, "Duplicate column name 'status_text'")` or similar. **Fix**: Remove duplicate entries, keeping only one per field: ```python seen = set() d['codes'] = [c for c in d['codes'] if not (c.get('field') in seen or seen.add(c.get('field')))] ``` **Why this happens**: Adding codes programmatically (e.g., Python dict.append) without checking for existing entries. Always verify the codes array has unique `field` values before saving. ## Creating Models Directory (New Modules) When a module lacks a `models/` directory but needs table definitions: 1. **Create the directory**: `mkdir -p ~/repos/{module}/models` 2. **Create JSON files**: One file per table, following the spec above 3. **Update build.sh**: Add DDL generation logic (see `references/build-sh-ddl-generation.md`) 4. **Update .gitignore**: Exclude generated `models/mysql.ddl.sql` 5. **Generate and verify**: Run `build.sh`, inspect generated DDL **json2ddl shebang workaround**: The `/d/ymq/repos/sage/py3/bin/json2ddl` script has a hardcoded shebang (`#!/home/hermesai/repos/sage/py3/bin/python3`). On systems where this path doesn't exist, invoke it explicitly: ```bash /d/ymq/repos/sage/py3/bin/python3 /d/ymq/repos/sage/py3/bin/json2ddl mysql . ``` ## File Management Requirements ### Storage Location - All table definition files **must** be stored in the `models/` directory of the module - Each table gets exactly one JSON file - Both `.xlsx` (original source) and `.json` (canonical format) may coexist in `models/` ### Naming Convention - Filename format: `{table_name}.json` - Example: A table named `users` would be stored as `models/users.json` ### Git Structure (sage/pkgs modules) Modules under `sage/pkgs/` are **independent git repos**, not tracked by the parent sage repo (`.gitignore` excludes `pkgs/`). Each module has its own `.git` directory. When committing model changes: ```bash cd ~/repos/sage/pkgs/llmage && git add models/*.json && git commit -m "..." ``` NOT `cd ~/repos/sage && git add pkgs/...` ## XLSX to JSON Conversion Table definitions originally exist as `.xlsx` files (multi-sheet Excel: `summary`, `fields`, `validation`, `codes`, `coding`, `help`). The canonical format for CRUD and DDL generation is JSON. **Conversion script**: `~/repos/sage/xlsx2json_models.py` ```bash cd ~/repos/sage && python3 xlsx2json_models.py # all modules cd ~/repos/sage && python3 xlsx2json_models.py llmage # single module cd ~/repos/sage && python3 xlsx2json_models.py --dry-run # preview ``` **DDL generation from xlsx**: `~/repos/sage/py3/bin/xls2ddl mysql /path/to/models/` See `references/xlsx-conversion.md` for full details: xlsx sheet structure, conversion logic, module list. ## Validation Rules Summary 1. **Primary Key**: Must be an **array** (e.g., `["id"]`), NOT a string. The DDL template uses `','.join(summary[0].primary)` which breaks with strings 2. **Field Types**: Use abstract types (`str`, `int`, `timestamp`), NOT database-native types (`varchar(64)`, `datetime2`) 3. **Field Length**: Required for `str`, `char`, `float`, `double`, `ddouble`; must be positive integer 4. **Decimal Places**: Required for `float`, `double`, `ddouble`; must be positive integer 5. **Index Fields**: Must use key name `idxfields` (NOT `fields` or `columns`), value must be an array 6. **Index Names**: Must be unique within each table 7. **Field Comment**: Use `title` in fields for the comment rendered in DDL 8. **File Location**: Must be in `models/` directory 9. **File Naming**: Must match table name exactly with `.json` extension ## Batch Validation & Fix When auditing or migrating all modules, use the validation script: ```bash python3 ~/.hermes/skills/software-development/database-table-definition-spec/scripts/validate_models_json.py python3 ~/.hermes/skills/software-development/database-table-definition-spec/scripts/validate_models_json.py --fix ``` The script checks all `~/repos/*/models/*.json` files against this spec. With `--fix`, it auto-corrects: - `primary` as string → wraps in array - Database-native types (VARCHAR, DECIMAL, BIGINT) → abstract types with length/dec - `length`/`dec` as strings → integers - Missing `dec` for float/double → defaults to `2` - Missing `length` for numeric types → defaults to `15` Common sources of violations: xlsx→json conversion (xlsx2json_models.py may emit float fields without `dec`), manual JSON edits using SQL types, and legacy modules that predate the spec. ## Example Complete Definition ```json { "summary": [ { "name": "products", "title": "产品目录表", "primary": ["id"], "catelog": "entity" } ], "fields": [ { "name": "id", "title": "主键ID", "type": "str", "length": 32, "nullable": "no" }, { "name": "name", "title": "产品名称", "type": "str", "length": 255, "nullable": "no" }, { "name": "price", "title": "单价", "type": "double", "length": 10, "dec": 2, "nullable": "no", "default": "0.00" }, { "name": "created_at", "title": "创建时间", "type": "timestamp", "nullable": "no" } ], "indexes": [ { "name": "idx_products_name", "idxtype": "index", "idxfields": ["name"] } ], "codes": [ { "field": "category_id", "table": "categories", "valuefield": "id", "textfield": "name" } ] } ``` ## Manual DDL / Migration Pitfalls ### ALWAYS specify `COLLATE utf8mb4_unicode_ci` for manual CREATE TABLE xls2ddl generates all tables with `COLLATE utf8mb4_unicode_ci`. Any table created manually (via `mysql -e "CREATE TABLE ..."` or direct SQL) defaults to the database collation, which is often `utf8mb4_general_ci`. This causes: ``` OperationalError: (1267, "Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT) for operation '='") ``` **Fix for existing tables**: `ALTER TABLE {tbl} CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` **Prevention**: Always append `COLLATE utf8mb4_unicode_ci` to manual CREATE TABLE statements. Memory rule: `xls2ddl:utf8mb4_unicode_ci标准。collation不一致→Illegal mix of collations。新入表先诊断后ALTER TABLE统一。` ### `ADD COLUMN IF NOT EXISTS` is MariaDB-only, NOT MySQL MySQL 8.0 does NOT support `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. Use a prepared-statement workaround instead: ```sql SET @col = (SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='sage' AND TABLE_NAME='t' AND COLUMN_NAME='c'); SET @sql = IF(@col=0, 'ALTER TABLE t ADD COLUMN c VARCHAR(3) DEFAULT ''X''', 'SELECT ''exists'''); PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt; ``` Or just run the ALTER and catch the duplicate-column error. When writing manual INSERT statements (migration scripts, data setup, seed data): **ALWAYS use explicit column names** to avoid column count mismatch errors when schema evolves: ```sql -- GOOD: Explicit columns - survives schema changes INSERT INTO pricing_program (id, name, ownerid, providerid, pricing_belong, discount, description, pricing_spec) VALUES ('pp_001', 'Name', '0', 'provider_id', 'provider', 1.0, 'desc', 'spec'); -- BAD: Positional VALUES - breaks if columns added/reordered INSERT INTO `pricing_program` VALUES ('pp_001', 'Name', '0', 'provider_id', 'provider', 'desc', 'spec'); ``` **Common failures** (2026-06-02 incident): - `pricing_program` table has 8 columns: `id, name, ownerid, providerid, pricing_belong, discount, description, pricing_spec` - `pricing_program_timing` has 6 columns: `id, ppid, name, pricing_data, enabled_date, expired_date` - INSERT without column names silently omits columns or throws "Column count doesn't match value count" errors **Schema discovery**: When unsure of column count/order, query `INFORMATION_SCHEMA.COLUMNS` or check existing correct INSERTs in the codebase before writing new ones. ## Schema Discovery (Reverse Engineering Existing Tables) When you need to discover an existing table's actual schema across the Sage codebase, see `references/schema-discovery-patterns.md`. It documents the multi-source cross-referencing approach: model JSONs → DDL SQLs → migration scripts → CRUD JSONs → Python/DSPY query patterns. Essential for data mart design, migration planning, and debugging. ## Integration Notes - This specification works with the `sqlor-database-module` skill - Table definitions are used to generate actual database schema via sqlor's DDL templates - DDL templates are database-specific: `sqlor/ddl_template_mysql.py`, `sqlor/ddl_template_postgresql.py`, `sqlor/ddl_template_sqlserver.py`, `sqlor/ddl_template_oracle.py`, etc. - Abstract types are mapped to database-native types by these templates at DDL generation time - CRUD operations reference these table definitions - Frontend components may use field metadata for form generation