pipeline_core/skills_library/all/database-table-definition-spec/SKILL.md

name	version	description	trigger_conditions
database-table-definition-spec	1.0.0	定义业务表结构（models/*.json 四段式 summary/fields/indexes/codes）时必读——字段类型/约束/索引/编码引用的标准格式。不加载会产出不规范表定义（缺审计/软删除字段、金额类型用错）。设计库整体架构时用 database-design。	User needs to create or modify database table definitions in JSON format Task involves generating table definition files for the models directory Working with sqlor-database-module table specifications

Database Table Definition Specification

Overview

This skill defines the standardized JSON format for database table definitions used with the sqlor-database-module framework. Table definitions are database-agnostic — the actual type mapping to each database (MySQL, PostgreSQL, Oracle, SQL Server, SQLite, etc.) is handled by sqlor's DDL templates.

JSON Structure Specification

Root Object

The table definition is a JSON object with four main sections:

{
    "summary": [...],
    "fields": [...], 
    "indexes": [...],
    "codes": [...]
}

Summary Section (Required - Exactly One Record)

"summary": [
    {
        "name": "table_name",           // Required: Actual table name
        "title": "Table Title",         // Required: Human-readable title
        "primary": ["id"],              // Required: Array of primary key field names. Always ["id"] for single-key tables
        "catelog": "entity|relation|dimession|indication"  // Optional: Table category
    }
]

Critical: primary must be an array (e.g., ["id"]), NOT a string. The DDL template uses ','.join(summary[0].primary) which would produce "i,d" if given the string "id".

Fields Section (Required - One or More Records)

"fields": [
    {
        "name": "field_name",           // Required: Field name
        "title": "Field Title",         // Required: Human-readable title (rendered as COMMENT in DDL)
        "type": "str",                  // Required: Abstract type (see Supported Types below)
        "length": 32,                   // Required for str/char/float/double/ddouble: positive integer
        "dec": 2,                       // Required for float/double/ddouble: positive integer  
        "nullable": "yes|no",           // Optional: "no" renders as NOT NULL, omitted means nullable
        "default": "default_value"      // Optional: Default value
    }
]

Supported Abstract Types: These types are database-agnostic. The actual SQL type mapping is defined in sqlor's DDL templates (sqlor/ddl_template_mysql.py, sqlor/ddl_template_postgresql.py, etc.).

Abstract Type	Description	length	dec
str	String/text	Required (>0)	No
char	Fixed-length string	Required (>0)	No
short	Small integer	No	No
int	Integer	No	No
long	Big integer	No	No
float	Float number	Required (>0)	Required (>0)
double	Double number	Required (>0)	Required (>0)
ddouble	Double-double precision	Required (>0)	Required (>0)
decimal	Decimal/fixed-point (alias for double)	Required (>0)	Required (>0)
date	Date	No	No
time	Time	No	No
datetime	Date and time	No	No
timestamp	Timestamp	No	No
text	Long text	No	No
bin	Binary data	No	No

Pitfalls:

• date vs timestamp — UI control depends on type: The CRUD framework generates different form controls based on field type. type: "timestamp" renders a date+time picker; type: "date" renders a date-only picker. If the business requirement is date-only (e.g., registration date, business date), use date — using timestamp will show time components in the UI that confuse users. Example: created_at for "注册日期" should be date, not timestamp, while last_login (exact login time) should remain timestamp.
• NEVER use string format for length/dec: Do NOT write "length": "15,2" for decimal fields. The length and dec must be separate integer keys: "length": 15, "dec": 2. Using a string like "15,2" will cause the DDL generator to produce invalid SQL (e.g., DECIMAL('15,2') instead of DECIMAL(15,2)). This was a recurring bug across financial_management and other modules.
• Do NOT omit length/dec for float/double/ddouble: These types MUST have both length and dec as positive integers. Omitting them causes the DDL generator to produce FLOAT or DECIMAL() without precision, which fails in MySQL. Use length: 5, dec: 2 for temperature-like values (0.00-1.00 range), length: 15, dec: 2 for monetary amounts.

Rules:

• id field must use str type with length: 32 (or larger if needed)
• For types str, char, float, double, ddouble: length must be >0 integer
• For types float, double, ddouble: dec must be >0 integer
• Do NOT use database-native types like varchar(64), decimal(15,2) in the type field. Use abstract types with length and dec parameters.
• The mapping from abstract types to database-specific types is in sqlor's DDL templates (e.g., sqlor/ddl_template_mysql.py)

Indexes Section (Optional)

"indexes": [
    {
        "name": "idx_unique_name",      // Required: Unique index name per table
        "idxtype": "unique|index",      // Required: Index type ("unique" or "index")
        "idxfields": ["field1", "field2"] // Required: Array of field names (MUST be array)
    }
]

Critical: idxfields must be an array (e.g., ["customer_id"]), NOT a string. Do NOT use fields or columns as the key name — only idxfields is recognized by the DDL template.

Codes Section (Optional)

"codes": [
    {
        "field": "target_field_name",   // Required: Field that will have coded values
        "table": "source_table_name",   // Required: Source table for lookup values  
        "valuefield": "source_value_field", // Required: Field containing actual values
        "textfield": "source_display_field", // Required: Field containing display text
        "cond": "where_condition"       // Optional: Filter condition for source data
    }
]

CRITICAL PITFALL: appcodes_kv cond MUST use parentid=, NEVER id=

When table is appcodes_kv (the most common codes source), the cond field MUST filter by parentid, NOT id:

// ✅ CORRECT — parentid matches the dict group key
{"field": "status", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "parentid='pipeline_status'"}

// ❌ WRONG — id is the row primary key, not the group key; returns 0 or 1 row
{"field": "status", "table": "appcodes_kv", "valuefield": "k", "textfield": "v", "cond": "id='pipeline_status'"}

The parentid field in appcodes_kv links items to their parent group (defined in appcodes table). Using id= silently produces empty dropdowns. This mistake has occurred repeatedly — always verify cond uses parentid= when referencing appcodes_kv.

CRITICAL PITFALL: appcodes and appcodes_kv data MUST be inserted together

The appcodes and appcodes_kv tables form a parent-child relationship. When adding dictionary data, you MUST insert records into BOTH tables — never only appcodes_kv:

-- Step 1: Add parent record to appcodes
INSERT INTO `appcodes` (`id`, `name`, `hierarchy_flg`) VALUES
  ('user_status', '用户状态', '0')
ON DUPLICATE KEY UPDATE name=VALUES(name);

-- Step 2: Add child records to appcodes_kv
INSERT INTO `appcodes_kv` (`id`, `parentid`, `k`, `v`) VALUES
  ('abc12345', 'user_status', '0', '可用'),
  ('def67890', 'user_status', '1', '禁用')
ON DUPLICATE KEY UPDATE v=VALUES(v);

appcodes ID naming convention: The appcodes.id field (which becomes appcodes_kv.parentid) MUST be a meaningful string under 31 characters. Use descriptive names like user_status, pipeline_status, order_type — not random IDs or UUIDs. This ID is referenced in codes[].cond across all modules, so readability matters.

Table structure reference:

• appcodes (parent): id VARCHAR(32) PK, name VARCHAR(255), hierarchy_flg VARCHAR(1) — hierarchy_flg='0' for flat lists, '1' for hierarchical
• appcodes_kv (child): id VARCHAR(32) PK, parentid VARCHAR(32), k VARCHAR(32), v VARCHAR(255) — unique on (parentid, k)

Two types of codes entries:

1. Dictionary codes (table=appcodes_kv): Use parentid= cond, valuefield: "k", textfield: "v". Data comes from init/data.json Format B.
2. Foreign key codes (table=other table): Use valuefield: "id", textfield: "display_field", no cond needed. References another module's table for dropdown population.

CRITICAL PITFALL: codes.table MUST NOT use module.table dot notation

The codes[].table value is consumed by TWO code paths:

• Edit form dropdowns (get_code_desc → alter_field): c.table builds params.table, then alter_field overrides dataurl with the CRUD JSON alters value. Dot notation here is harmless.
• Filter/search dropdowns (build_filter_field_list): c.table is passed directly as params.table to the bricks client → bricks calls get_code.dspy?table=<value> → get_code.dspy runs SELECT ... FROM <table>. The dot is NOT resolved — it becomes invalid SQL FROM module.table.

Symptom: CRUD filter dropdowns 500 error. Sage log shows MySQL error: SELECT command denied to user ... for table module.table.

Wrong:

{"field": "providerid", "table": "supplychain.suppliers", "valuefield": "id", "textfield": "supplier_name"}

Correct — plain table name; dbname already routes to the right database:

{"field": "providerid", "table": "suppliers", "valuefield": "id", "textfield": "supplier_name"}

CRITICAL PITFALL: Duplicate codes entries cause SQL Duplicate column name errors

When the codes array has two entries for the same field, the xls2crud template generates two LEFT JOINs on the same code table with identical column aliases (e.g., both produce status_text), producing SQL error OperationalError(1060): Duplicate column name 'status_text'. The CRUD list page returns 500.

Symptom: List page returns 500 with (1060, "Duplicate column name 'status_text'") or similar.

Fix: Remove duplicate entries, keeping only one per field:

seen = set()
d['codes'] = [c for c in d['codes'] if not (c.get('field') in seen or seen.add(c.get('field')))]

Why this happens: Adding codes programmatically (e.g., Python dict.append) without checking for existing entries. Always verify the codes array has unique field values before saving.

Creating Models Directory (New Modules)

When a module lacks a models/ directory but needs table definitions:

1. Create the directory: mkdir -p ~/repos/{module}/models
2. Create JSON files: One file per table, following the spec above
3. Update build.sh: Add DDL generation logic (see references/build-sh-ddl-generation.md)
4. Update .gitignore: Exclude generated models/mysql.ddl.sql
5. Generate and verify: Run build.sh, inspect generated DDL

json2ddl shebang workaround: The /d/ymq/repos/sage/py3/bin/json2ddl script has a hardcoded shebang (#!/home/hermesai/repos/sage/py3/bin/python3). On systems where this path doesn't exist, invoke it explicitly:

/d/ymq/repos/sage/py3/bin/python3 /d/ymq/repos/sage/py3/bin/json2ddl mysql .

File Management Requirements

Storage Location

• All table definition files must be stored in the models/ directory of the module
• Each table gets exactly one JSON file
• Both .xlsx (original source) and .json (canonical format) may coexist in models/

Naming Convention

• Filename format: {table_name}.json
• Example: A table named users would be stored as models/users.json

Git Structure (sage/pkgs modules)

Modules under sage/pkgs/ are independent git repos, not tracked by the parent sage repo (.gitignore excludes pkgs/). Each module has its own .git directory. When committing model changes:

cd ~/repos/sage/pkgs/llmage && git add models/*.json && git commit -m "..."

NOT cd ~/repos/sage && git add pkgs/...

XLSX to JSON Conversion

Table definitions originally exist as .xlsx files (multi-sheet Excel: summary, fields, validation, codes, coding, help). The canonical format for CRUD and DDL generation is JSON.

Conversion script: ~/repos/sage/xlsx2json_models.py

cd ~/repos/sage && python3 xlsx2json_models.py          # all modules
cd ~/repos/sage && python3 xlsx2json_models.py llmage    # single module
cd ~/repos/sage && python3 xlsx2json_models.py --dry-run # preview

DDL generation from xlsx: ~/repos/sage/py3/bin/xls2ddl mysql /path/to/models/

See references/xlsx-conversion.md for full details: xlsx sheet structure, conversion logic, module list.

Validation Rules Summary

1. Primary Key: Must be an array (e.g., ["id"]), NOT a string. The DDL template uses ','.join(summary[0].primary) which breaks with strings
2. Field Types: Use abstract types (str, int, timestamp), NOT database-native types (varchar(64), datetime2)
3. Field Length: Required for str, char, float, double, ddouble; must be positive integer
4. Decimal Places: Required for float, double, ddouble; must be positive integer
5. Index Fields: Must use key name idxfields (NOT fields or columns), value must be an array
6. Index Names: Must be unique within each table
7. Field Comment: Use title in fields for the comment rendered in DDL
8. File Location: Must be in models/ directory
9. File Naming: Must match table name exactly with .json extension

Batch Validation & Fix

When auditing or migrating all modules, use the validation script:

python3 ~/.hermes/skills/software-development/database-table-definition-spec/scripts/validate_models_json.py
python3 ~/.hermes/skills/software-development/database-table-definition-spec/scripts/validate_models_json.py --fix

The script checks all ~/repos/*/models/*.json files against this spec. With --fix, it auto-corrects:

• primary as string → wraps in array
• Database-native types (VARCHAR, DECIMAL, BIGINT) → abstract types with length/dec
• length/dec as strings → integers
• Missing dec for float/double → defaults to 2
• Missing length for numeric types → defaults to 15

Common sources of violations: xlsx→json conversion (xlsx2json_models.py may emit float fields without dec), manual JSON edits using SQL types, and legacy modules that predate the spec.

Example Complete Definition

{
    "summary": [
        {
            "name": "products",
            "title": "产品目录表",
            "primary": ["id"],
            "catelog": "entity"
        }
    ],
    "fields": [
        {
            "name": "id",
            "title": "主键ID",
            "type": "str",
            "length": 32,
            "nullable": "no"
        },
        {
            "name": "name",
            "title": "产品名称", 
            "type": "str",
            "length": 255,
            "nullable": "no"
        },
        {
            "name": "price",
            "title": "单价",
            "type": "double",
            "length": 10,
            "dec": 2,
            "nullable": "no",
            "default": "0.00"
        },
        {
            "name": "created_at",
            "title": "创建时间",
            "type": "timestamp",
            "nullable": "no"
        }
    ],
    "indexes": [
        {
            "name": "idx_products_name",
            "idxtype": "index", 
            "idxfields": ["name"]
        }
    ],
    "codes": [
        {
            "field": "category_id",
            "table": "categories",
            "valuefield": "id", 
            "textfield": "name"
        }
    ]
}

Manual DDL / Migration Pitfalls

ALWAYS specify COLLATE utf8mb4_unicode_ci for manual CREATE TABLE

xls2ddl generates all tables with COLLATE utf8mb4_unicode_ci. Any table created manually (via mysql -e "CREATE TABLE ..." or direct SQL) defaults to the database collation, which is often utf8mb4_general_ci. This causes:

OperationalError: (1267, "Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT) for operation '='")

Fix for existing tables: ALTER TABLE {tbl} CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Prevention: Always append COLLATE utf8mb4_unicode_ci to manual CREATE TABLE statements. Memory rule: xls2ddl:utf8mb4_unicode_ci标准。collation不一致→Illegal mix of collations。新入表先诊断后ALTER TABLE统一。

ADD COLUMN IF NOT EXISTS is MariaDB-only, NOT MySQL

MySQL 8.0 does NOT support ALTER TABLE ... ADD COLUMN IF NOT EXISTS. Use a prepared-statement workaround instead:

SET @col = (SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='sage' AND TABLE_NAME='t' AND COLUMN_NAME='c');
SET @sql = IF(@col=0, 'ALTER TABLE t ADD COLUMN c VARCHAR(3) DEFAULT ''X''', 'SELECT ''exists''');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;

Or just run the ALTER and catch the duplicate-column error.

When writing manual INSERT statements (migration scripts, data setup, seed data):

ALWAYS use explicit column names to avoid column count mismatch errors when schema evolves:

-- GOOD: Explicit columns - survives schema changes
INSERT INTO pricing_program (id, name, ownerid, providerid, pricing_belong, discount, description, pricing_spec)
VALUES ('pp_001', 'Name', '0', 'provider_id', 'provider', 1.0, 'desc', 'spec');

-- BAD: Positional VALUES - breaks if columns added/reordered
INSERT INTO `pricing_program` VALUES ('pp_001', 'Name', '0', 'provider_id', 'provider', 'desc', 'spec');

Common failures (2026-06-02 incident):

• pricing_program table has 8 columns: id, name, ownerid, providerid, pricing_belong, discount, description, pricing_spec
• pricing_program_timing has 6 columns: id, ppid, name, pricing_data, enabled_date, expired_date
• INSERT without column names silently omits columns or throws "Column count doesn't match value count" errors

Schema discovery: When unsure of column count/order, query INFORMATION_SCHEMA.COLUMNS or check existing correct INSERTs in the codebase before writing new ones.

Schema Discovery (Reverse Engineering Existing Tables)

When you need to discover an existing table's actual schema across the Sage codebase, see references/schema-discovery-patterns.md. It documents the multi-source cross-referencing approach: model JSONs → DDL SQLs → migration scripts → CRUD JSONs → Python/DSPY query patterns. Essential for data mart design, migration planning, and debugging.

Integration Notes

• This specification works with the sqlor-database-module skill
• Table definitions are used to generate actual database schema via sqlor's DDL templates
• DDL templates are database-specific: sqlor/ddl_template_mysql.py, sqlor/ddl_template_postgresql.py, sqlor/ddl_template_sqlserver.py, sqlor/ddl_template_oracle.py, etc.
• Abstract types are mapped to database-native types by these templates at DDL generation time
• CRUD operations reference these table definitions
• Frontend components may use field metadata for form generation