yumoqing/pipeline_core

Fork 0

yumoqing 1abaa4ea2e feat: 技能库(skills_library 213个Hermes技能)+skill_pack技能集安装能力+ocai-h5-dev技能集(37个应用/模块开发技能)

2026-08-16 15:17:43 +08:00

104 KiB

Raw Blame History

name

description

author

llmage Module Skill

Overview

llmage: LLM lifecycle in Sage (registration, display, inference, usage, billing); providers wired via uapi config.

Accounting Failure Tracking (记账失败)

仪表盘失败数≠记账失败记录列表条数是口径差（llmusage 当前态 vs 只追加审计表，重试/双写会膨胀）。诊断 SQL、文件地图、已知 bug 见 references/accounting-failed-tracking.md。

Module Location

~/repos/llmage/

External Service Registration

Registering external GPU/production services into Sage: see references/external-service-registration.md.

Product Sync Chain & providerid Backfill

llmage→product_management product sync (load_product_category_product → import_category_products.dspy), backfill SQL for product.providerid, and end-to-end verify/cleanup recipe: see references/product-sync-providerid.md. Key pitfall: if a synced field comes back empty, check the source-side SELECT column list first — getattr(row,'col','') silently degrades to '' when the sender never selected the column (2026-08 providerid bug).

For adding new LLM models from existing providers (llm + llm_api_map + uapiio tables), see references/new-llm-model-sql-pattern.md. It covers the three-table relationship, SQL templates, worked examples (kimi-k3), verification approach, and common pitfalls.

For adding third-party LLM providers that require multimodal base64 input (no external URL support, e.g. Kimi/Moonshot): see references/multimodal-uapi-base64.md. Covers pure Jinja2 uapi template patterns with minimal helper functions registered on ServerEnv.

Architecture: llmage + uapi Collaboration

llmage (model business layer)          uapi (API gateway layer)
     │                                      │
     │  llm table                           │
     │  upappid ─────────────────────────→│ upapp table (baseurl, myappid, ownerid, auth_apiname)
     │      (llm has NO apiname field!)    │
     │  ──1:N──> llm_api_map ────N:1──→│  (each upapp owns its uapi records directly)
     │      (apiname, query_apiname,      │
     │       query_period, ppid,           │
     │       llmcatelogid)                 │
     │      apiname ─────────────────────→│ uapi table (httpmethod, path, headers, upappid)
     │                                    │
     │                                    │ upappkey table (apikey, secretkey)
     │                                    │
     │  UpAppApi(request)                 │
     │  .stream_linify() ────────────────→│ StreamHttpClient → External LLM API
     │  .call() ─────────────────────────→│ sync/streaming HTTP call

Key: uapiset table removed. auth_apiname moved from uapiset to upapp. uapi links to upapp via upappid (was apisetid). Each upapp owns its uapi records directly. CRITICAL: apiname is in llm_api_map, NOT in llm table. JOIN to uapi must go through llm_api_map.apiname, not llm.apiname (which doesn't exist).

Inference Flow

User clicks model card
    │
    ▼
llm_dialog.ui (LlmIO widget)
    │  ← if model has multiple catelogs, renders tab buttons
    │     (HBox + Button per catelog, default = isdefaultcatelog='1')
    │     Each tab reloads UI with ?id=xxx&catelogid=yyy
    │
    ▼
llminference.dspy → inference_generator()
    │  ← reads params_kw.llmcatelogid (set by bricks LlmModel)
    │  ← passes to get_llm(llmid, catelogid) to select correct llm_api_map row
    │
    ├── llm.stream == 'async' ──→ async_uapi_request() ── submit task + background polling
    ├── llm.stream == False ────→ sync_uapi_request() ─── one-time sync call
    └── llm.stream == True ─────→ uapi_request() ──────── streaming (SSE)

Key Tables

Table	Purpose	Key Fields
llm	Model definition (base metadata)	id, name, model, providerid, ownerid, upappid, enabled_date, expired_date
llm_api_map	Model capability + catalog mapping (1:N)	id, llmid, llmcatelogid, apiname, query_apiname, query_period, ppid
llmusage	Call records (active)	id, llmid, userid, userorgid, usages, status(SUCCEEDED/FAILED), amount, cost, use_time, accounting_status, taskid
llmusage_history	Archived call records (read-only)	Same as llmusage + backup_time
llmusage_accounting_failed	Accounting failure tracking	id, llmusageid, llmid, userid, userorgid, failed_reason, failed_time, retry_count, handled
llmcatelog	Model categories. ID standardization in progress: migrating to meaningful abbreviations (t2t, t2v, etc.) managed via appcodes system.	id, name, description

Note: llm_catalog_rel table 废弃 — catalog relationship is now maintained directly in llm_api_map. llm table no longer has llmcatelogid, query_apiname, query_period, or ppid. All SQL JOINs that previously used llm_catalog_rel must now use llm_api_map (with distinct to handle 1:N duplication). llm.json's subtable is now llm_api_map. Management UI: /llmage/llm_api_map_manage.ui.

llmcatelog ID standardization: ✅ COMPLETED (2026-05-30). The llmcatelog table now uses meaningful abbreviations as IDs (t2t, t2i, t2v, i2v, r2v, tts, asr, vision, etc.). All v1 API endpoints use backward-compatible SQL matching (b.id = ${x}$ OR b.name = ${x}$) so callers can pass either the new ID or legacy Chinese name. See references/llmcatelog-id-migration.md for the complete migration record.

llm_api_map: A multi-ability model (e.g., video generation with t2v/i2v/ref2v) has ONE row in llm and multiple rows in llm_api_map. Each row links to one llmcatelogid and carries ability-specific fields (apiname, query_apiname, query_period, ppid). This table replaces the deprecated llm_catalog_rel — it carries BOTH catalog relationship AND API capability mapping.

get_llm() Architecture (Refactored — No More 6-Table JOIN)

get_llm() was refactored from a 6-table implicit JOIN to a 3-step cached approach:

Step 1: get_llmage_llm() — 3-table JOIN (llmage DB)

sql = """select a.id, a.name, a.model, a.providerid, a.description,
a.iconid, a.upappid, a.ownerid, a.min_balance, a.status,
m.llmcatelogid, m.apiname, m.query_apiname, m.query_period, m.ppid, m.isdefaultcatelog,
lc.name as catelogname
from llm a
join llm_api_map m on a.id = m.llmid
join llmcatelog lc on m.llmcatelogid = lc.id
where 1=1
"""
# Conditional WHERE logic:
# - llmid + catelogid: filter by BOTH (user selected specific catelog)
# - llmid only: filter by isdefaultcatelog='1' (default fallback)
# - catelogid only: filter by catelogid (list all models of that type)
ns = {}
if llmid:
    sql += " and a.id = ${llmid}$"
    ns['llmid'] = llmid
    if catelogid:
        sql += " and m.llmcatelogid = ${catelogid}$"
        ns['catelogid'] = catelogid
    else:
        sql += " and m.isdefaultcatelog = '1'"
elif catelogid:
    sql += " and m.llmcatelogid = ${catelogid}$"
    ns['catelogid'] = catelogid

Step 2: _get_uapi_cached() — uapi DB lookup (cached, 5min TTL)

uapi = await get_uapi(llm.upappid, llm.apiname)
# Returns: ioid, stream, callbackurl, auth_apiname, headers, params, data, response

Step 3: _get_uapiio_cached() — uapiio DB lookup (cached, 5min TTL)

uapiio = await sor.R('uapiio', {'id': uapi.ioid})
# Returns: input_fields (JSON text)

Merge into llm result

llm.ioid = uapi.ioid
llm.stream = uapi.stream
llm.callbackurl = uapi.callbackurl
llm.input_fields = uapiio.input_fields if uapiio else '{}'

Performance: First call = 3 DB round-trips; subsequent calls = 1 DB + 2 memory cache hits. Faster than old 6-table JOIN after first request.

Cache invalidation: invalidate_uapi_cache(upappid, apiname) for specific entries, no-args for full flush. Exported via env.invalidate_uapi_cache.

LLM API Map Management UI

Available at /llmage/llm_api_map_manage.ui for logged-in users. Triggered from llm CRUD tool with llmid parameter. Features:

Add model capability mappings (catalog, API, query API, polling period, billing program)
View all current mappings in a table
Delete individual mappings
API endpoints: api/llm_api_map_list.dspy, api/llm_api_map_create.dspy, api/llm_api_map_delete.dspy, api/llm_api_map_options.dspy
llmid is a hidden field passed via {{params_kw.llmid}} — not user-selectable

ServerEnv Functions (via load_llmage())

Available in .dspy files via globals():

Function	Purpose
`get_llm(llmid, catelogid=None)`	Get model info with uapi/uapiio (cached). Internally calls get_llmage_llm() + cached uapi lookups. For API calls/inference. When `catelogid` is passed, filters to that specific llm_api_map entry (critical for multi-catelog models).
`get_llmage_llm(llmid=None, catelogid=None)`	Lightweight accessor: llm + llm_api_map + llmcatelog only (for display, accounting, pricing — NOT for API calls). When `llmid` is passed without `catelogid`, filters to `isdefaultcatelog='1'`.
`get_llm_catelogs(llmid)`	Returns ALL catelog entries for a model: `[{catelogid, catelogname, isdefault}]`. Use in UI templates to render catelog selection tabs when a model has multiple llm_api_map entries.
`invalidate_uapi_cache(upappid=None, apiname=None)`	Invalidate uapi/uapiio process-level cache. Call when uapi config changes.
`inference(request, params_kw)`	Inference entry (auto stream response)
`inference_generator(request, params_kw)`	Inference generator (yields JSON lines)
`get_llms_by_catelog(catelogid=None)`	Get models grouped by category. Optional `catelogid` filters to a single category. Uses `JOIN llm_api_map` internally (with `distinct`). Each model record includes `pricing_display` — a list of pricing display_text strings from all distinct ppids via `get_pricing_display()`.
`get_llms_sort_by_provider()`	Get models grouped by provider
`get_llmcatelogs()`	Get all categories
`get_llmproviders()`	Get all providers
`checkCustomerBalance(llmid, userorgid)`	Check if user has enough balance
`llm_charging(ppid, llmusage)`	Calculate cost via pricing program
`llm_query_price(llmid, config_data)`	Query price for a model
`get_asynctask_status(request, taskid)`	Get async task status
`get_today_asynctask_list(userid)`	Get today's async task list
`query_task_status(request, luid)`	Manually poll task status
`llm_query_orders(userorgid, page)`	Query order history
`backup_accounted_llmusage(cutoff_date)`	Backup accounted records with use_date < cutoff_date to llmusage_history using INSERT SELECT, then DELETE from llmusage. Called by backend_accounting() on date change.
`get_failed_accounting_records(filters, page, page_size)`	Search accounting failure records with optional filters (userorgid, llmid, handled, start_date, end_date)
`retry_accounting(llmusageid)`	Reset a failed accounting record to 'created' status for re-processing. Updates failure record: handled='1', retry_count+1. Called by retry_accounting.dspy from the failed accounting UI.

OpenAI-Compatible v1 API Endpoints

All endpoints use base path /llmage/v1. Unified parameter: all endpoints use catelogid (not lctype or llmcatelogid).

Endpoint	Method	Purpose	Required Params
`/v1/chat/completions`	POST	Text generation (streaming/sync)	model, prompt or messages. Optional: `catelogid` (default "t2t")
`/v1/models`	GET	List available models (OpenAI format)	Optional: `catelogid` (filter)
`/v1/models/catelog`	GET	List models by catalog (full detail)	catelogid. Optional: `exclude_id`
`/v1/tasks`	GET	Query async task status	taskid
`/v1/video/generations`	POST	Video generation (async)	model, catelogid, prompt
`/v1/image/generations`	POST	Image generation (sync only)	model, catelogid, prompt
`/v1/music/generations`	POST	Music generation (sync)	model, catelogid="music_gen", prompt, lyrics
`/v1/audio/speech`	POST	Text-to-speech / TTS (stream)	model, catelogid="tts", prompt. Optional: speaker, speed, emotion
`/v1/audio/transcriptions`	POST	Speech recognition / ASR (sync)	model, catelogid="asr", audio_file
`/v1/pricing`	GET	Model pricing display info	model. Optional: catelogid (default "t2t")

Backward-compatible catalog matching: All v1 endpoints accept catelogid as either the new abbreviation ID (t2t, t2v, i2v, etc.) or the legacy Chinese name (文生文, 文生视频, 图生视频). SQL uses (b.id = ${catelogid}$ OR b.name = ${catelogid}$).

Common flow: validate params → lookup llm via llm + llm_api_map + llmcatelog JOIN → checkCustomerBalance() → dispatch to inference() (routes to sync/streaming/async based on model config).

RBAC: All v1 endpoints require logined permission. Each endpoint needs both directory and file path entries in load_path.py:

/llmage/v1/video/generations logined
/llmage/v1/video/generations/index.dspy logined

File naming: v1 endpoints can use either index.dspy (under a subdirectory, e.g. v1/models/index.dspy → /v1/models) or a direct file (e.g. v1/models/catelog.dspy → /v1/models/catelog). Direct files work for flat endpoints that don't need their own directory. RBAC entries must match the actual file path exactly.

Three Inference Modes

1. Streaming (uapi_request) — for SSE-capable LLMs

async for line in uapi_request(request, llm, callerid, callerorgid, params_kw):
    yield line  # JSON lines: {"content": "..."}, {"usage": {...}}

2. Synchronous (sync_uapi_request) — for one-shot responses

result = await sync_uapi_request(request, llm, callerid, callerorgid, params_kw)

3. Asynchronous (async_uapi_request) — for long-running tasks (video generation)

result = await async_uapi_request(request, llm, callerid, callerorgid, params_kw)
# Returns: {"taskid": "xxx", "status": "PENDING"}
# Background auto-polls via query_task_status()

Billing System

Online calls do NOT charge immediately — mark accounting_status='created'
Background task backend_accounting() runs every 10 seconds, processes all 'created' records
Balance check: checkCustomerBalance(llmid, userorgid) returns False if balance < llm.min_balance. ALL requests go through balance check — no bypass for owner organization.
Failure tracking: When accounting fails, llm_accoung_failed(luid, reason) records the failure in llmusage_accounting_failed table with failed_reason, failed_time, and handled='0'. Use get_failed_accounting_records() to search failures. Frontend at /llmage/failed_accounting.ui.
History backup: backend_accounting() detects date changes in its 10-second loop (tracking last_backup_date). When the date changes, it computes yesterday = today - 1 day and calls backup_accounted_llmusage(yesterday) exactly once per day. This uses INSERT INTO ... SELECT FROM (single SQL) to batch-copy accounting_status='accounted' records with use_date < yesterday to llmusage_history, then DELETE (single SQL) to remove them from llmusage. Keeps the active table lean while retaining yesterday's data for one full day.
Accounting status values: 'created' (pending), 'accounted' (success), 'failed' (error — see llmusage_accounting_failed for details)

Pricing Engine Internals

See references/pricing-engine-internals.md for:

price_factors: flat vs price_factors: <fieldname> behavior
Per-model ppid isolation requirement (shared ppid → billing mismatches)
Cache behavior (get_ppid_pricing caches by {ppid}.{date})
enabled_date requirement
Accounting query conditions
Verification flow with concrete numbers

Permission Setup for New Endpoints

Deployment workflow: See references/sage-endpoint-deployment.md for the complete 6-step process to deploy new v1 endpoints to tokentest/production Sage instances (deploy dspy → update load_path.py → run load_path.py → fix rolepermission → restart Sage → verify).

Every new .ui and .dspy file needs RBAC permission records. Without them, users get HTTP 401.

Preferred: Shell Script (`scripts/setup_<module>_perms.sh`)

Follow the Sage convention used in setup_rbac_perms.sh. set_role_perm.py handles both permission insertion AND role assignment — if the path doesn't exist in permission, it auto-creates it:

#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SAGE_DIR="$(cd "$SCRIPT_DIR/../.." && pwd 2>/dev/null || echo "")"
cd "$SAGE_DIR"

set_perm() {
    python set_role_perm.py "$1" "$2"
}

PERM_ROLES=("owner.superuser" "owner.admin" "reseller.admin" "reseller.operator")
FEATURE_PATHS=("/llmage/new_page.ui" "/llmage/api/new_action.dspy")

for p in "${FEATURE_PATHS[@]}"; do
    for role in "${PERM_ROLES[@]}"; do
        set_perm "${role}" "${p}"
    done
done

v1 API endpoints need SEPARATE customer role permissions. The v1 endpoints are customer-facing (Bearer token auth via dapi module). Admin/operator roles alone are NOT sufficient — customers get 403 without explicit customer role entries:

CUSTOMER_ROLES=("customer.admin" "customer.user")
V1_API_PATHS=(
    "/llmage/v1/chat/completions/index.dspy"
    "/llmage/v1/video/generations/index.dspy"
    "/llmage/v1/image/generations/index.dspy"
    "/llmage/v1/music/generations/index.dspy"
    "/llmage/v1/audio/speech/index.dspy"
    "/llmage/v1/audio/transcriptions/index.dspy"
    "/llmage/v1/models/index.dspy"
    "/llmage/v1/models/catelog.dspy"
    "/llmage/v1/tasks/index.dspy"
    "/llmage/v1/pricing/index.dspy"
)
for p in "${V1_API_PATHS[@]}"; do
    for role in "${CUSTOMER_ROLES[@]}"; do
        set_perm "${role}" "${p}"
    done
done

Similarly, in load_path.py, customer v1 paths need a separate PATHS_V1_CUSTOMER list registered under customer roles, distinct from the PATHS_LOGINED list:

PATHS_V1_CUSTOMER = [
    f"/{MOD}/v1/chat/completions/index.dspy",
    f"/{MOD}/v1/video/generations/index.dspy",
    f"/{MOD}/v1/image/generations/index.dspy",
    f"/{MOD}/v1/music/generations/index.dspy",
    f"/{MOD}/v1/audio/speech/index.dspy",
    f"/{MOD}/v1/audio/transcriptions/index.dspy",
    f"/{MOD}/v1/models/index.dspy",
    f"/{MOD}/v1/models/catelog.dspy",
    f"/{MOD}/v1/tasks/index.dspy",
]
# In main():
for role in ["customer.admin", "customer.user"]:
    total += register_role_paths(role, PATHS_V1_CUSTOMER)

Fallback: Production SQL Migration

For production database changes (executed by the user):

INSERT INTO permission (id, path) 
SELECT REPLACE(UUID(), '-', ''), '/your/new/path.dspy' 
WHERE NOT EXISTS (SELECT 1 FROM permission WHERE path = '/your/new/path.dspy');

ID must be 32-char hex string (no dashes). Use REPLACE(UUID(), '-', '') in SQL or getID() in Python.

Permission Path Convention

Paths are relative to the module's wwwroot/ directory:

wwwroot/index.ui → /llmage/index.ui
wwwroot/api/list.dspy → /llmage/api/list.dspy

Data Isolation Pattern

Do NOT add redundant orgid/ownerid columns to child tables. Instead, JOIN to the parent table's ownerid for filtering and permission verification:

# In DSPY files
user_orgid = await get_userorgid()
# JOIN parent table for ownership check
sql = "select m.* from llm_api_map m join llm l on m.llmid = l.id where l.ownerid = ${ownerid}$"

Publish/Unpublish (上架/下架)

Models have a status field controlling visibility:

published — visible to users in all display pages and usable for inference/API calls
unpublished (default) — only visible in CRUD management interface

Affected Query Points (all filter `a.status = 'published'`)

utils.py: get_llms_by_catelog_to_customer(), get_llms_by_catelog(), get_llm(), get_llmproviders(), get_llms_sort_by_provider()
v1 endpoints: chat/completions, image/generations, video/generations (model name lookup)
User pages: t2t/index.dspy, get_type_llms.dspy, list_catelog_models.dspy, list_paging_catelog_llms.dspy, llmcheck.dspy

NOT filtered (admin/internal)

api/llm_list.dspy — CRUD list shows ALL models
api/llm_create.dspy, api/llm_update.dspy — CRUD operations
accounting.py — backend accounting processes all records
get_llm_by_model() — utility function, not called by user-facing code

Migration

sql/add_status_field.sql: ALTER TABLE + UPDATE SET status='published' for existing models + index on status.

Testing v1 APIs

Authentication

v1 endpoints use Bearer token auth via the dapi module. The auth chain:

bearer_auth() extracts token from Authorization: Bearer <key> header
get_apikey_user() encrypts the key via password_encode() (RC4) and looks it up in downapikey table
Joins users and downapp tables; checks downapp.allowedips for IP whitelist
Returns user if all checks pass, None otherwise → 401/403

Different downapp keys have different permission scopes. A key that works for /v1/chat/completions may return 401 on /v1/image/generations or /v1/video/generations. When testing, verify the key has access to ALL target endpoints.

Testing Workflow

When writing automated test scripts for llmage v1 APIs:

Always read the API key from ~/.hermes/config.yaml (model.api_key), never hardcode or manually type it. Hardcoded keys lead to wasted debugging time on auth failures.
Include catelogid for image/video endpoints: These are mandatory parameters. Without catelogid, endpoints return 400 "Missing required parameter".
catelogid accepts abbreviation IDs or Chinese names: Pass "t2v", "t2i", "i2v" (recommended) or legacy Chinese names "文生视频", "文生图", "图生视频". Both work via backward-compatible SQL matching.
/v1/chat/completions: catelogid is optional (defaults to "t2t"). Only model + messages required.
Handle non-JSON responses: Some endpoints return 401: Unauthorized as plain text, not JSON. Parse with try/except.

See references/v1-api-testing.md for a complete test script template.

Pitfalls

See references/pitfalls-27-33.md for additional pitfalls (accounting, tenantid, ASR uapi, backend process).

llm.stream field controls mode: 'async' = async task, False = sync, True = streaming
新增v1端点前必须先学习现有模式: 不要盲目创建 dspy 文件。先读现有 v1 端点（如 /v1/chat/completions/index.dspy）了解正确的 dspy 结构、SQL JOIN 模式、错误处理方式。用户明确纠正："不能自己瞎做，改学习一下uapi和llmage的API接口skill和规范"。下午会话的工作应该被继承，不是重新发明。默认使用 openai_403()/openai_400()/openai_429() 错误返回（非 UiError）。/v1/media/{apiname} 每端点需独立目录+index.dspy（非catch-all）。模板和部署流程见 references/v1-media-dspy-template.md。
llm ↔ uapi bridge: llm table's upappid + llm_api_map's apiname must match existing uapi configuration. JOIN chain: llm → upapp → uapi (no uapiset).
BufferedLLMs cache: Model definitions cached by date ({llmid}.{date}), auto-expire on day change
IO persistence: Input/output stored as JSON files via FileStorage; llmusage.ioinfo is the webpath
API key protection: Exception messages are sanitized by erase_apikey() — Bearer tokens replaced with XXXXXXXX
Async query_apiname: Can be comma-separated; each API name is polled in sequence until SUCCEEDED/FAILED
query_period: Polling interval in seconds (default 30), configured in llm_api_map table (NOT llm table)
ppid: Pricing program ID is in llm_api_map table. llm_query_price reads it from the BufferedLLMs result.
New model requires uapi setup first: upapp → uapi → upappkey → llm + llm_api_map (in that order). NO uapiset.
input_fields: Model input field definitions stored in uapiio table, auto-linked by BufferedLLMs
llm_api_map migration: When adding llm_api_map to an existing system, migrate existing llm data first (apiname, query_apiname, query_period, ppid) into llm_api_map rows before deploying code changes.
uapiset removed: All JOINs must not reference uapiset. auth_apiname is now on the upapp table. uapi links to upapp via upappid (each upapp owns its own uapi records).
bricks subtable limitation: A CRUD JSON subtables[] can only bind ONE subtable. When switching from llm_catalog_rel to llm_api_map as the subtable, you cannot have simultaneous CRUD for both tables from llm.json. Create an independent management UI (llm_api_map_manage.ui) with its own list/create/delete DSPYs.
llm_catalog_rel 已废弃: 所有 SQL JOIN 中引用 llm_catalog_rel 的地方必须改为 llm_api_map。因为 llm_api_map 是 1:N 关系，select 需加 distinct 去重。受影响文件：utils.py (get_llms_by_catelog, get_llms_by_catelog_to_customer)、所有 *.dspy 查询文件、init.py 事件绑定。
Pre-existing llmusage.json float fields missing dec: The existing models/llmusage.json and models/llmusage.xlsx define responsed_seconds, finish_seconds, amount, cost as "type": "float" with "length": 18 but NO "dec". json2ddl produces invalid double(18,) syntax. When creating new history/failed tables that mirror these fields, use "type": "double" with both "length": 18 AND "dec": 5 (for amount/cost) or "dec": 3 (for seconds). The pre-existing issue in llmusage.json itself is a known debt item — changing it would regenerate DDL for the existing table.
Pre-existing CRUD gaps: json/llm.json and json/llm_api_map.json are missing the mandatory editable paragraph (spec Pitfall 7). json/llm.json also has browserfields.alters referencing "ppid" — this field was moved from llm to llm_api_map during the uapiset removal migration, so the alters reference is broken. These are known debt items.
Read-only CRUD pattern for archive tables: When a table is append-only or read-only (e.g., llmusage_history), still provide editable in the CRUD JSON per spec, but point create/update/delete DSPYs to stub endpoints that return {'success': False, 'message': '只读'}. This satisfies the spec validator while enforcing read-only at the API level.
backup_accounted_llmusage uses INSERT SELECT for efficiency: The function executes a single INSERT INTO llmusage_history SELECT FROM llmusage followed by a single DELETE FROM llmusage — both with the same WHERE clause (accounting_status='accounted' AND use_date < cutoff_date). Do NOT rewrite as Python loop (sor.C + sor.D per row) — this is orders of magnitude slower for large datasets and lacks atomicity. The cutoff_date is always yesterday (today - 1 day), passed by backend_accounting() on date change.
Indexing for large llmusage/llmusage_history/llmusage_accounting_failed tables: These tables grow rapidly. Essential indexes:

llmusage: (accounting_status, use_date) — for backup SELECT queries
llmusage_history: (use_date), (userid), (userorgid), (llmid), (backup_time) — for historical queries
llmusage_accounting_failed: (handled), (failed_time), (use_date), (userid), (userorgid), (llmid) — for failure record searches and filtering

Accounting failure lifecycle: When backend_accounting() fails to process a record, llm_accoung_failed(luid, reason) does two things: (a) sets llmusage.accounting_status='failed' so the record stays in the active table, and (b) INSERTs a tracking row into llmusage_accounting_failed with retry_count=0, handled='0'. Failed records can be retried via the "重试" button on /llmage/failed_accounting.ui, which calls retry_accounting.dspy to reset accounting_status='created' (picked up by the next loop iteration) and mark the failure record as handled with retry_count+1. Manual handling without retry: mark handled='1', set handled_time/handled_note.
Graceful table-existence detection for migrations: When adding new tables (e.g., llmusage_history, llmusage_accounting_failed) to a live production system, use _table_exists(sor, tablename) probe before querying to prevent crashes before the migration SQL is executed. Pattern: await sor.sqlExe(f"SELECT 1 FROM {tablename} LIMIT 0", {}) wrapped in try-except. Returns True if table exists, False otherwise. Module code should fallback gracefully when the table doesn't exist yet. See references/graceful-migration-patterns.md in the sqlor-database-module skill.
v1 API endpoints require explicit RBAC entries for deep paths AND customer roles: Endpoints under /v1/ subdirectories (e.g., /v1/video/generations/index.dspy) need explicit RBAC permission entries. The parent /llmage logined entry does NOT cover nested paths. Each endpoint needs both the directory and the index.dspy file path registered. Additionally, v1 endpoints are customer-facing (Bearer token auth via dapi) — they need customer.admin and customer.user role permissions, NOT just admin/operator roles. Without customer role entries, customers get 403 even with valid API keys. Missing entries cause 403 errors with no indication in server logs — the request is rejected at the RBAC middleware level before reaching the .dspy file. Direct file endpoints (e.g. v1/models/catelog.dspy) only need the file path entry, not a separate directory entry.
get_llmage_llm() vs get_llm() — architecture after refactor:
- get_llmage_llm(llmid=None, catelogid=None) — lightweight 3-table JOIN (llm + llm_api_map + llmcatelog). Use for display, accounting, pricing, listing. Returns None when called with llmid and not found; returns list when called with catelogid or no args.
- get_llm(llmid, catelogid=None) — calls get_llmage_llm() internally, then adds uapi/uapiio fields via cached lookups (not JOIN). Use when you need input_fields, stream, ioid, callbackurl (i.e., inference/API calls). Returns merged DictObject.
- Refactored from 6-table JOIN to 3-step + cache: Step 1 = get_llmage_llm() (3-table JOIN), Step 2 = _get_uapi_cached(upappid, apiname) (process-level cache, 5min TTL), Step 3 = _get_uapiio_cached(ioid) (process-level cache, 5min TTL). Fields merged into result: llm.ioid, llm.stream, llm.callbackurl, llm.input_fields.
- Cache management: invalidate_uapi_cache(upappid=None, apiname=None) — call when uapi config changes. Specific key invalidation with both args; full flush with no args. Exported via env.invalidate_uapi_cache.
- Pitfall: get_llm() still requires uapi records to exist (via _get_uapi_cached). If a model has no uapi config yet, get_llm() returns None but get_llmage_llm() returns the model data. Use the right one for the context.
- All non-inference code paths (checkCustomerBalance, llm_accounting, llm_query_price) should use get_llmage_llm().
v1 endpoints use backward-compatible SQL matching, BUT llm_api_map migration is incomplete: All v1 endpoints (v1/chat/completions, v1/video/generations, v1/image/generations, t2t/index.dspy, get_type_llms.dspy) use WHERE (b.id = ${catelogid}$ OR b.name = ${catelogid}$) for matching against llmcatelog. External API callers can pass the abbreviation ID (e.g., "t2v") or Chinese name (e.g., "文生视频"). However: llm_api_map.llmcatelogid was NOT updated during migration — it still references old IDs (text2text, text2image, text2video, image2text). This breaks the JOIN chain llm → llm_api_map → llmcatelog for image/vision/TTS catalogs. Only t2t works because some llm_api_map records were individually updated. Fix: UPDATE llm_api_map to use new abbreviation IDs matching llmcatelog.id.
v1 image/video endpoints require catelogid parameter: Unlike /v1/chat/completions (where catelogid defaults to "文生文"), image and video endpoints return 400 "Missing required parameter" if catelogid is omitted. Always include it in test payloads.
API testing: always read key from ~/.hermes/config.yaml: When writing automated test scripts for v1 APIs, read model.api_key from config.yaml programmatically. Never hardcode or manually type the API key — the user corrected this explicitly. Hardcoded keys waste debugging time on auth failures when the real issue is just using the wrong key.
Different downapp keys have different v1 endpoint scopes: The downapikey + downapp auth system means some API keys only work for certain v1 endpoints. A key that successfully calls /v1/chat/completions may return 401 on /v1/image/generations or /v1/video/generations. When all text models pass but all image/video models fail with 401, the issue is likely the key's permission scope, not the API itself.
llmcatelog vs llm_api_map ID migration inconsistency (critical): After the llmcatelog ID standardization, llmcatelog.id was updated to new abbreviations (t2t, t2v, t2i, etc.) but llm_api_map.llmcatelogid still references the OLD IDs (text2text, text2video, text2image, image2text, etc.). This causes JOIN failures: catelogid=t2i matches llmcatelog.id='t2i' but llm_api_map.llmcatelogid='text2image' doesn't JOIN to it. Result: ?catelogid=t2i returns 0 models, while ?catelogid=t2t returns 23 (because t2t happens to exist in llm_api_map too). Fix required: UPDATE llm_api_map SET llmcatelogid to new abbreviations matching llmcatelog.id. Until fixed, image/vision/TTS catalog filters are broken on v1/models endpoint.
uapi data template can be empty — causes upstream "Parameters error": When a uapi record has an empty data template (like the 即梦 tongyi-wan t2i models), the upstream API call sends no request body, resulting in a 400 "Parameters error" from the external provider. The v1 endpoint code itself works correctly — it's the uapi configuration that's incomplete. When debugging 400 errors from v1 endpoints, always check the uapi.data template is populated, not just the v1 endpoint code.
upappkey quota can cause 400/429 from upstream: When the upstream provider's API key (stored in upappkey table) has exhausted its quota, the external API returns 429 "insufficient_quota" which gets passed through to the v1 caller. This is indistinguishable from a parameter error without checking the upstream response details.
Video generation API responses are Python dict strings, not JSON: The async video generation endpoints return responses using Python dict syntax (single quotes), not valid JSON. resp.json() throws JSONDecodeError. Always parse with ast.literal_eval() as fallback. Content-Type header may be text/html even on HTTP 200. This applies to all async inference responses (video, image generation).
API callers must use uapiio input_fields names exactly: When calling /v1/video/generations or /v1/image/generations, the parameter names in the request body must match the name values defined in the model's uapiio.input_fields. For example, if uapiio defines {"name": "image_file", ...}, the API request must use image_file as the key — NOT image_url. The uapi.data Jinja2 template references {{image_file}} and will throw UndefinedError if the field name doesn't match. To discover correct field names: llm → llm_api_map → uapi.ioid → uapiio.input_fields.
Seedance Ref2V requires ratio parameter despite doc saying optional: The doubao-seedance-2-0-260128 model's r2v endpoint returns 400 Bad Request from upstream Volcengine Ark API if ratio is omitted. API.md documents it as optional with default value, but the upstream API actually requires it. Always include ratio (e.g., "16:9") in Seedance Ref2V requests.
Vidu Ref2V has two separate proxy bugs:
- viduq3-pro r2v: All parameter combinations (images array, subjects array, image_file) return "Parameters error" from upstream Vidu API. The uapi.data template's parameter structure doesn't match what Vidu expects.
- viduq2-pro/vidu2.0 r2v: URL path duplication — upapp.baseurl already contains /ent/v2, and uapi.path also includes it, producing malformed URL like https://api.vidu.cn/ent/v2/ent/v2/reference2video.
- viduq3-turbo: Not configured for r2v catelog at all (no llm_api_map entry).
Video API responses use inconsistent status values: Different providers return different status strings — Seedance uses "CREATED", HappyHorse uses "PENDING", Vidu uses "created". Client code should normalize to uppercase for comparison.
API documentation drift — update docs whenever modifying ANY service behavior: This applies to v1 endpoints, v1/media GPU service endpoints, and any service whose external API contract changes. When you modify a service endpoint's parameters (adding task_type, changing response format, adding new modes like separate_full), the API docs MUST be updated in the same session. The user explicitly corrected this: "api文档也也要同步更新". This is NOT optional.\n\n Multiple doc locations: ~/repos/llmage/wwwroot/api_doc.md, ~/repos/llmage/docs/API.md, ~/repos/dashboard_for_sage/wwwroot/api_doc.md. At minimum update dashboard_for_sage/wwwroot/api_doc.md (customer-facing) and llmage/docs/API.md (developer-facing).\n\n Also update the model description in the llm table or API doc model list when adding capabilities (e.g., changing "人声/伴奏分离" to "人声/伴奏分离（支持MIDI乐器替换）").\n\n Public API doc (~/repos/sage/wwwroot/public/api/api_zh.md) was left stale for months — the old doc still referenced llmid (UUID) and /llmage/video instead of model+catelogid and /llmage/v1/video/generations. When adding/changing v1 endpoints, update the API doc in the same session. Always use model (string name) + catelogid (abbreviation) in examples, never llmid (UUID).
DashScope API versioning — wan2.7 uses completely different endpoint and request format: Newer DashScope models (wan2.7 series) use a different API endpoint and request body structure than older models (wan2.2 series). wan2.2: endpoint /api/v1/services/aigc/text2image/image-synthesis, request body {"model": "...", "input": {"prompt": "..."}, "parameters": {...}}. wan2.7: endpoint /api/v1/services/aigc/multimodal-generation/generation, request body {"model": "...", "input": {"messages": [{"role": "user", "content": [{"text": "..."}]}]}, "parameters": {...}}. When integrating new DashScope models, always check the official API documentation for the correct endpoint and request format — do not assume backward compatibility with existing tongyi-wan uapi configurations. The llm.apiname field must point to a uapi record configured for the correct endpoint version. See references/dashscope-api-versioning.md for endpoint comparison and uapi configuration examples.
DashScope status endpoint is shared — do NOT create duplicate status uapi records: DashScope's task query endpoint GET /tasks/{task_id} is unified across ALL models under the same provider (upappid=tongyi-wan). Whether the task was created via /text2image/, /multimodal-generation/, or /image-generation/, the status query is always the same. If a status uapi (e.g., wan-t2i-status) already exists for the upappid, reuse it in llm_api_map.query_apiname. Creating per-model status uapis wastes records and causes confusion.
Prefer sync mode when DashScope supports it: Official docs explicitly recommend sync ("一次请求即可获得结果，流程简单，推荐大多数场景使用"). Sync uses stream='sync' in uapi and eliminates the need for status polling (no query_apiname needed in llm_api_map). Only use async for genuinely long-running tasks where sync would timeout.
Async task polling stuck at CREATED after schema migration: After moving query_apiname and query_period from llm to llm_api_map, the get_llm_llmusage() function in asyncinference.py still used sor.R('llm', {'id': llmid}) — a direct table read that only returns llm table fields. This caused llm.query_apiname to be None, crashing query_task_status() with an AttributeError that was silently swallowed, leaving llmusage.status permanently stuck at "CREATED". Fix: Replace sor.R('llm', ...) with a JOIN query: SELECT a.*, m.query_apiname, m.query_period FROM llm a JOIN llm_api_map m ON a.id = m.llmid WHERE a.id = ${llmid}$ AND m.isdefaultcatelog = '1'. General rule: When schema migration moves fields between tables, scan ALL code paths (not just the ones you're actively working on) for direct table reads (sor.R) that need updating. Background tasks like polling are especially vulnerable because errors may be silently caught.
"Model exists but doesn't work" diagnostic checklist: When a model is registered in the llm table but fails during inference or doesn't appear in category listings, check these in order:
1. llm_api_map entry exists? — SELECT * FROM llm_api_map WHERE llmid = '<llm_id>'. Without this, the model has no API capability mapping and won't appear in get_llmage_llm() results.
2. uapi record matches the model's API version? — llm_api_map.apiname must point to a uapi record whose path and data template match the provider's current API format. New model versions (e.g., wan2.7 vs wan2.2) may need entirely new uapi records.
3. upappkey exists for the provider? — SELECT * FROM upappkey WHERE upappid = '<upapp_id>'. Without API credentials, calls will fail.
4. llm.status = 'published'? — Unpublished models are invisible to users.
5. llm_api_map.llmcatelogid matches llmcatelog.id? — After the ID standardization, old IDs (text2image, text2video) won't JOIN to new abbreviation IDs (t2i, t2v).
Async task response uses taskstatus field, NOT status: When submitting an async image/video generation request, the response body is {"taskid": "xxx", "taskstatus": "PENDING"} — note the field name is taskstatus, not status. The /v1/tasks query endpoint also returns taskstatus inside data. The API doc previously showed status and an outer {"status":"ok","data":{...}} wrapper for submit — this was wrong. Actual submit response is direct (no wrapper): {"taskid":"...","taskstatus":"PENDING"}. Task query wraps in {"status":"ok","data":{"taskid":"...","taskstatus":"PENDING"}}. When SUCCEEDED, the data block contains image/output fields directly. Always use taskstatus when parsing async responses programmatically.
Sync inference responses have NO outer wrapper: When a uapi record has stream='sync', the response returned to the caller is the uapi.response Jinja2 template output directly — there is NO {"status":"ok","data":{...}} envelope. For image generation sync models, the client receives exactly: {"status":"SUCCEEDED","usage":{...},"image_count":N,"image":[...]}. Async models DO have a wrapper on submit ({"status":"ok","data":{"taskid":"..."}}) but the final result from /v1/tasks query is also direct. Never add a wrapper layer in API documentation response examples for sync models. This was corrected by the user three times in one session — zero tolerance.
Token redaction breaks file writes containing Bearer auth patterns: The system redacts strings matching ***...*** or Bearer *** patterns in transit. When writing files (Python scripts, HTML, markdown) that contain Bearer token placeholders like Authorization: Bearer ***, the redaction strips the pattern mid-string, leaving unterminated string literals, broken template literals, or truncated code. Fix: When generating files with Bearer auth examples via Python scripts, construct ALL strings containing the token pattern entirely from chr() calls. Example: TK = chr(42)*3 + 'API_KEY' + chr(42)*3 then AUTH = SQ + ''.join([chr(c) for c in [65,117,...]]) + TK + SQ. Never write the literal token pattern as a string constant in the script file itself — even in comments, since the write_file tool scans the entire content. This pitfall applies to any file generation task involving API key examples, not just llmage docs.
Always verify response formats with actual API calls before documenting: API documentation response examples MUST be tested with real curl calls before being written to docs. Previously documented formats were wrong for months — usage fields, error shapes, URL domains were all invented/approximated rather than tested. Workflow: 1) curl the endpoint with a real request, 2) capture the exact JSON response, 3) use that as the doc example (redact only sensitive tokens). Different providers return different response structures (e.g., wan2.7 usage has tokens, qwen usage has dimensions; error responses have two distinct shapes). Never assume uniformity across providers.
API doc changes must be holistic — docs + database + code together: When changing API documentation (e.g., removing async mode, changing response formats), you MUST also check and update the underlying database configuration. User corrected this explicitly: "这里修改了话，你还要检查之前的sql中的sql的变更". Checklist for any API doc change: (a) Update api_doc.md text, (b) Verify actual API response matches new doc via curl, (c) Check uapi.stream field — must match sync/async claim in docs, (d) Check llm_api_map.query_apiname — must be NULL for sync models, populated for async, (e) Check uapi.data template — request body format must match docs, (f) If changing sync↔async, provide SQL migration statements for the user to execute on production. Never just edit the markdown and stop — the doc describes behavior controlled by database state, not by the text itself.
Filtering llmusage by catalog requires llm_api_map subquery (NOT llm table): When filtering llmusage or llmusage_history records by llmcatelogid, you CANNOT query the llm table — it has no llmcatelogid column. The catalog relationship is in llm_api_map. Correct pattern:

WHERE llmid IN (SELECT llmid FROM llm_api_map WHERE llmcatelogid = ${llmcatelogid}$)

Wrong pattern (causes SQL error 1054 "Unknown column"):

WHERE llmid IN (SELECT id FROM llm WHERE llmcatelogid = ${llmcatelogid}$)  -- WRONG: llm table has no llmcatelogid

This mistake is easy to make because the schema has migrated multiple times (llm_catalog_rel → llm → llm_api_map). Always check the table structure before writing the subquery. See models/llm.json and models/llm_api_map.json for current field lists.

Cross-table UNION ALL pagination for llmusage + llmusage_history: When querying historical inference records that span both active (llmusage) and archived (llmusage_history) tables, use UNION ALL with a subquery for pagination. The count query wraps the UNION in a subquery; the data query adds ORDER BY + LIMIT/OFFSET on the outer query. Both tables share the same schema (same columns). Example pattern:

-- Count
select count(*) as cnt from (
    select id from llmusage where userid = ${userid}$
    union all
    select id from llmusage_history where userid = ${userid}$
) t

-- Paginated data
select id, llmid, use_date, use_time, userid, usages, ioinfo, status, ...
from (
    select id, llmid, use_date, use_time, userid, usages, ioinfo, status, ...
    from llmusage where userid = ${userid}$
    union all
    select id, llmid, use_date, use_time, userid, usages, ioinfo, status, ...
    from llmusage_history where userid = ${userid}$
) t
order by use_time desc
limit {page_size} offset {offset}

ioinfo reading: The ioinfo column stores a webpath string (e.g., /llmio/182/138/79/46/xxx.json), NOT the actual JSON. To read the content in a .dspy file:

from ahserver.filestorage import FileStorage
import aiofiles
fs = FileStorage()
real_path = fs.realPath(webpath)
async with aiofiles.open(real_path, 'rb') as f:
    bin_data = await f.read()
io_content = json.loads(bin_data.decode('utf-8'))

Wrap in try/except — webpath may be None or file may not exist. The parsed JSON has input (dict) and output (list) keys. usages field may be a JSON string that needs json.loads() too.

Non-inference v1 endpoints (pricing, etc.) use direct SQL ppid lookup, not get_llm(): When building a v1 endpoint that queries model metadata (not inference), use a direct SQL JOIN to get ppid from llm_api_map, not get_llm() which requires uapi records. Pattern:

sql = """select m.ppid from llm a
join llm_api_map m on a.id = m.llmid
where a.model = ${model}$
    and a.status = 'published'
    and m.ppid is not null
    and m.isdefaultcatelog = '1'
"""
recs = await sor.sqlExe(sql, {'model': model})
ppid = recs[0].ppid

Then call the appropriate env function (e.g., env.get_pricing_display(ppid)). Use request._run_ns to get env, and get_sor_context(env, 'llmage') for DB access. Return json.dumps() directly. The /v1/pricing endpoint follows this pattern.

apiname is in llm_api_map, NOT in llm — common SQL JOIN error: The llm table does NOT have an apiname column. It lives in llm_api_map. When writing SQL that joins llm → uapi, you MUST go through llm_api_map:

-- ✅ CORRECT — apiname from llm_api_map (alias m)
select a.*, e.ioid, e.stream
from llm a
join llm_api_map m on a.id = m.llmid
join upapp c on a.upappid = c.id
join uapi e on c.id = e.upappid and m.apiname = e.name
where a.id = ${llmid}$

-- ❌ WRONG — llm table (alias a) has no apiname column
select a.*, e.ioid, e.stream
from llm a
join upapp c on a.upappid = c.id
join uapi e on c.id = e.upappid and a.apiname = e.name
where a.id = ${llmid}$

Error: (1054, "Unknown column 'a.apiname' in 'ON'") or similar. This mistake appeared in 3 files (4 occurrences) before being caught — list_paging_catelog_llms.dspy, llmcheck.dspy (2x), api/llm_launch_check_api.dspy. When grepping for this pattern, search for a.apiname across all dspy files.

list_paging_catelog_llms.dspy must handle missing llmid: The SQL where x.id != ${llmid}$ hardcodes llmid, causing (1054, "Unknown column 'llmid'") when the endpoint is called without llmid (e.g., direct URL access). Fix: Use conditional SQL:

llmid = params_kw.get('llmid')
if llmid:
    sql += " and x.id != ${llmid}$"

This pattern applies to any .dspy file where optional parameters are used in WHERE clauses.

Multi-catelog model selection — UI must present catelog tabs: A single model can have MULTIPLE llm_api_map entries (e.g., qwen-max has both t2t and vision catelogs). The llm_dialog.ui template MUST handle this by:
Calling get_llm_catelogs(llmid) to get ALL catelog entries
If len(catelogs) > 1, rendering an HBox with Button widgets per catelog
Default selection = the one with isdefaultcatelog='1', or first entry if none is default
Each tab button links to ?id={{llmid}}&catelogid={{catelogid}} to reload the UI
Passing llmcatelogid to BOTH the LlmIO models[] array AND the list_models_url query string
The bricks LlmModel widget sends llmcatelogid in inference requests automatically

Do NOT just pick one catelog silently — the user must choose which capability to use. get_llmage_llm(llmid) without catelogid defaults to isdefaultcatelog='1', which is correct for the default tab but wrong if the user wants a different catelog.

Jinja2 scoping: When computing active_catelogid inside a {% for %} loop, {% set %} does NOT propagate out of the loop. Use {% set ns = namespace(active_catelogid=...) %} — see bricks-framework skill "LlmIO Widget" section for the correct/incorrect patterns.

Bricks side: bricks.LlmModel.inputdata2uploaddata() includes this.opts.llmcatelogid in both FormData and JSON request bodies. bricks.LlmIO.open_search_models() passes this.models[0].llmcatelogid to list_paging_catelog_llms.dspy so related models are filtered by the same catelog.

list_paging_catelog_llms.dspy must return llmcatelogid: Set r.llmcatelogid = llmcatelogid in the response loop so models added from the search popup carry the correct catelog for inference.

Both get_llms_by_catelog() and get_llms_sort_by_provider() include pricing display data: Both functions batch-query ALL distinct ppid values for each model (from llm_api_map via a single IN (...) query), call env.get_pricing_display(ppid) for each, and store display_text in llm.pricing_display (a list of strings). Both templates (show_llms_cards.ui and show_llms_cards_by_provider.ui) render pricing text in a Filler with css: "pricing-box". Card height must be cheight: 16 (not 12) to fit pricing info. Both functions handle get_pricing_display returning None gracefully (if pd: pricing_list.append(...)). N+1 query pitfall: Always use a single batch SELECT DISTINCT llmid, ppid FROM llm_api_map WHERE llmid IN (...) query to build a pp_map dict, then iterate per-model. Never query ppid per-model inside the loop.
llminference.dspy MUST check checkCustomerBalance return value: The inference API entry point calls checkCustomerBalance() but historically the return value f was assigned and never checked — the code continued to inference() regardless. The UI layer (llm_dialog.ui) gates display on the same check, but direct API calls bypass the UI. Fix (mandatory in llminference.dspy):

f = await checkCustomerBalance(params_kw.llmid, userid, userorgid)
if not f:
\treturn UiError(title='llm inference', message='余额不足或模型未配置定价')

Owner org bypass: checkCustomerBalance returns True immediately when llm.ownerid == userorgid (self-owned models skip balance/pricing checks). This is by design for internal usage but means models without pricing are usable by their owner. General rule: Every .dspy inference/API endpoint that calls a permission/balance check MUST verify the return value before proceeding. Assigning to a variable without checking is a silent security gap.

get_pricing_display returns None for missing pricing data: When PricingProgram.get_ppid_pricing(ppid) finds no pricing record for a given ppid+date, it raises an exception. get_pricing_display wraps this in try/except and returns None. Callers MUST check: if pd: pricing_list.append(pd.get('display_text', '')). This prevents exception log spam when card pages iterate over all models (many may lack pricing).
test_pricing type mismatch after ppt_db2app: get_pricing_program_timeing(pptid) calls ppt_db2app(ppt) which converts ppt.pricing_data from YAML string to Python dict. But get_pricing_from_ymalstr(data, yamlstr) expects a YAML string. Fix: Check type and re-serialize:

yamlstr = yaml.dump(ppt.pricing_data, allow_unicode=True) if isinstance(ppt.pricing_data, dict) else ppt.pricing_data

This pattern applies anywhere ppt_db2app output feeds into a YAML-string-expecting function.

External GPU service registration — uapi records MUST be complete: Multi-endpoint ahserver services on GPU need ONE llm model per endpoint (unique constraint on llm_api_map). Do NOT create skeleton uapi records with only id/name/path/upappid — inference will fail silently because get_llm() caches uapi records and expects stream/data/response/ioid to be populated. Missing response = no usage in output = billing chain broken.

Registration order: upapp → uapiio → uapi (with stream/data/response/ioid/headers) → upappkey → llm → llm_api_map → pricing_program → pricing_program_timing.

uapi response template: Use {%if error%}...{%else%}...{%endif%} blocks for GPU error pass-through. Jinja2 {{var or ""}} does NOT handle undefined variables — only {%if %} works in Sage's environment. Async submit responses skip usage; status query responses include it.

ppid per model: Each model needs its own ppid (e.g. pp_ktv_asr). Sharing one ppid across models causes billing mismatches because llm_charging iterates all pricing_timing under that ppid.

pricing YAML: price_factors: flat = fixed fee (usage_value=1). price_factors: <fieldname> = per-unit billing where the field comes from the usage JSON. enabled_date must be set (not NULL).

GPU parameter discovery: GPU services may expect different param names than API docs. Query each endpoint directly to discover actual names, then map in uapi data template.

See references/ktv-service-registration-pattern.md for the complete worked example.

uapi response template design for GPU services: Use pass-through pattern — GPU service returns usage directly in its JSON response, template passes it through with {{json.dumps(usage, ensure_ascii=False)}}. Never hardcode usage field names (e.g. audio_seconds) in the template; let the GPU service own the pricing factor semantics. Async submit responses (stream='async') do NOT need usage; the status query endpoint's response template does.

uapi stream classification: Sync (stream='sync') for quick operations; Async (stream='async' + llm_api_map.query_apiname pointing to a status-query uapi) for long GPU tasks.

New v1 endpoint for custom catalogs: When models use a non-standard catelogid (e.g. ktv_pipeline), create a dedicated endpoint (e.g. /v1/pipeline/submit/index.dspy) that hardcodes the catelogid. Pattern matches /v1/video/generations but without requiring prompt as mandatory. Update all 3 doc locations (llmage docs/API.md, dashboard_for_sage api_doc.md, llmage wwwroot/api_doc.md if it exists). KTV has 13 models (synth-generate removed 2026-07-06 — 产线内部调 LLM 生成合成指令，不对外暴露；saved 12 sync + 1 async). If a model is no longer needed externally, remove its llm/llm_api_map/uapi/pricing records but keep the corresponding uapi for internal use if needed.

See references/ktv-service-registration-pattern.md for the complete worked example, references/gpu-parameter-discovery.md for GPU parameter name mapping, references/uapi-response-error-handling.md for Jinja2 error handling, references/pipeline-app-api-adapter.md for the env-var-driven pipeline-app adapter pattern with model selection + cost estimation, references/gpu-demucs-longtasks-pattern.md for the async submit+status longtasks/Redis pattern used by GPU demucs service, and scripts/verify-ktv-migration.py for migration integrity checks (updated to 13 models).

Provider table is modelprovider, NOT llmprovider: Schema: id, name, label, description. The name llmprovider does not exist.
llm.enabled_date and expired_date are REQUIRED for API visibility: Models missing these don't appear in /v1/models. Set enabled_date='2025-01-01', expired_date='2099-12-31' for new models.
ownerid='0' for platform-public models: Org-specific ownerid hides models from other orgs. Set to '0' for cross-org visibility.

Cache keyed by {ppid}.{date}, Sage restart clears.

Sage SQL scripts belong in the sage repo, NOT foms

58b. Every uapi response MUST carry "usage" for billing engine: The billing pipeline reads usage from API responses and matches pricing_data fields. Example: {"status":"ok","usage":{"audio_seconds":120}} paired with pricing YAML using audio_seconds as factor.

Sage SQL scripts belong in the sage repo, NOT foms
uapi response Jinja2 — only {%if error%} handles undefined variables reliably: GPU errors like {"error":"..."} lack expected fields → jinja2.exceptions.UndefinedError → empty 200 response. {{var or ""}} fails (Jinja2 or checks truthiness, not definition). default() filter unavailable in Sage. See references/uapi-response-error-handling.md.
upappkey ownerid must be real user ID: get_calluserid() JOINs upappkey→users→upapp on ownerid/orgid. ownerid='0' fails. Use actual user ID (e.g. UiEi7hKqAmU1-jqQEVhZe). Sage restart required.
pricing_program_timing.enabled_date cannot be NULL: checkCustomerBalance fails if no pricing covers current date → 429.
v1/models requires ppid IS NOT NULL: get_llms_by_catelog_to_customer() filters m.ppid is not null. Set ppid for all llm_api_map records.
Param naming: _file not _url: File parameters use _file convention matching Sage's image/video/audio endpoints. _url confuses programmers who expect it to be a text prompt. Applies to uapiio, uapi data templates, and API docs.
Regression test after any llmage change: Call all existing v1 endpoints + new endpoints via tokentest API. Verify accounting. User: "修改后应该做回归测试的，不能省步骤".
Async submit response MUST contain both taskid and status fields: The asyncinference module reads d.taskid and d.status from the submit result via DictObject(**json.loads(b)). If the uapi submit response template only returns taskid/taskstatus but not status, d.status is undefined and the async task lifecycle breaks. Correct: {"taskid":"{{task_id}}","taskstatus":"PENDING","status":"PENDING"}. Additionally, the status query endpoint's response must use key status (not taskstatus) because asyncinference checks new_output.get('status'). The first poll also accesses lastoutout['status'] (bracket access, NOT .get()) which raises KeyError if ioinfo has no output yet — this is handled by ensuring the submit step writes the initial ioinfo with {"status":"PENDING"}. See references/async-demucs-debug-trace.md for the full debug trace including path template {{taskid}} matching, ioinfo KeyError, and venv contamination from pip -e.
GPU submit endpoint MUST use longtasks-returned task_id: longtasks.submit_task(payload) generates a 32-char task_id via getID() and returns {'task_id': '...'}. Endpoints that generate their own 12-char UUID and ignore the return create mismatched IDs → status queries return "not_found" or "no task". Found in demucs, realesrgan, and ASR services. Fix: result = await longtasks.submit_task(payload); task_id = result['task_id']. Additionally, task_type in the submit payload must match what the worker's process_task() dispatches on — mismatch (e.g., 'upscale' vs 'upscale_video') causes Unknown task_type error. See references/gpu-demucs-longtasks-pattern.md.
Worker return status MUST be uppercase: Sage asyncinference lifecycle compares new_output['status'] against 'SUCCEEDED' and 'FAILED'. GPU workers returning lowercase 'success'/'failed'/'ok' break the lifecycle — task stays PENDING forever. Found in demucs, realesrgan, and ASR. Fix all worker return dicts to use uppercase. Also add "usage" field to worker output for billing chain: "usage": {"audio_seconds": round(duration, 3)}.
Pipeline-app venv contamination: Installing pipeline-app pkgs via pip -e into Sage's venv creates editable installs (__editable__*.pth finders) that map packages like ahserver to pipeline-app's copy. This silently overrides Sage's own ahserver, breaking dspy globals (getID not defined) and causing 500s across all v1 endpoints. Fix: (a) Run pipeline-app separately with its own PYTHONPATH, never install into Sage's venv; (b) If contamination occurs, remove __editable__* files from Sage's site-packages/, remove any copied pipeline_* directories, and restore missing core packages (ahserver, sqlor, appbase, rbac) from pipeline-app's pkgs/ directory by copying their source folders back.
GPU ahserver endpoints are DIRECTORIES with index.dspy, NOT .dspy files: The ahserver router maps /api/xxx → app/api/xxx/index.dspy (directory route). Creating app/api/xxx.dspy (file) produces "invalid path" error. Every new GPU endpoint needs: mkdir -p app/api/<endpoint-name> + create index.dspy inside it.
ASR language code must be ISO 639-1, NOT 'auto': faster_whisper requires explicit language codes (zh, en, ja, etc.). Passing 'auto' as language causes ValueError: 'auto' is not a valid language code. Hardcode to the expected language in the submit endpoint: 'language': params_kw.get('language', 'zh'). This was found after successive GPU service restarts and Python venv changes — the error message is verbose (lists all 99 accepted codes) and appears in the Redis task result under error field. GPU service Python: ASR on port 9925 uses /data/ymq/aligner/py3/bin/python3 which has both ahserver and faster_whisper — other venvs (system Python, demucs_venv) may be missing one or both. Kill old PID before restart: ss -tlnp sport = :9925 to find PID.
{{model}} in uapi data template sends Chinese name → upstream API 400: uapi data 中 "model": "{{model}}" 渲染为中文名（如 "快乐马-1.0-文生视频"），但 Alibaba DashScope 等上游期望英文 identifier（"happyhorse-1.0-t2v"）。排查：grep 'model=' /d/apitest/sage/logs/sage.log | tail -3 看实际发出的 HTTP body。修复：硬编码英文名：UPDATE uapi SET data = REPLACE(data, '\"model\": \"{{model}}\"', '\"model\": \"happyhorse-1.0-t2v\"')。影响所有 tongyi-wan 新模型。发现方法：不在 llm 表里搜 /v1/models 返回的模型名，而是用 SELECT name,model FROM llm WHERE model LIKE '%happyhorse%' 查 model 列（英文 identifier），再用该 name 调 API。
apiname is in llm_api_map, NOT in llm — common SQL JOIN error: The llm table does NOT have an apiname column. It lives in llm_api_map. When writing SQL that joins llm → uapi, you MUST go through llm_api_map:

-- ✅ CORRECT — apiname from llm_api_map (alias m)
select a.*, e.ioid, e.stream
from llm a
join llm_api_map m on a.id = m.llmid
join upapp c on a.upappid = c.id
join uapi e on c.id = e.upappid and m.apiname = e.name
where a.id = ${llmid}$

ASR singing voice timing offset — all word starts are ~0.3s too early: faster_whisper timestamps trigger on sound onset, not vocal peak. For singing voice, every word starts before the actual sung syllable. Fix: apply global +0.3s offset to all word timestamps before generating ASS subtitles. This is systematic, not progressive drift. Also, ASR produces noise segments ("好啊", filler sounds) and can merge adjacent lyric lines → need manual SEG_TO_LYRIC calibration to skip garbage. See references/ass-karaoke-subtitles.md for patterns.
Seedance/video generation idfile URLs expire quickly — download immediately: Async video tasks return a video URL (e.g., http://127.0.0.1:9180/idfile?path=/tmp/...). These temp files may be cleaned within minutes. Poll for SUCCEEDED status, then download the video file IMMEDIATELY in the same script. Consider re-submitting if the download fails with 404. The video field (not output_video_url) holds the URL in task query response data.: pipeline-app 的 ahserver 污染后，dspy 预加载 globals 中 getID 不可用 → NameError。临时修复：sed -i 's/getID()/str(__import__("uuid").uuid4()).replace("-","")/' /d/apitest/sage/wwwroot/llmage/v1/*/generations/index.dspy。根治：不要将 pipeline-app 包安装到 Sage venv（见 Pitfall 75）。
Worker return status MUST be uppercase: Sage asyncinference lifecycle compares new_output['status'] against 'SUCCEEDED' and 'FAILED'. GPU workers returning lowercase 'success'/'failed' break the lifecycle — task stays PENDING forever. Fix all worker return dicts to use uppercase status values. Also add "usage" field to worker output for billing chain.
Pipeline-app venv contamination: Installing pipeline-app pkgs via pip -e into Sage's venv creates editable installs (__editable__*.pth finders) that map packages like ahserver to pipeline-app's copy. This silently overrides Sage's own ahserver, breaking dspy globals (getID not defined) and causing ModuleNotFoundError for Sage-only packages. Fix: (a) Run pipeline-app separately with its own PYTHONPATH, never install into Sage's venv; (b) If contamination occurs, remove __editable__* files from Sage's site-packages/, remove any copied pipeline_* directories, and restore missing core packages (ahserver, sqlor, appbase, rbac) from pipeline-app's pkgs/ directory by copying their source folders back.
GPU ahserver endpoints are DIRECTORIES with index.dspy, NOT .dspy files: The ahserver router maps /api/xxx → app/api/xxx/index.dspy (directory route). Creating app/api/xxx.dspy (file) produces "invalid path" error. Every new GPU endpoint needs: mkdir -p app/api/<endpoint-name> + create index.dspy inside it.
DSPY handler 硬编码 payload 导致新功能静默失败：GPU 服务的 API handler 在 app/api/{endpoint}/index.dspy 中构建 payload 传给 longtasks worker。DSPY 是 hot-reload 的无需重启，但若 handler 只构建固定字段（如 payload = {'task_type': 'separate'}），即使 worker 已支持新 task_type 也会静默回退默认模式而不报错。排查：对比 worker log 中的 task.payload 与 curl 请求体。修复：透传 params_kw 中所有业务字段。详见 references/ktv-service-registration-pattern.md Pitfall 节。
llmusage.id 格式判断 — 区分标准推理记录与手动插入记录：getID() 生成 21 字符 nanoid（不是 32 字符 UUID）。llmusage 模型定义 VARCHAR(32)，两种长度都能存。标准推理流程（_inference_generator → sync_uapi_request/async_uapi_request/uapi_request）写入的特征：id=21-char nanoid + ioinfo=非空 webpath + transno=非空（来自 DSPY 的 uuid 或 getID）。非标准记录特征（手动 SQL 插入/测试脚本）：id=32-char UUID hex + ioinfo=NULL + transno=NULL。KTV 14 个 v1/media 端点全部走标准 inference() 路径，不存在"自己写推理逻辑"。排查时先看 ioinfo 和 transno 是否为空——双空即是手动插入。
product_accounting 替代 llm_accounting：product_management/core.py 的 product_accounting() + backend_accounting() 已替代 llmage/accounting.py 的 llm_accounting()。新流程走分销链（product_owner → ... → seller），每条 DictObject 会计条目带 currency + base_amount。汇率实时查询通过 accounting/exchange.py 的 get_exchange_rate(from, to, rate_type) 函数——支持正向和反向回退查找。修改计费逻辑时改 product_management，不要改已废弃的 llm_accounting。
CRUD logined_userorgid 会隐藏平台级记录：json/*.json CRUD 配置中的 logined_userorgid: "ownerid" 会自动添加 WHERE ownerid = <当前用户orgid>。若数据 ownerid='0'（平台级），当前用户 org 不是 '0' → 列表不显示，搜索也无结果。修复：删除 logined_userorgid 行，或将数据 ownerid 设为与用户匹配的 org。同时检查 data_filter 中 op: "=" 的字段，空值时可能生成 WHERE field = '' 也过滤掉所有结果，改为 op: "LIKE"。此问题导致 pricing 模块 KTV 定价（pp_ktv，ownerid 原为 NULL，补设为 'provider'）搜索不到。
asyncinference 不会自动调 llm_charging：asyncinference.py 的 query_task_status() 在任务 SUCCEEDED 后保存 usages 但不计算 amount/cost。导致 llmusage.amount=NULL，backend_accounting()→llm_accounting() 无法记账。症状：usages 有值但 amount 为 NULL，accounting_status 停在 created。修复：在 modify_llmusage(ns) 后、返回前加 llm_charging 调用（注意 tab 缩进），详见 references/asyncinference-llm-charging-fix.md。
uapi response 模板引号被 MySQL 吞掉：UPDATE uapi SET response = '{\"status\"...}' 中 \" 被 MySQL 解释为转义后吃掉。结果模板变成 {status:...}（无效 JSON）→ Jinja2 渲染后 json.loads() 失败。修复：用 heredoc SQL 文件。详见 references/uapi-template-json-quoting-pitfall.md。
GPU longtasks 返回嵌套 result：RUNNING 时无 result 键，SUCCEEDED 时 result 含 segments/usage。模板须用顶层 {%if status == "SUCCEEDED"%} 做守卫，不能 {%if result.status%}（result 未定义→UndefinedError）。详见 references/uapi-template-json-quoting-pitfall.md。
uapiio.input_fields 格式必须是数组，不能是对象：bricks LlmIO 构造函数调用 this.input_fields.forEach(...)，要求 input_fields 是 JSON 数组字符串。若 uapiio 中存为对象格式 {"field_name":{"type":"string",...}}，JSON.parse 后是普通对象，.forEach 不可用 → TypeError: this.input_fields.forEach is not a function。正确格式：[{"name":"field_name","label":"标签","uitype":"text","required":true}]。uitype 取值：text（文本输入）、audio/video/image（文件上传）、file（通用文件）。症状：llm_dialog.ui 加载模型时前端 js 报错，模型选择卡片不显示。修复：UPDATE uapiio SET input_fields = '<array-json>'，重启 Sage 清 process-level 缓存。
定价显示币种 — get_ppid_pricing 必须查 a.currency：PricingProgram.get_ppid_pricing() SQL 中未 select a.currency 导致 get_pricing_display() 无法获取币种信息，模型卡片始终显示"元"。修复：SQL 加 a.currency，get_pricing_display() 中从 record 取 currency 并附加到 display_text 首行（如 【KTV产线计费】定价: ($)）。currency_symbols 映射：CNY→元, USD→$, JPY→¥, GBP→£。修改后需 Sage 重启（process-level pricing 缓存 key 为 {ppid}.{date}）。
product_accounting 已替代 llm_accounting：product_management/core.py 的 product_accounting() + backend_accounting() 已接管 llmage 的记账。新记账走分销链，每个 DictObject 会计条目带 currency + base_amount。汇率实时查询通过 accounting/exchange.py。修改计费逻辑时改 product_management，不要改已废弃的 llmage/accounting.py 中的 llm_accounting()。
pricing 币种显示修复三步：① get_ppid_pricing SQL 加 a.currency → ② get_pricing_display 从 record 取 currency 附加到 display_text → ③ 替换 YAML unit_label 中硬编码的"元"为正确币种符号。修改后定价缓存需失效（key={ppid}.{date}，Sage 重启自动清）。缺失任一步都会导致卡片仍显示"元"。
uapiio input_fields 格式修复脚本模式：当所有 uapiio 记录都是错误的对象格式 {field: {type, required}} 时，用 Python 脚本批量转换并输出 UPDATE SQL：for ioid, fields in records: arr = [{"name": k, "label": v.description, "uitype": guess_uitype(k), "required": v.required} for k,v in fields.items()]; UPDATE uapiio SET input_fields='{json}'。uitype 推断：含 audio→audio, video→video, image→image, file→file, 其他→text。执行后 Sage 重启清 process-level uapiio cache。
Schema migration MUST run BEFORE code that references new columns: Adding a column to a model JSON and writing code that queries it does NOT make the column exist in the database. The migration SQL (ALTER TABLE) must be executed on the target database BEFORE deploying code that references the new column. Otherwise: OperationalError: (1054, "Unknown column 'a.currency' in 'SELECT'"). This applies to all multi-module changes (accounting, pricing, llmage, product_management). Checklist: (a) Write migration SQL with all CREATE TABLE + ALTER TABLE statements, (b) Execute on test DB first, (c) Verify column exists with SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_NAME='tablename', (d) Deploy code, (e) Restart Sage.
Exchange rate data must be inserted after table creation: The exchange_rate table is created by migration SQL, but the actual rate data (USD/CNY, JPY/CNY, GBP/CNY buy/sell/mid prices) must be INSERTED separately. Use legitimate sources (Bank of China https://www.boc.cn/sourcedb/whpj/). Buy rate < sell rate (platform spread).
pricing_program CRUD 搜索不到记录: 两个常见原因 — (a) logined_userorgid: "ownerid" 过滤掉了其他机构所有的定价（如 KTV 定价 ownerid='0'），需移除；(b) data_filter 中 op: "=" 空值时可能生成 WHERE field = ''，改为 op: "LIKE"。
uapiio.input_fields 必须是数组格式: bricks LlmIO 调用 this.input_fields.forEach()，对象格式 {"field":{...}} 会报 TypeError: ...forEach is not a function。正确格式: [{"name":"field","label":"标签","uitype":"text","required":true}]。
定价显示币种: get_ppid_pricing() SQL 加 a.currency，get_pricing_display() 用 currency_symbols 替换 YAML unit_label 中硬编码的"元"。三步缺一不可。
providerid dropdown 必须查询 suppliers 表而非 organization：models/llm.json 中 codes.table = supplychain.suppliers (valuefield=id, textfield=supplier_name)。CRUD JSON llm.json 的 browserfields.alters.providerid 指向 get_search_providerid.dspy，该 dspy 必须查询 suppliers 表（select id as providerid, supplier_name as providerid_text from suppliers where status='1'），不能查询 organization 表。使用 get_sor_context(request._run_ns, 'rbac') 连接 sage 数据库。同一模式适用于 pricing_program.providerid（pricing 模块的 CRUD JSON 也需配置 alters）。
v1 endpoint RBAC: directory AND file path both need rolepermission: Sage checks /llmage/v1/audio/transcriptions (directory) at RBAC level, not just the index.dspy file. Register BOTH paths: INSERT INTO rolepermission ... WHERE path='/llmage/v1/audio/transcriptions' AND ...WHERE path='/llmage/v1/audio/transcriptions/index.dspy'. Missing directory entry = 403 even with valid Bearer token.
llm_api_map.llmcatelogid mismatch with API catelogid: When the API caller passes catelogid=asr but the model's llm_api_map.llmcatelogid is ktv_pipeline (or any non-matching value), the v1 endpoint SQL JOIN returns empty → "model not found" or "Parameters error". Fix: UPDATE llm_api_map SET llmcatelogid='asr' to match the standard catelog ID. Then restart Sage to invalidate caches. Check all models under each catelog after any catalog migration.
Bearer token key creation for testing: Generate: from ahserver.globalEnv import password_encode; enc = password_encode(plaintext). INSERT into downapikey with dappid='MF2QQSJ0FtBOeVv2rNoGM' (test app). allowedips=NULL → no IP restriction. Auth flow: client sends Bearer <plaintext> → Sage password_encode() → lookup downapikey.apikey → JOIN downapp.
Async GPU status query uapi: nested result paths + Jinja2 guards: When the GPU task status endpoint returns data nested under result (e.g., {"status":"RUNNING","result":{"status":"SUCCEEDED","usage":{...}}}), the uapi response template must access result.status, result.usage, etc. Template like {"status":"{{result.status}}","usage":{{json.dumps(result.usage)}}} fails with Jinja2 UndefinedError when task is RUNNING (no result yet). Fix with conditional: {"status":"{{status}}"{%if result.status == "SUCCEEDED"%},"usage":{{json.dumps(result.usage,ensure_ascii=False)}}{%endif%}}. Use top-level status for lifecycle (RUNNING/SUCCEEDED/FAILED), nested result.* only for final data. Without this guard, asyncinference polls get corrupted → tasks marked FAILED despite GPU success.
v1 endpoint RBAC: directory AND file path both need rolepermission: Sage checks /llmage/v1/audio/transcriptions (directory) at RBAC level, not just the index.dspy file. Register BOTH paths: INSERT INTO rolepermission ... WHERE path='/llmage/v1/audio/transcriptions' AND ...WHERE path='/llmage/v1/audio/transcriptions/index.dspy'. Missing directory entry = 403 even with valid Bearer token.
llm_api_map.llmcatelogid mismatch: When API catelogid parameter doesn't match the model's llm_api_map.llmcatelogid, the v1 endpoint SQL JOIN returns empty → "model not found" or "Parameters error". Common: model under ktv_pipeline but caller uses catelogid=asr. Fix: UPDATE llm_api_map SET llmcatelogid='asr' to match the standard catelog ID. Then restart Sage to invalidate caches.
Bearer token key creation for testing: Generate: from ahserver.globalEnv import password_encode; enc = password_encode(plaintext). INSERT into downapikey with dappid='MF2QQSJ0FtBOeVv2rNoGM' (test app). allowedips=NULL → no IP restriction. Auth flow: client sends Bearer <plaintext> → Sage password_encode() → lookup downapikey.apikey → JOIN downapp.
ASR singing voice timing offset — all word starts are ~0.3s too early: faster_whisper timestamps trigger on sound onset, not vocal peak. For singing voice, every word starts before the actual sung syllable. Fix: apply global +0.3s offset to all word timestamps before generating ASS subtitles. This is systematic, not progressive drift. Also, ASR produces noise segments ("好啊", filler sounds) and can merge adjacent lyric lines → need manual SEG_TO_LYRIC calibration to skip garbage. See references/ass-karaoke-subtitles.md for patterns.
Seedance/video generation idfile URLs expire quickly — download immediately: Async video tasks return a video URL (e.g., http://127.0.0.1:9180/idfile?path=/tmp/...). These temp files may be cleaned within minutes. Poll for SUCCEEDED status, then download the video file IMMEDIATELY in the same script. Consider re-submitting if the download fails with 404. The video field (not output_video_url) holds the URL in task query response data.
KTV video scene production workflow: Use seedance-2-0 t2v for scene generation (via /v1/video/generations, catelogid=t2v). Submit all scenes in parallel (they run asynchronously). Each scene needs ratio: "16:9" and duration parameters. Download completed scenes immediately after polling SUCCEEDED. Use ffmpeg concat demuxer to merge scenes losslessly (-c copy). For dual-track KTV: -map 2:a -map 1:a puts accompaniment on track 0 (default) and original on track 1. Use subtitles=file.ass filter for subtitle burn-in.
{{model}} in uapi data template sends Chinese name → upstream API 400: uapi data 中 "model": "{{model}}" 渲染为中文名（如 "快乐马-1.0-文生视频"），但上游期望英文 identifier（"happyhorse-1.0-t2v"）。修复：硬编码英文名：UPDATE uapi SET data = REPLACE(data, '\"model\": \"{{model}}\"', '\"model\": \"happyhorse-1.0-t2v\"')。HappyHorse t2v 修复后 27/27 视频场景全部成功提交并完成。此问题影响所有新模型。
uapi 参数命名：文件类参数用 _file，不用 _url：与 Sage 现有的 image_file/audio_file/video_file 保持一致。单个文件用 _file（如 audio_file），多个文件用 _files 数组，比对类用 _file1/_file2。_url 会让程序员误解这是 prompt 一类的文本参数。uapiio.input_fields、uapi.data 模板、API 文档三处的参数名必须一致。 用户明确纠正过：看到 "video_url":"{{prompt}}" 会搞晕人类程序员。
llmage API 变更后必须做回归测试：修改 uapi 配置、新增 v1 端点、或更新文档后，必须通过 token.opencomputing.cn 的 API 端点实际调用验证。测试范围：现有端点（/v1/chat/completions, /v1/image/generations, /v1/video/generations）确保不退化 + 新端点功能正确 + 计费链路完整（调用后等 12s 让 backend_accounting 10s 循环处理，检查 accounting_status 不是 'failed'）。用户明确要求："修改后应该做llmage模块api的回归测试的，不能省步骤"。API key 从 ~/.hermes/config.yaml 的 custom_providers 读取。参考 references/v1-api-testing.md。
uapi response 模板必须在 Jinja2 中处理 undefined 变量：当 GPU 服务返回错误（如 {"error":"images list required"}）而非预期的 {"status":"ok","faces":[...],...} 时，直接引用 {{status}}、{{faces}} 等字段会抛出 jinja2.exceptions.UndefinedError。Sage 的 uapi 层在 call() 方法中 catch 了这个异常但只 debug 不 re-raise，导致返回空 body（HTTP 200 + 0 字节）。修复：所有 response 模板必须用 {%if error%}{"error":"{{error}}"}{%else%}...{%endif%} 块包裹。不能用 {{status or ""}}（Jinja2 的 or 对 undefined 变量无效），不能用 default() 过滤器（Sage 的 Jinja2 环境可能不支持）。async 提交端点无需此处理（其 response 只有 taskid + taskstatus）。排查空响应时先 grep 'pipeline\|get_calluserid' /d/apitest/sage/logs/sage.log 看是否有 UndefinedError 被捕获。
upappkey 鉴权要求 ownerid 匹配实际用户：sync_uapi_request 调用 get_calluserid(upappid, orgid=llm.ownerid) 来获取用于调用上游服务的 API 用户。该函数执行三表 JOIN upappkey a, users b, upapp c WHERE b.orgid = c.ownerid AND a.ownerid = b.id。因此 upappkey.ownerid 必须是一个 users.id，该用户的 orgid 必须等于 upapp.ownerid。ownerid='0' 是平台公开标记，不匹配任何真实用户 → get_calluserid 抛异常 → HTTP 200 + 空 body。修复：参照同类服务（如 minimax）设置 upappkey.ownerid 为实际用户 ID，orgid 匹配 upapp.ownerid。tokentest 上有效用户：UiEi7hKqAmU1-jqQEVhZe。Sage 重启清除缓存后才生效。

67a. pricing_program 表字符集必须与 xls2ddl 标准一致：xls2ddl 建表标准为 utf8mb4_unicode_ci。生产环境若 pricing_program 列使用了 utf8mb4_general_ci，会与 organization 表 JOIN 时报 Illegal mix of collations 错误。修复：ALTER TABLE pricing_program MODIFY ownerid VARCHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; — 同理处理 providerid。所有 DDL 通过 xls2ddl 生成，字符集必须一致。

pricing_program_timing.enabled_date 不能为 NULL：即使 llm.min_balance=0，checkCustomerBalance 仍然检查当前日期是否在 pricing_program_timing.enabled_date 和 expired_date 范围内。enabled_date=NULL 时检查失败 → 429 insufficient_quota。修复：UPDATE pricing_program_timing SET enabled_date='2025-01-01' WHERE ppid='pp_xxx' AND enabled_date IS NULL。
记账验证不能只说"链路正常"：用户要求实测 accounting_status='accounted' + amount/cost 与定价一致。验证方法：插入含有效 usages={"次":1} 的 llmusage 记录（status='SUCCEEDED', accounting_status='created', 使用真实 userid/userorgid），等 15s 让 backend_accounting 10s 循环处理，查 accounting_status、amount、cost。所有 14 个端点逐个验证，不能只测一个就声称全过。用户明确纠正："记账链路正常没有用，我需要的是14个API的记账全部成功记账正确"。
tokentest 测试环境完全掌控，不要每步都申请审批：SSH apitest@tokentest.opencomputing.cn（免密），mysql -utest -ptest123 sage，Sage 端口 9180。API key 从 config.yaml 的 custom_providers 读取。rolepermission 表列名为 roleid/permid。生产环境是 token.opencomputing.cn，测试是 tokentest.opencomputing.cn，不要混淆。新增端点流程：1) 部署 dspy 到 wwwroot 2) load_path.py 加路径 3) ./py3/bin/python load_path.py 4) 重启 Sage。参考 references/sage-endpoint-deployment.md。
uapi response Jinja2 模板 — {%if error%} 模式是唯一可靠的容错方式：GPU 服务返回 {"error":"..."} 时，模板中任何对 status/usage 等不存在字段的引用都会导致 UndefinedError → 空响应。{{status or ""}} 对 undefined 变量无效（Jinja2 的 or 只判断 truthiness，不处理 undefined）。default() 过滤器在 Sage 的 Jinja2 环境可能不可用。只有 {%if error%}...{%else%}...{%endif%} 块能可靠处理。排查空响应：grep 'get_calluserid\|UndefinedError\|response_template' /d/apitest/sage/logs/sage.log | tail -20。
{{model}} in uapi data template sends Chinese name → upstream API 400：uapi data 中 "model": "{{model}}" 渲染为中文名，但上游期望英文 identifier。修复：硬编码英文名。
新增v1端点前必须先学习现有模式：不要盲目创建 dspy 文件。先读 /v1/chat/completions/index.dspy（请求体参数模式）和 /v1/pipeline/submit/index.dspy（catelogid硬编码+inference模式）了解正确的 dspy 结构。用户纠正："不能自己瞎做，你改学习一下uapi和llmage的API接口skill和规范"。关键模式：SQL JOIN llm → llm_api_map → llmcatelog、checkCustomerBalance() 返回值检查、inference() vs inference_generator() 的选择。错误的dspy导致500或空响应。默认使用 openai_403()/openai_400()/openai_429() 错误返回（非 UiError）。
路径参数化端点需要独立目录：/v1/media/{apiname} 中每个 apiname 必须有独立目录和 index.dspy。ahserver 将 /v1/media/asr-transcribe 路由到 v1/media/asr-transcribe/index.dspy，而非 v1/media/index.dspy。不能用 catch-all 模式。生成14个目录用Python脚本（非手动复制）。异步任务状态查询复用已有 /v1/tasks，不需要创建 /v1/status。部署流程见 references/v1-media-endpoint-deployment.md。
getID() broken after venv contamination — ALL v1 dspy endpoints need uuid fix：pipeline-app 的 ahserver 通过 pip -e 污染 Sage venv 后，dspy 预加载 globals 中 getID 不可用 → NameError。统计日志中受影响端点：pipeline/submit, image/generations, video/generations。临时修复：sed -i 's/getID()/str(__import__("uuid").uuid4()).replace("-","")/' /d/apitest/sage/wwwroot/llmage/v1/*/generations/index.dspy。根治：从不将 pipeline-app 包安装到 Sage venv（见 Pitfall 75）。
uapi response 模板 — 透传 usage，不硬编码字段名：GPU 服务在响应 JSON 中直接返回 usage 对象，模板用 {{json.dumps(usage, ensure_ascii=False)}} 原样透传。不要从散列字段拼装 usage（如 "usage":{"audio_seconds":{{audio_seconds|float}}}）— 这样 pricing 要素变化时模板就得改。async 提交端点（stream='async'）的 response 不含 usage（由 status 查询端点返回），其余所有 response 模板必须含 "usage"。

File Handling in uapi templates

Sage's uapi layer handles file upload/download through Jinja2 templates and ServerEnv globals.

b64media2url Extension Bug (DashScope Incompatibility)

CRITICAL: b64media2url() saves base64 data to a temp file via base642file(), then returns a URL like /idfile?path=/tmp/xxx/yyy/zzz.mp3. DashScope (and some other providers) validate the URL path extension — but they see /idfile (no extension) and reject it with InvalidURL: File type is not supported. Allowed types are: .wav, .mp3.

Root cause: FileStorage.webpath() returns a relative path, and the URL is constructed as entire_url('/idfile?path=') + quote(webpath). The path segment is /idfile, not /xxx.mp3.

Workaround for now: Upload files to external storage (OSS, CDN) and pass direct HTTPS URLs instead of base64. Or fix b64media2url() to use a URL path that includes the file extension (e.g., /idfile/xxx.mp3?path=...).

MIME type mapping: MIME_EXT in appPublic/base64_to_file.py maps audio/mpeg → mp3 (correct), NOT audio/mp3. When constructing data URLs, use data:audio/mpeg;base64,... for MP3 files, not data:audio/mp3;base64,.... Wrong MIME type causes the file to be saved with an unknown extension.

File Upload → API Request (`b64media2url`)

When uapiio defines file fields (uitype: "image"/"audio"/"video"), uploaded files are stored on the server and params_kw holds relative paths. The b64media2url(request, filename) function (registered in ServerEnv globals) converts them to base64 data URLs for API requests.

Usage in uapi.data templates:

{# Single file #}
"image_url": "{{b64media2url(request, image_file)}}"

{# Multiple files #}
"images": ["{{b64media2url(request, image_file1)}}", "{{b64media2url(request, image_file2)}}"]

{# Audio #}
"audio_url": "{{b64media2url(request, audio_file)}}"

Internal logic: If the value already starts with data:, return as-is. Otherwise, use FileStorage().realPath() to get the server path, read the file, and encode to base64.

uapiio field definition:

{"name": "image_file", "label": "首帧图片", "uitype": "image"},
{"name": "audio_file", "label": "配音文件", "uitype": "audio"},
{"name": "video_file", "label": "输入视频", "uitype": "video"}

Generated File Output (Async Models)

Model-generated files (images, videos, audio) are stored as JSON files. The llmusage.ioinfo column holds the webpath (e.g., /llmio/182/138/79/46/xxx.json) rather than full JSON content.

ioinfo JSON structure:

{
  "input": {"model": "wan2.1", "prompt": "...", "image_url": "data:image/png;base64,..."},
  "output": [
    {"model": "wan2.1", "content": "", "finish": "0", "llmusageid": "xxx"},
    {"model": "wan2.1", "content": "https://cdn.example.com/generated.mp4", "finish": "1", "usage": {...}}
  ]
}

Async response template (uapi.response) — extract result URL from external API:

{"taskid": "{{task_id}}",
{% if state == 'SUCCEEDED' %}"status": "SUCCEEDED", "result_url": "{{output_video_url}}"
{% elif state == 'FAILED' %}"status": "FAILED"
{% else %}"status": "PENDING"{% endif %}}

File Handling Pitfalls

b64media2url requires request as first argument: b64media2url(request, filename)
Variable names in uapi.data must match uapiio input_fields[].name exactly
Undefined file fields cause jinja2.exceptions.UndefinedError — ensure optional fields are checked
base64 encoding inflates file size ~33% — respect provider size limits

uapi Architecture Change

The uapiset table has been removed. auth_apiname moved to upapp, uapi now links via upappid. See references/uapi-architecture-change.md for full migration details and code diffs.

Backend Accounting

Backend accounting runs as a standalone process (NOT in the webapp). See references/backend-accounting-deployment.md for the complete deployment flow, sys.path fix for pricing imports, known bugs (usages overwrite, template syntax), and troubleshooting guide.

The accounting chain: llmage writes llmusage (accounting_status='created') → product_management.ProductManager.backend_accounting() as a standalone bin/backend_accounting.py process polls via get_accounting_llmusages() → product_accounting() creates accounting entries via consume_accounting().

Accounting & Backup Details

See references/accounting-logic.md for the complete accounting loop, failure recording, and backup logic documentation.

See references/backend-accounting-architecture.md for the standalone process model and data flow between llmage and product_management.

Deployment Flow

See references/sage-deployment-flow.md for the corrected deployment procedure (source venv → git pull → pip install . → restart).

uapi Template Debugging

See references/uapi-template-debugging.md for common pitfalls: MySQL quote escaping, Jinja2 guards ({%if status == \"SUCCEEDED\"%} not {%if result.status%} — result is undefined during RUNNING), GPU response nesting (result.segments not top-level), and llmcatelogid mismatches.

Also see references/tenantid-setup.md for v1 endpoint tenantid configuration from organization.parentid.

Pitfalls

SQLor sort 参数 vs 显式 SQL ORDER BY — 只能二选一: SQLor 的 params['sort'] 会自动生成 ORDER BY。如果 SQL 已经显式添加了 ORDER BY，必须从 params 中移除 sort 键，否则同时存在的两条 ORDER BY 导致排序冲突：

# ❌ BOTH — 冲突
params = {'today': today, 'sort': ['catelog_id', 'providerid']}
sql += " order by m.llmcatelogid, a.providerid, a.name"

# ✅ 只保留显式 ORDER BY
params = {'today': today}
sql += " order by m.llmcatelogid, a.providerid, a.name"

list_llms/get_llms_by_catelog_to_customer 同 catelog 被拆成多组: SELECT DISTINCT 无 ORDER BY 时 MySQL 返回顺序不可控。分组循环 if cid != r.catelog_id 假设同 catelog 的所有行连续——不连续则生成多个重复组。修复：显式 ORDER BY m.llmcatelogid, a.providerid, a.name 并移除 params['sort']。

API Documentation

Multiple API doc locations must be kept in sync when adding/changing v1 endpoints:

Location	Format	Purpose
`~/repos/llmage/wwwroot/api_doc.md`	Markdown	Primary v1 API reference (in the llmage repo, served in-app)
`~/repos/llmage/docs/API.md`	Markdown	Duplicate copy in llmage repo — must stay in sync with api_doc.md
`~/repos/dashboard_for_sage/wwwroot/api_doc.md`	Markdown	Dashboard reference copy
`~/repos/dashboard_for_sage/wwwroot/api_doc.html`	HTML (Apifox-style)	Customer-facing interactive doc (left nav + right detail + collapsible params)
`~/repos/sage/wwwroot/public/api/api_zh.md`	Markdown	Legacy static file (may be stale)

When adding a new v1 endpoint, update BOTH llmage repo docs AND commit+push in the same session. The two llmage docs (api_doc.md and docs/API.md) are near-duplicates and must be updated together.

api_doc.html structure: Single-file HTML with embedded JS. API endpoints defined in a apiData JavaScript array. Each entry has: id, method, path, title, desc, params (sections with rows), requestExample, responseExamples, optional models (per-model tabs), optional errors, optional curlExamples: [{ title: "label", code: \curl ...` }]. New endpoints must be added to the correct group's itemsarray. TherenderContent()function renderscurlExamples` as titled code blocks if present.

API doc update checklist when adding/modifying v1 endpoints:

Update api_doc.md with new endpoint section (params, request/response examples, curl test cases per model)
Update api_doc.html — add entry to the correct group in apiData JS array (include curlExamples array)
All examples must use model (model name string) + catelogid (t2t/t2i/t2v/i2v/ref2v), NOT llmid (UUID)
Response examples MUST be verified with real curl calls — read the actual response field from the uapi SQL script or database, then render the Jinja2 output as a concrete JSON example. There is NO outer wrapper — the response IS the uapi.response template output directly. Never invent/approximate response formats. User has zero tolerance for invented response formats. See references/image-gen-api-testing-2026-06-03.md for tested image gen response formats.
Include all available models — do not silently remove models from docs. If a model exists in the database, it must appear in the doc. User corrected: wan2.7 models were removed from docs after a prior cleanup session but should have been retained.
curl test cases: Each API endpoint needs curl examples. For endpoints supporting multiple models, provide one curl per model variant (e.g., sync vs async, different providers). Markdown: ### curl section with code blocks. HTML: curlExamples: [{ title: "...", code: \...` }]` array on each endpoint object.
Commit and push both files in the same session

Image generation response format (ALL models are sync — no async image generation):

// Success (wan2.7 series — usage has tokens):
{"status":"SUCCEEDED","usage":{"image_count":1,"input_tokens":22,"output_tokens":2,"size":"1024*1024","total_tokens":24},"image_count":1,"image":["https://token.opencomputing.cn/idfile?path=/tmp/...png"]}

// Success (qwen-image series — usage has dimensions):
{"status":"SUCCEEDED","usage":{"height":1024,"image_count":1,"width":1024},"image_count":1,"image":["https://token.opencomputing.cn/idfile?path=/tmp/...png"]}

// Error — missing required parameter (400):
{"status":"error","data":{"message":"Missing required parameter: prompt"}}

// Error — invalid model or params (400):
{"error":{"message":"Parameters error.","type":"invalidparameters","param":null,"code":"invalidparameters"}}

Key differences from old docs:

usage field varies by provider: wan2.7 returns input_tokens/output_tokens/size/total_tokens; qwen returns height/width/image_count
Image URLs are proxied through token.opencomputing.cn/idfile?path=... — NOT direct DashScope OSS links
Two distinct error response shapes (parameter validation vs model/param errors)
Size format uses * separator: 1024*1024, NOT 1024x1024
No async image models — all image generation is synchronous. /v1/tasks is only for video generation.
Critical: Do NOT wrap sync responses in {"status": "ok", "data": {...}}. Do NOT use OpenAI-style {"id": "luid_xxx", "object": "image.generation", ...}. The response is the uapi.response template output verbatim.

Image Generation Model Inventory

See references/image-gen-models.md for the complete inventory of text2image models, their uapi configurations, and sync/async patterns extracted from production database.

Video API Testing Findings

See references/video-api-testing-findings.md for API parameter quirks discovered during testing: HappyHorse I2V parameter naming, Seedance Ref2V required fields, Vidu Ref2V proxy issues.

Audio API Model Inventory

See references/audio-api-models.md for the complete inventory of TTS, ASR, and music generation models extracted from production database, including uapi/uapiio configurations.

Onboarding New Model Types from SQL Dumps

When adding a new model category (e.g., TTS, ASR, music) to the v1 API, follow this workflow:

Step 1: Extract model records from SQL dump

Production backups live at ~/db/prod.*.sql. The INSERT lines are massive (180KB+ single lines). Use Python to parse tuples:

# extract_models.py — extract specific model records from SQL dump
import re

def extract_tuple(line, search_id):
    """Extract a full SQL INSERT tuple by searching for an ID string."""
    idx = line.find(f"'{search_id}'")
    if idx < 0:
        return None
    start = line.rfind('(', 0, idx)
    if start < 0:
        return None
    depth = 0
    in_str = False
    j = start
    while j < len(line):
        c = line[j]
        if c == "'" and (j == 0 or line[j-1] != '\\'):
            in_str = not in_str
        if not in_str:
            if c == '(':
                depth += 1
            elif c == ')':
                depth -= 1
                if depth == 0:
                    return line[start:j+1]
        j += 1
    return None

Step 2: Identify relevant table line numbers

grep -n 'CREATE TABLE.*llm\b' ~/db/prod.*.sql     # llm table structure
grep -n 'CREATE TABLE.*uapi\b' ~/db/prod.*.sql    # uapi table structure  
grep -n 'CREATE TABLE.*uapiio' ~/db/prod.*.sql    # uapiio table structure

Key tables and their INSERT line positions (typical):

llm: model definitions (name, model, upappid, providerid)
llm_api_map: model→catalog→API mapping (llmid, llmcatelogid, apiname, ppid)
uapi: API endpoint configs (path, headers, data template, response template)
uapiio: input field definitions (input_fields JSON)

Step 3: Search for relevant records

Use grep on specific INSERT lines for keyword matching:

sed -n '<INSERT_LINE>p' ~/db/prod.*.sql | grep -oi "'[^']*music[^']*'"

Step 4: Create v1 endpoint

Follow the existing pattern from v1/video/generations/index.dspy or v1/image/generations/index.dspy:

Validate required params (model, catelogid, + type-specific params)
SQL JOIN: llm → llm_api_map → llmcatelog with (b.id = ${lctype}$ OR b.name = ${lctype}$)
checkCustomerBalance() → inference()

Step 5: Update three places

docs/API.md — add endpoint section with params table, models table, request/response examples
scripts/load_path.py — add to both PATHS_LOGINED and PATHS_V1_CUSTOMER
Commit + push — single commit covering all three files

Pitfall: SQL dump lines are enormous

The INSERT lines for uapi and uapiio tables can be 180KB+ (single line with all records). Never try to cat or read_file these lines directly. Always use Python parsing with the extract_tuple() approach above, targeting specific IDs found via grep.

llmcatelog ID Standardization

See references/llmcatelog-id-migration.md for the complete migration plan: standardizing llmcatelog.id from random UUIDs to meaningful abbreviations (t2t, t2v, i2v, etc.) using the appcodes system, with full impact analysis of affected files across modules.

104 KiB Raw Blame History Unescape Escape

llmage Module Skill

Overview

Accounting Failure Tracking (记账失败)

Module Location

External Service Registration

Product Sync Chain & providerid Backfill

Architecture: llmage + uapi Collaboration

Inference Flow

Key Tables

get_llm() Architecture (Refactored — No More 6-Table JOIN)

Step 1: get_llmage_llm() — 3-table JOIN (llmage DB)

Step 2: _get_uapi_cached() — uapi DB lookup (cached, 5min TTL)

Step 3: _get_uapiio_cached() — uapiio DB lookup (cached, 5min TTL)

Merge into llm result

LLM API Map Management UI

ServerEnv Functions (via load_llmage())

OpenAI-Compatible v1 API Endpoints

Three Inference Modes

1. Streaming (uapi_request) — for SSE-capable LLMs

2. Synchronous (sync_uapi_request) — for one-shot responses

3. Asynchronous (async_uapi_request) — for long-running tasks (video generation)

Billing System

Pricing Engine Internals

Permission Setup for New Endpoints

Preferred: Shell Script (scripts/setup_<module>_perms.sh)

Fallback: Production SQL Migration

Permission Path Convention

Data Isolation Pattern

Publish/Unpublish (上架/下架)

Affected Query Points (all filter a.status = 'published')

NOT filtered (admin/internal)

Migration

Testing v1 APIs

Authentication

Testing Workflow

Pitfalls

File Handling in uapi templates

b64media2url Extension Bug (DashScope Incompatibility)

File Upload → API Request (b64media2url)

Generated File Output (Async Models)

File Handling Pitfalls

uapi Architecture Change

Backend Accounting

Accounting & Backup Details

Deployment Flow

uapi Template Debugging

Pitfalls

API Documentation

Image Generation Model Inventory

Video API Testing Findings

Audio API Model Inventory

Onboarding New Model Types from SQL Dumps

Step 1: Extract model records from SQL dump

Step 2: Identify relevant table line numbers

Step 3: Search for relevant records

Step 4: Create v1 endpoint

Step 5: Update three places

Pitfall: SQL dump lines are enormous

llmcatelog ID Standardization

104 KiB

Raw Blame History

Preferred: Shell Script (`scripts/setup_<module>_perms.sh`)

Affected Query Points (all filter `a.status = 'published'`)

File Upload → API Request (`b64media2url`)