yumoqing/pipeline_core

Fork 0

yumoqing 1abaa4ea2e feat: 技能库(skills_library 213个Hermes技能)+skill_pack技能集安装能力+ocai-h5-dev技能集(37个应用/模块开发技能)

2026-08-16 15:17:43 +08:00

29 KiB

Raw Blame History

name

description

version

author

license

metadata

llmage-api-testing

Test and debug llmage module API endpoints for LLM model validation. Use when verifying production/test environment model availability, diagnosing auth failures (403 vs 401), or running comprehensive model test suites across different function categories (text2text, video generation, image generation, etc.).

1.0.0

Hermes Agent

MIT

hermes

llmage API Testing

Testing and debugging llmage module API endpoints for LLM model validation across production and test environments.

Testing Conventions (User Preferences)

Use pure curl only. Never pipe through python3 -m json.tool, jq, or any post-processor when running API tests. The user reads raw curl output directly in their terminal. If formatting is needed, do it in a separate step, not inline with the test command.

Do NOT use bash for loops to iterate models if it involves any processing beyond raw curl. Run each test as a separate command, or use parallel terminal calls for independent models.

When to Use This Skill

Verifying which models are available in production vs test environments
Running comprehensive model test suites across different function categories
Diagnosing authentication failures (403 "You don't have access to this model" vs 401 Unauthorized)
Validating model-to-catalog mappings in the database
Troubleshooting why /v1/models works but /v1/chat/completions fails

Quick Start

Test a single model:

curl -X POST "https://token.opencomputing.cn/llmage/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-max","messages":[{"role":"user","content":"Hello"}]}'

API Endpoints

GET /v1/models

List all available models for the authenticated user.

Response:

{
  "object": "list",
  "data": [
    {"id": "qwen3-max", "object": "model", "created": 1748044800, "owned_by": "opencomputing.ai"}
  ]
}

Auth: Bearer token in Authorization header. Does NOT call get_user(), so works with any valid token.

POST /v1/chat/completions

Text generation (文生文). Calls get_user() immediately - returns 403 if user not found.

Request:

{
  "model": "qwen3-max",
  "messages": [{"role": "user", "content": "Hello"}]
}

Success: Returns OpenAI-compatible response with choices.

403 Error: {"error": {"message": "You don't have access to this model.", "type": "invalid_request_error", "code": "model_not_found"}}

POST /v1/video/generations

Video generation (文生视频, 图生视频, 参考生视频). Also calls get_user().

Request (文生视频):

{
  "model": "wan2.6-t2v",
  "catelogid": "t2v",
  "prompt": "A cat running on grass",
  "duration": "5s"
}

Request (图生视频):

{
  "model": "wan2.6-i2v",
  "catelogid": "i2v",
  "prompt": "A cute cat moving",
  "image_file": "https://example.com/cat.jpg",
  "duration": "5s"
}

POST /v1/image/generations

Image generation (文生图).

Request:

{
  "model": "jimeng-4.0",
  "catelogid": "t2i",
  "prompt": "A beautiful sunset",
  "size": "1024x1024"
}

Authentication Flow

How Auth Works

Bearer token validation: Extracted from Authorization: Bearer <token> header
Encode & lookup: The token is RC4-encoded with config.password_key, then matched against encrypted keys in the downapikey table (NOT upappkey — that stores upstream provider keys)
User lookup: get_user() returns the user record if a match is found; None otherwise
Permission check: Verifies the user has access to the requested model via SQL join across llm, llm_api_map, and llmcatelog tables

For full auth debugging workflow — finding keys in the DB, decrypting stored keys, tracing RC4 encoding, and interpreting server logs — see references/auth-debugging.md.

Why /v1/models Works But /v1/chat/completions Fails

/v1/models endpoint:

Does NOT call get_user()
Simply lists models without user-specific permission checks
Returns 200 with model list even if token is invalid for actual usage

/v1/chat/completions (and other v1 endpoints):

Calls get_user() immediately at line ~21 of wwwroot/v1/chat/completions/index.dspy
If get_user() returns None → returns 403 "You don't have access to this model"
This happens when:
- Token is valid but not associated with a registered user
- Token is for a different environment (test vs prod)
- User account is disabled or expired

Diagnosing 403 vs 401

403 "You don't have access to this model":

Token accepted but get_user() returned None
Most likely: API key is for wrong environment or not registered
Fix: Use correct API key for the target environment

401 Unauthorized:

Token validation failed at the framework level
Most likely: Invalid token format or missing Authorization header
Fix: Check token format, ensure Authorization: Bearer <token> header is present

429 Too Many Requests / insufficient_quota:

User IS authenticated (key found in downapikey table) but balance check failed
The full check chain in llmage/accounting.py:checkCustomerBalance: self-owned skip → TPAC/local balance → ppid validation → get_ppid_pricing call → balance comparison
Self-owned models (user's org owns the provider) skip the entire check
Critical bug: pricing/pricing.py may be missing from appPublic.share_cache import cache_get, cache_set — this causes get_ppid_pricing to fail with name 'cache_get' is not defined, cascading to a false 429. Fix requires restart.
Fix: Top up account balance, use self-owned model, or set min_balance=0
Full debug flow: See references/llmage-balance-check-flow.md

Production (token.opencomputing.cn) authenticates with Bearer API keys. The sage TEST server (127.0.0.1:9180) serves the same llmage dspy endpoints but authenticates via sage login session cookie — the dspys call get_user() and return 403 without a session. Bearer/apikey headers do NOT work on 9180 (all header variants → 401).

Login — endpoint /rbac/userpassword_login.dspy, JSON body, param name is passwd (the dspy RC4-encodes internally):

curl -s -c /tmp/sage_cookie.txt -X POST \
  http://127.0.0.1:9180/rbac/userpassword_login.dspy \
  -H 'Content-Type: application/json' \
  -d '{"username":"sword","passwd":"Aa@123456"}'
# success: {"widgettype":"Message",...,"title":"Logined"} + Set-Cookie: AIOHTTP_SESSION=...

(Alternate login: /rbac/user/up_login.dspy with form-encoded password param — see sage-module-deployment skill.)

Call endpoints with the cookie jar. Paths must match load_path.py registration exactly — /llmage/ MOD prefix + full dspy path:

curl -s -b /tmp/sage_cookie.txt http://127.0.0.1:9180/llmage/v1/models/index.dspy
curl -s -b /tmp/sage_cookie.txt -X POST \
  http://127.0.0.1:9180/llmage/v1/chat/completions/index.dspy \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-max","messages":[{"role":"user","content":"hi"}]}'

Registered v1 paths (llmage load_path.py): chat/completions, image/generations, video/generations, music/generations, audio/speech, audio/transcriptions, models, pricing, tasks — all as /llmage/v1/<...>/index.dspy.

Pitfalls:

Bare /v1/models or /v1/chat/completions (no /llmage prefix, no .dspy suffix) → 403 Forbidden, log shows invalid path (unregistered in ahserver)
/login does not exist on 9180; login lives in the rbac module
Plain-text password in the JSON body is correct here — do NOT pre-encode it
Login returns success but cookie jar empty: check Set-Cookie header directly (-D) — session cookie is AIOHTTP_SESSION, HttpOnly

Database Schema for Model Mapping

Key Tables

llm - Model definitions:

id (varchar 32): Primary key
model (varchar 100): API model name (e.g., "qwen3-max")
name (varchar 100): Display name
status (varchar 16): Must be 'published' for API access

llmcatelog - Function categories:

id (varchar 32): Primary key — meaningful abbreviations: t2t, t2i, t2v, i2v, r2v, tts, asr, vision, ai_search, digital_human, music_gen, text_cls, 3d_gen, video_tool, translate
name (varchar 100): Chinese name (e.g., "文生文", "文生视频")
description (text): Category description

llm_api_map - Model-to-API routing (8 columns, replaces old llm_catalog_rel):

id (varchar 32): Primary key
llmid (varchar 32): FK to llm.id
llmcatelogid (varchar 32): FK to llmcatelog.id
apiname (varchar 100): API endpoint name in upapp
query_apiname (varchar 100): Async task result query endpoint (nullable)
query_period (long): Task polling interval in seconds (default 30)
ppid (varchar 32): FK to pricing_program.id (nullable)
isdefaultcatelog (varchar 1): "1" if this is the model's primary catalog

Catalog Matching: Backward-Compatible (ID or Name)

The V1 API endpoints use backward-compatible SQL: WHERE (b.id = ${catelogid}$ OR b.name = ${catelogid}$) — they match on BOTH llmcatelog.id (abbreviation like t2v) AND llmcatelog.name (Chinese name like 文生视频). This means:

The catelogid API parameter accepts either abbreviation IDs (t2v, i2v, r2v) or Chinese names
Recommended: use abbreviation IDs for consistency
Both forms work across all v1 endpoints (chat/completions, video/generations, image/generations)

Catalog ID Mapping (post-migration)

See references/llmcatelog-id-map.md for the full old→new ID mapping table.

Extracting Model-to-Category Mapping

From SQL dump:

# Extract API map relations
grep "INSERT INTO \`llm_api_map\`" ~/db/prod.sql > /tmp/llm_api_map.sql

# Extract model definitions
grep "INSERT INTO \`llm\` " ~/db/prod.sql > /tmp/llm_table.sql

Parse with Python:

import re
from collections import defaultdict

# Parse llm_api_map (multi-column: id, llmid, llmcatelogid, apiname, ...)
with open('/tmp/llm_api_map.sql') as f:
    line = f.read()
tuples = re.findall(r"\('([^']*)','([^']*)','([^']*)','([^']*)'[^)]*\)", line)
cat_by_llmid = defaultdict(list)
for row_id, llmid, catid, apiname in tuples:
    cat_by_llmid[llmid].append((catid, apiname))

# Parse llm table to map model name -> id
with open('/tmp/llm_table.sql') as f:
    line = f.read()
llm_rows = re.findall(r"\('([^']*)','([^']*)','([^']*)','[^']*','[^']*','[^']*','[^']*','[^']*','[^']*','[^']*',[^,]*,'([^']*)'\)", line)
model_to_id = {}
for row in llm_rows:
    llmid, name, model, status = row
    model_to_id[model] = llmid

# Map catalog IDs to Chinese names (post-migration IDs)
catmap = {
    't2t': '文生文',
    't2i': '文生图',
    'tts': '语音合成',
    't2v': '文生视频',
    'vision': '图像理解',
    'asr': '语音识别',
    'i2v': '图生视频',
    'r2v': '参考生视频',
    'video_tool': '视频工具',
    'translate': '语言翻译',
    'music_gen': '音乐生成',
    'digital_human': '数字人',
    '3d_gen': '3D生成',
    'text_cls': '文本分类',
    'ai_search': 'AI搜索'
}

Comprehensive Test Plan

Test Script Structure

Create /tmp/test_prod_models.py with:

Get model list from /v1/models
Map models to categories via database tables
Test each model with appropriate endpoint based on category
Collect results with status codes and timing

Category-Specific Test Functions

def test_chat(model, catelogid="t2t"):
    """Test text generation"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "messages": [{"role": "user", "content": "Hello"}],
        "stream": False
    }
    # POST to /v1/chat/completions

def test_video_gen(model, catelogid="t2v"):
    """Test text-to-video"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A cat running on grass",
        "duration": "5s"
    }
    # POST to /v1/video/generations

def test_i2v(model, catelogid="i2v"):
    """Test image-to-video"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A cute cat moving",
        "image_file": "https://example.com/cat.jpg",
        "duration": "5s"
    }
    # POST to /v1/video/generations

def test_image_gen(model, catelogid="t2i"):
    """Test text-to-image"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A beautiful sunset",
        "size": "512x512",
        "n": 1
    }
    # POST to /v1/image/generations

def test_music(model, catelogid="music_gen"):
    """Test music generation"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "Happy pop song",
        "lyrics": "[Verse]\nSunshine on my face"
    }
    # POST to /v1/chat/completions (music uses chat endpoint)

Note: Always use abbreviation IDs (t2t, t2v, i2v, r2v, t2i) for catelogid — they are the recommended format. Chinese names (文生文, 文生视频) are supported for backward compatibility but deprecated.

Critical for I2V: Use image_file as the parameter name, NOT image_url. The uapi.data Jinja2 template expects the exact field name from uapiio.input_fields.

Running Tests

# Build test plan: (model_id, catalog_id, catalog_name, test_function)
test_plan = [
    ("qwen3-max", "t2t", "文生文", test_chat),
    ("wan2.6-t2v", "t2v", "文生视频", test_video_gen),
    # ... more models
]

# Execute and collect results
results = []
for model, catid, catname, test_fn in test_plan:
    status, elapsed, detail = test_fn(model, catname)
    results.append((model, catname, status, elapsed, detail))

Video Generation API Testing & Debugging

Response Format: NOT JSON

Video generation APIs return Python dict strings (single-quoted), NOT valid JSON. resp.json() will throw JSONDecodeError. Use ast.literal_eval:

import ast

def parse_response(resp):
    text = resp.text.strip()
    if not text:
        return {}
    try:
        return json.loads(text)  # Try JSON first
    except:
        pass
    try:
        return ast.literal_eval(text)  # Python dict literal
    except:
        pass
    return {"raw": text[:300]}

The response Content-Type may be text/html even on HTTP 200. Always parse defensively.

Interpreting Response Status

status: "FAILED" — real failure. Check the error field for details.
status: "PENDING" / "RUNNING" / "SUBMITTED" — task accepted, can poll via /v1/tasks?taskid=xxx
status: "SUCCEEDED" — task completed, check output for result URLs

uapiio-Based Test Payload Construction

Critical workflow correction (user-directed): When testing video models, do NOT guess parameter names. Instead:

Look up the model's uapi record to find its ioid (input/output definition ID)
Look up the uapiio record by ioid to get input_fields (JSON text defining required/optional fields)
Construct test payloads using exactly the field names from input_fields

Example uapiio.input_fields for an i2v model:

[
  {"name": "prompt", "label": "提示词", "uitype": "textarea"},
  {"name": "image_file", "label": "首帧图片", "uitype": "image"},
  {"name": "duration", "label": "时长", "uitype": "select", "data": [...]}
]

The name field values (image_file, prompt, duration) are what the uapi.data Jinja2 template expects. Sending image_url when the template expects image_file causes jinja2.exceptions.UndefinedError: 'image_file' is undefined.

Common Video API Failure Patterns

Error	Meaning	Root Cause	Fix
`'image_file' is undefined`	Jinja2 template variable missing	API sent `image_url` but uapiio expects `image_file`	Use exact field names from uapiio.input_fields
`ERROR:400, message='Bad Request', url='https://...'`	Upstream API rejected request	Parameter format mismatch (duration, resolution, model name)	Check upstream provider's API docs for exact parameter format
`ERROR:404, url='.../ent/v2/ent/v2/reference2video'`	Duplicated path segment	uapi.path template has wrong base path (includes prefix that's already in baseurl)	Fix uapi.path to remove duplicate prefix
`没有找到模型` (model not found)	llm lookup failed	Model missing from `llm` table, or `llm_api_map` has no entry for this catelogid	Add llm record and llm_api_map entries
`ERROR:400` from dashscope/volces/vidu	Upstream 400	Usually parameter value issue (e.g., duration must be integer not "5s")	Check upstream API spec for exact value format
`400 Bad Request` from Seedance r2v	Missing required param	`ratio` parameter is required by Volcengine Ark API despite being documented as optional	Always include `ratio` (e.g., `"16:9"`) for Seedance Ref2V
`Parameters error` from Vidu r2v (all attempts)	Proxy template mismatch	uapi.data template structure doesn't match Vidu API spec	Check uapi.data template and uapiio.input_fields alignment
Status inconsistency across providers	Non-error quirk	Seedance returns `"CREATED"`, HappyHorse returns `"PENDING"`, Vidu returns `"created"`	Normalize to uppercase when comparing status values

Async Task Polling After Submission

When a video generation request returns a non-FAILED status with a taskid, poll for completion:

# Poll task status
resp = requests.get(f"{BASE}/tasks?taskid={taskid}", headers=H, timeout=30)
task_data = parse_response(resp)
status = task_data.get("status", "")
# Statuses: PENDING → RUNNING → SUCCEEDED/FAILED

The llmage backend also auto-polls via query_task_status() background task (configured by query_period in llm_api_map).

See references/video-api-debugging.md for full debugging workflow and failure classification.

Error Quick-Reference

When an API test fails, the error message tells you where in the pipeline the problem is:

Error	Meaning	Root Cause
`insufficient_quota`	Routing works, upstream accepted	Account balance is zero, needs top-up
`Parameters error`	Routed to upstream but params wrong	Model missing from `llm_api_map` table, or params don't match upstream API spec
`400 Missing required parameter`	Model found, request incomplete	Missing required field in request body
`403`	Auth rejected	Invalid or expired apikey
Model not found (404-like)	No matching record	Model not in `llm` table or `llm_api_map` has no entry for this catelogid

Diagnostic Steps (use grep on SQL dump at ~/db/prod.*.sql):

Check llm table — verify model exists with correct supplier:
```
grep -oP "'<model_name>'[^)]*" ~/db/prod.*.sql
```

Check llm_api_map — verify the model has an API mapping:

grep -oP "INSERT INTO \`llm_api_map\`[^;]*" ~/db/prod.*.sql | grep -oP "'<llm_id>[^)]*"

Common Fix: If a model is in llm but NOT in llm_api_map, INSERT a mapping record. Reference an existing working model's mapping as a template.

Parameters error is misleading — it often means the model has no llm_api_map entry, not that your request params are wrong. Always check the mapping table first.

Common Pitfalls

1. Using Wrong API Key for Environment

Symptom: All 46 test points fail with 403
Cause: API key is for test environment, not production
Fix: Use production API key for token.opencomputing.cn, test API key for test environment

2. Not Mapping Models to Correct Catalog

Symptom: Model returns 400 "model not found" even though it's in /v1/models
Cause: Testing with wrong catelogid parameter
Fix: Cross-reference database to get correct catalog ID for each model

3. Assuming /v1/models Validates Full Access

Symptom: /v1/models returns 45 models, but all fail with 403
Cause: /v1/models doesn't call get_user(), so doesn't validate user permissions
Fix: Understand that /v1/models only lists models, doesn't validate access

4. Large SQL Dump Causing OOM

Symptom: Python killed with exit code 137 when parsing full SQL dump
Cause: Loading entire 400MB SQL file into memory
Fix: Use grep to extract specific INSERT statements first, then parse smaller files

5. Missing catelogid Parameter

Symptom: 400 error "missing catelogid"
Cause: Video/image endpoints require catelogid to route to correct model handler
Fix: Always include catelogid for non-text endpoints

8. Full Endpoint Smoke Test (22 endpoints)

After any deployment or config change, verify ALL /llmage/v1/* endpoints return non-500 status codes. This catches import errors, cache listener failures, and DB pool exhaustion before the user reports them.

#!/bin/bash
# curl test for all /llmage/v1/* endpoints — pass/fail by HTTP code only
HOST="http://localhost:9180"
KEY="$(decrypt_api_key)"

endpoints=(
  "chat/completions:POST"  "image/generations:POST"  "video/generations:POST"
  "audio/speech:POST"      "audio/transcriptions:POST" "music/generations:POST"
  "models:GET"             "pricing:GET"              "tasks:GET"
  "pipeline/submit:POST"   "media/asr-transcribe:POST"
  "media/demucs-separate:POST" "media/face-compare:POST"
  "media/face-detect:POST"    "media/face-recognize:POST"
  "media/merge-video:POST"    "media/realesrgan-upscale:POST"
  "media/rvc-convert:POST"    "media/songrate-evaluate:POST"
  "media/subtitle-render:POST" "media/synth-generate:POST"
  "media/video-eval-evaluate:POST"
)

for item in "${endpoints[@]}"; do
  ep="${item%%:*}" method="${item##*:}"
  url="$HOST/llmage/v1/$ep"
  [ "$method" = "POST" ] && body='{"model":"test"}' || body=""
  code=$(curl -s -o /dev/null -w "%{http_code}" -X "$method" \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    ${body:+-d "$body"} "$url" 2>/dev/null)
  [ "$code" = "500" ] && echo "FAIL $code $ep" || echo "OK   $code $ep"
done

Key principle: Test ALL endpoints, not just the one you changed. A single import error in a shared module breaks everything. 500 means crash — 401/403/400 are all fine (they prove the endpoint route is alive).

当上游返回 400 Bad Request 或 "没找到模型" 但模型确实注册了，检查 uapi data 模板：

{{model}} 发送中文名而非 llm.model 英文标识符 → 硬编码英文名
{key:{{value}}} 缺少 JSON 引号 → 改为 {"key":"{{value}}"}
Async submit 响应缺少 status 字段 → 添加 "status":"PENDING"
GPU longtasks task_id 不匹配 → submit handler 使用 longtasks 返回值

详见 references/uapi-template-pitfalls.md。

7. MariaDB Collation 冲突

JOIN 报 Illegal mix of collations 时使用诊断 SQL 定位，用 ALTER TABLE ... CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci 修复。xls2ddl 标准为 utf8mb4_unicode_ci。详见 references/mariadb-collation-fix.md。

Debugging Checklist

When models fail with 403:

Verify API key is for correct environment

# Test token on models endpoint first
curl -s "https://token.opencomputing.cn/llmage/v1/models" \
  -H "Authorization: Bearer YOUR_KEY" | head -20

Check if API key is registered in downapikey

-- Find the user for a given encrypted API key
SELECT u.username, a.dappid, a.enabled_date, a.expired_date
FROM downapikey a JOIN users u ON a.userid=u.id
WHERE a.apikey = '<encrypted_key>';

See references/auth-debugging.md for encoding/decoding stored keys.

Verify model is published

SELECT * FROM llm WHERE model = 'qwen3-max' AND status = 'published';

Check catalog mapping exists

SELECT m.*, c.name as catelog_name, l.model FROM llm_api_map m
JOIN llmcatelog c ON m.llmcatelogid = c.id
JOIN llm l ON m.llmid = l.id
WHERE l.model = 'qwen3-max';

Test with verbose curl

curl -v "https://token.opencomputing.cn/llmage/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-max","messages":[{"role":"user","content":"Hi"}]}'

Environment URLs

Production: https://token.opencomputing.cn/llmage/v1 (note: opencomputing, not oppencomputing)
Test: Check your environment-specific URL
API docs: See ~/repos/llmage/docs/API.md for full endpoint reference

High-Concurrency Tuning

500 errors increase with concurrency — DB connection pool exhaustion

Each llmage request hits the DB 2-3 times. The sqlor connection pool hard caps:

Limit	Default	Location
Global semaphore	400	`sqlor/dbpools.py:29` `_GLOBAL_CONN_LIMIT`
Per-pool max	100	`sqlor/dbpools.py:59` `SqlorPool(maxconn=100)`
MariaDB	~151	`/etc/mysql/mariadb.conf.d/50-server.cnf`

2000 concurrent = ~6000 DB operations competing for 400 slots → 500 errors cascade. Raise limits + add model caching.

See sage-platform skill, section "Sage/llmage High-Concurrency Tuning" for full fix.

Load Testing & Performance Testing

For stress testing, load testing, and performance benchmarking (TPS, TTFT, response time, concurrency, RPS), see references/llm-load-testing.md. Key points:

Use worker pool pattern (persistent async workers), NOT batch mode
Escalating concurrency groups: 10 → 50 → 100 → 200 → 500 → 1000
Measure: TTFT (first token latency), TPS (tokens/sec), response time, RPS
Tool redaction pitfall: Construct API keys at runtime from char arrays to avoid Hermes tools breaking syntax

Performance Profiling (In-Process Timing)

To identify CPU hotspots inside the request path, instrument the running server with a lightweight timing module. The technique:

Create a timing module (timing.py) that monkey-patches json.loads/json.dumps and provides a @timed decorator for async functions. Use a file-based toggle (touch /tmp/sage_profile_enable) instead of env vars — env vars don't propagate through start scripts.
Insert decorators on key functions: auth (bearer_auth, get_apikey_user, objcheckperm), balance (reserve_balance, finalize_balance), inference (uapi_request, sync_uapi_request, _inference_generator), accounting (llm_charging, llm_accounting).
Critical pitfall — site-packages vs pkgs: Sage modules may load from BOTH site-packages (installed editable) and pkgs/ (source). Always check module.__file__ to determine which copy is active before editing. Modifying the wrong copy = probes silently don't fire.
Critical pitfall — sys.path insertion: Adding sys.path.insert(0, "/d/apitest/sage/pkgs") to import a helper module can shadow editable installs. For example, pkgs/llmage/__init__.py may be empty while the editable-installed version has content — inserting pkgs at position 0 causes Python to find the empty version, breaking all imports from that package. Prefer copying the helper module into site-packages instead.
Start clean: Kill ALL stale workers before restarting. Multiple start.sh runs accumulate workers; pgrep -f "app/sage.py" reveals the true count. Kill old PIDs explicitly (kill -9 <pid>) before starting fresh.

Full profiling script: references/perf-profiling-timing.py. Load-test script with CPU monitoring: templates/llm-stress-cpu.py.

Known bottleneck (2026-08-05, 4-core tokentest): objcheckperm permission JOIN query (rolepermission ⋈ userrole) takes 128ms p50 / 547ms max — 77% of total request time. API key lookup (downapikey SQL) is only ~2ms. JSON parsing is <0.1ms. Optimize permission caching before anything else.

Two implementation patterns:

Pattern	File	When to Use
Persistent worker pool	`references/llm-load-testing.md`	Sustained max-throughput testing; TPS/SLA measurement
Slot-filling (simpler)	`templates/load-test-stream.py`	Quick TTFB/QPM baseline; per-minute breakdown; easy to modify
Enhanced error detection	`templates/concurrent-load-test-v2.py`	When you need per-error-type breakdown (HTTP status, stream errors, connection errors); samples error details; ideal for debugging failures under load

Both patterns use unique prompts per request to avoid caching effects. The v2 template adds comprehensive error classification: HTTP status code bucketing, stream content validation (data: prefix, [DONE] signal, JSON parse errors, API error fields), connection-level exception subclassing (timeout vs reset vs disconnected vs SSL), and error sampling for root-cause analysis.

llmage: Core llmage module architecture and implementation
api-testing: General API testing patterns and best practices
database-debugging: SQL dump parsing and schema analysis

References

llmage API documentation: ~/repos/llmage/docs/API.md
llmage source code: ~/repos/llmage/wwwroot/v1/
Database schema: See ~/db/prod-*.sql or ~/db/test-*.sql dumps
Load testing patterns: references/llm-load-testing.md
Performance profiling (in-process timing): references/perf-profiling-timing.py
CPU-aware load test template: templates/llm-stress-cpu.py
Auth debugging (downapikey, RC4, DB creds): references/auth-debugging.md
Balance check & pricing debug flow: references/llmage-balance-check-flow.md

29 KiB Raw Blame History