---
name: llmage-api-testing
description: "Test and debug llmage module API endpoints for LLM model validation. Use when verifying production/test environment model availability, diagnosing auth failures (403 vs 401), or running comprehensive model test suites across different function categories (text2text, video generation, image generation, etc.)."
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [llmage, api, testing, mlops, debugging, production, authentication]
    related_skills: [llmage, api-testing]
---

# llmage API Testing

Testing and debugging llmage module API endpoints for LLM model validation across production and test environments.

## Testing Conventions (User Preferences)

**Use pure curl only.** Never pipe through `python3 -m json.tool`, `jq`, or any post-processor when running API tests. The user reads raw curl output directly in their terminal. If formatting is needed, do it in a separate step, not inline with the test command.

**Do NOT use bash `for` loops** to iterate models if it involves any processing beyond raw curl. Run each test as a separate command, or use parallel terminal calls for independent models.

## When to Use This Skill

- Verifying which models are available in production vs test environments
- Running comprehensive model test suites across different function categories
- Diagnosing authentication failures (403 "You don't have access to this model" vs 401 Unauthorized)
- Validating model-to-catalog mappings in the database
- Troubleshooting why `/v1/models` works but `/v1/chat/completions` fails

## Quick Start

Test a single model:

```bash
curl -X POST "https://token.opencomputing.cn/llmage/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-max","messages":[{"role":"user","content":"Hello"}]}'
```

## API Endpoints

### GET /v1/models
List all available models for the authenticated user.

**Response:**
```json
{
  "object": "list",
  "data": [
    {"id": "qwen3-max", "object": "model", "created": 1748044800, "owned_by": "opencomputing.ai"}
  ]
}
```

**Auth**: Bearer token in Authorization header. Does NOT call `get_user()`, so works with any valid token.

### POST /v1/chat/completions
Text generation (文生文). Calls `get_user()` immediately - returns 403 if user not found.

**Request:**
```json
{
  "model": "qwen3-max",
  "messages": [{"role": "user", "content": "Hello"}]
}
```

**Success**: Returns OpenAI-compatible response with choices.

**403 Error**: `{"error": {"message": "You don't have access to this model.", "type": "invalid_request_error", "code": "model_not_found"}}`

### POST /v1/video/generations
Video generation (文生视频, 图生视频, 参考生视频). Also calls `get_user()`.

**Request (文生视频):**
```json
{
  "model": "wan2.6-t2v",
  "catelogid": "t2v",
  "prompt": "A cat running on grass",
  "duration": "5s"
}
```

**Request (图生视频):**
```json
{
  "model": "wan2.6-i2v",
  "catelogid": "i2v",
  "prompt": "A cute cat moving",
  "image_file": "https://example.com/cat.jpg",
  "duration": "5s"
}
```

### POST /v1/image/generations
Image generation (文生图).

**Request:**
```json
{
  "model": "jimeng-4.0",
  "catelogid": "t2i",
  "prompt": "A beautiful sunset",
  "size": "1024x1024"
}
```

## Authentication Flow

### How Auth Works

1. **Bearer token validation**: Extracted from `Authorization: Bearer <token>` header
2. **Encode & lookup**: The token is RC4-encoded with `config.password_key`, then matched against encrypted keys in the **`downapikey`** table (NOT `upappkey` — that stores upstream provider keys)
3. **User lookup**: `get_user()` returns the user record if a match is found; None otherwise
4. **Permission check**: Verifies the user has access to the requested model via SQL join across `llm`, `llm_api_map`, and `llmcatelog` tables

For full auth debugging workflow — finding keys in the DB, decrypting stored keys, tracing RC4 encoding, and interpreting server logs — see `references/auth-debugging.md`.

### Why /v1/models Works But /v1/chat/completions Fails

**`/v1/models` endpoint:**
- Does NOT call `get_user()`
- Simply lists models without user-specific permission checks
- Returns 200 with model list even if token is invalid for actual usage

**`/v1/chat/completions` (and other v1 endpoints):**
- Calls `get_user()` immediately at line ~21 of `wwwroot/v1/chat/completions/index.dspy`
- If `get_user()` returns `None` → returns 403 "You don't have access to this model"
- This happens when:
  - Token is valid but not associated with a registered user
  - Token is for a different environment (test vs prod)
  - User account is disabled or expired

### Diagnosing 403 vs 401

**403 "You don't have access to this model":**
- Token accepted but `get_user()` returned None
- Most likely: API key is for wrong environment or not registered
- **Fix**: Use correct API key for the target environment

**401 Unauthorized:**
- Token validation failed at the framework level
- Most likely: Invalid token format or missing Authorization header
- **Fix**: Check token format, ensure `Authorization: Bearer <token>` header is present

**429 Too Many Requests / insufficient_quota:**
- User IS authenticated (key found in `downapikey` table) but balance check failed
- The full check chain in `llmage/accounting.py:checkCustomerBalance`: self-owned skip → TPAC/local balance → ppid validation → `get_ppid_pricing` call → balance comparison
- Self-owned models (user's org owns the provider) skip the entire check
- **Critical bug**: `pricing/pricing.py` may be missing `from appPublic.share_cache import cache_get, cache_set` — this causes `get_ppid_pricing` to fail with `name 'cache_get' is not defined`, cascading to a false 429. Fix requires restart.
- **Fix**: Top up account balance, use self-owned model, or set `min_balance=0`
- **Full debug flow**: See `references/llmage-balance-check-flow.md`

## Test-Server Cookie-Session Auth (sage 9180)

Production (`token.opencomputing.cn`) authenticates with Bearer API keys. The sage TEST server (127.0.0.1:9180) serves the same llmage dspy endpoints but authenticates via sage login **session cookie** — the dspys call `get_user()` and return 403 without a session. Bearer/apikey headers do NOT work on 9180 (all header variants → 401).

1. Login — endpoint `/rbac/userpassword_login.dspy`, JSON body, param name is **`passwd`** (the dspy RC4-encodes internally):
```bash
curl -s -c /tmp/sage_cookie.txt -X POST \
  http://127.0.0.1:9180/rbac/userpassword_login.dspy \
  -H 'Content-Type: application/json' \
  -d '{"username":"sword","passwd":"Aa@123456"}'
# success: {"widgettype":"Message",...,"title":"Logined"} + Set-Cookie: AIOHTTP_SESSION=...
```
(Alternate login: `/rbac/user/up_login.dspy` with form-encoded `password` param — see `sage-module-deployment` skill.)

2. Call endpoints with the cookie jar. Paths must match `load_path.py` registration exactly — `/llmage/` MOD prefix + full dspy path:
```bash
curl -s -b /tmp/sage_cookie.txt http://127.0.0.1:9180/llmage/v1/models/index.dspy
curl -s -b /tmp/sage_cookie.txt -X POST \
  http://127.0.0.1:9180/llmage/v1/chat/completions/index.dspy \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-max","messages":[{"role":"user","content":"hi"}]}'
```
Registered v1 paths (llmage load_path.py): chat/completions, image/generations, video/generations, music/generations, audio/speech, audio/transcriptions, models, pricing, tasks — all as `/llmage/v1/<...>/index.dspy`.

Pitfalls:
- Bare `/v1/models` or `/v1/chat/completions` (no `/llmage` prefix, no `.dspy` suffix) → 403 Forbidden, log shows `invalid path` (unregistered in ahserver)
- `/login` does not exist on 9180; login lives in the rbac module
- Plain-text password in the JSON body is correct here — do NOT pre-encode it
- Login returns success but cookie jar empty: check `Set-Cookie` header directly (`-D`) — session cookie is `AIOHTTP_SESSION`, HttpOnly

## Database Schema for Model Mapping

### Key Tables

**`llm`** - Model definitions:
- `id` (varchar 32): Primary key
- `model` (varchar 100): API model name (e.g., "qwen3-max")
- `name` (varchar 100): Display name
- `status` (varchar 16): Must be 'published' for API access

**`llmcatelog`** - Function categories:
- `id` (varchar 32): Primary key — meaningful abbreviations: `t2t`, `t2i`, `t2v`, `i2v`, `r2v`, `tts`, `asr`, `vision`, `ai_search`, `digital_human`, `music_gen`, `text_cls`, `3d_gen`, `video_tool`, `translate`
- `name` (varchar 100): Chinese name (e.g., "文生文", "文生视频")
- `description` (text): Category description

**`llm_api_map`** - Model-to-API routing (8 columns, replaces old `llm_catalog_rel`):
- `id` (varchar 32): Primary key
- `llmid` (varchar 32): FK to `llm.id`
- `llmcatelogid` (varchar 32): FK to `llmcatelog.id`
- `apiname` (varchar 100): API endpoint name in upapp
- `query_apiname` (varchar 100): Async task result query endpoint (nullable)
- `query_period` (long): Task polling interval in seconds (default 30)
- `ppid` (varchar 32): FK to `pricing_program.id` (nullable)
- `isdefaultcatelog` (varchar 1): "1" if this is the model's primary catalog

### Catalog Matching: Backward-Compatible (ID or Name)

The V1 API endpoints use backward-compatible SQL: `WHERE (b.id = ${catelogid}$ OR b.name = ${catelogid}$)` — they match on BOTH `llmcatelog.id` (abbreviation like `t2v`) AND `llmcatelog.name` (Chinese name like `文生视频`). This means:
- The `catelogid` API parameter accepts either abbreviation IDs (`t2v`, `i2v`, `r2v`) or Chinese names
- Recommended: use abbreviation IDs for consistency
- Both forms work across all v1 endpoints (chat/completions, video/generations, image/generations)

### Catalog ID Mapping (post-migration)

See `references/llmcatelog-id-map.md` for the full old→new ID mapping table.

### Extracting Model-to-Category Mapping

From SQL dump:

```bash
# Extract API map relations
grep "INSERT INTO \`llm_api_map\`" ~/db/prod.sql > /tmp/llm_api_map.sql

# Extract model definitions
grep "INSERT INTO \`llm\` " ~/db/prod.sql > /tmp/llm_table.sql
```

Parse with Python:

```python
import re
from collections import defaultdict

# Parse llm_api_map (multi-column: id, llmid, llmcatelogid, apiname, ...)
with open('/tmp/llm_api_map.sql') as f:
    line = f.read()
tuples = re.findall(r"\('([^']*)','([^']*)','([^']*)','([^']*)'[^)]*\)", line)
cat_by_llmid = defaultdict(list)
for row_id, llmid, catid, apiname in tuples:
    cat_by_llmid[llmid].append((catid, apiname))

# Parse llm table to map model name -> id
with open('/tmp/llm_table.sql') as f:
    line = f.read()
llm_rows = re.findall(r"\('([^']*)','([^']*)','([^']*)','[^']*','[^']*','[^']*','[^']*','[^']*','[^']*','[^']*',[^,]*,'([^']*)'\)", line)
model_to_id = {}
for row in llm_rows:
    llmid, name, model, status = row
    model_to_id[model] = llmid

# Map catalog IDs to Chinese names (post-migration IDs)
catmap = {
    't2t': '文生文',
    't2i': '文生图',
    'tts': '语音合成',
    't2v': '文生视频',
    'vision': '图像理解',
    'asr': '语音识别',
    'i2v': '图生视频',
    'r2v': '参考生视频',
    'video_tool': '视频工具',
    'translate': '语言翻译',
    'music_gen': '音乐生成',
    'digital_human': '数字人',
    '3d_gen': '3D生成',
    'text_cls': '文本分类',
    'ai_search': 'AI搜索'
}
```

## Comprehensive Test Plan

### Test Script Structure

Create `/tmp/test_prod_models.py` with:

1. **Get model list** from `/v1/models`
2. **Map models to categories** via database tables
3. **Test each model** with appropriate endpoint based on category
4. **Collect results** with status codes and timing

### Category-Specific Test Functions

```python
def test_chat(model, catelogid="t2t"):
    """Test text generation"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "messages": [{"role": "user", "content": "Hello"}],
        "stream": False
    }
    # POST to /v1/chat/completions

def test_video_gen(model, catelogid="t2v"):
    """Test text-to-video"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A cat running on grass",
        "duration": "5s"
    }
    # POST to /v1/video/generations

def test_i2v(model, catelogid="i2v"):
    """Test image-to-video"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A cute cat moving",
        "image_file": "https://example.com/cat.jpg",
        "duration": "5s"
    }
    # POST to /v1/video/generations

def test_image_gen(model, catelogid="t2i"):
    """Test text-to-image"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "A beautiful sunset",
        "size": "512x512",
        "n": 1
    }
    # POST to /v1/image/generations

def test_music(model, catelogid="music_gen"):
    """Test music generation"""
    payload = {
        "model": model,
        "catelogid": catelogid,
        "prompt": "Happy pop song",
        "lyrics": "[Verse]\nSunshine on my face"
    }
    # POST to /v1/chat/completions (music uses chat endpoint)
```

**Note**: Always use abbreviation IDs (`t2t`, `t2v`, `i2v`, `r2v`, `t2i`) for `catelogid` — they are the recommended format. Chinese names (`文生文`, `文生视频`) are supported for backward compatibility but deprecated.

**Critical for I2V**: Use `image_file` as the parameter name, NOT `image_url`. The uapi.data Jinja2 template expects the exact field name from uapiio.input_fields.

### Running Tests

```python
# Build test plan: (model_id, catalog_id, catalog_name, test_function)
test_plan = [
    ("qwen3-max", "t2t", "文生文", test_chat),
    ("wan2.6-t2v", "t2v", "文生视频", test_video_gen),
    # ... more models
]

# Execute and collect results
results = []
for model, catid, catname, test_fn in test_plan:
    status, elapsed, detail = test_fn(model, catname)
    results.append((model, catname, status, elapsed, detail))
```

## Video Generation API Testing & Debugging

### Response Format: NOT JSON

Video generation APIs return **Python dict strings** (single-quoted), NOT valid JSON. `resp.json()` will throw `JSONDecodeError`. Use `ast.literal_eval`:

```python
import ast

def parse_response(resp):
    text = resp.text.strip()
    if not text:
        return {}
    try:
        return json.loads(text)  # Try JSON first
    except:
        pass
    try:
        return ast.literal_eval(text)  # Python dict literal
    except:
        pass
    return {"raw": text[:300]}
```

The response Content-Type may be `text/html` even on HTTP 200. Always parse defensively.

### Interpreting Response Status

- `status: "FAILED"` — **real failure**. Check the `error` field for details.
- `status: "PENDING"` / `"RUNNING"` / `"SUBMITTED"` — task accepted, can poll via `/v1/tasks?taskid=xxx`
- `status: "SUCCEEDED"` — task completed, check output for result URLs

### uapiio-Based Test Payload Construction

**Critical workflow correction (user-directed)**: When testing video models, do NOT guess parameter names. Instead:

1. Look up the model's uapi record to find its `ioid` (input/output definition ID)
2. Look up the `uapiio` record by `ioid` to get `input_fields` (JSON text defining required/optional fields)
3. Construct test payloads using **exactly** the field names from `input_fields`

Example uapiio.input_fields for an i2v model:
```json
[
  {"name": "prompt", "label": "提示词", "uitype": "textarea"},
  {"name": "image_file", "label": "首帧图片", "uitype": "image"},
  {"name": "duration", "label": "时长", "uitype": "select", "data": [...]}
]
```

The `name` field values (`image_file`, `prompt`, `duration`) are what the uapi.data Jinja2 template expects. Sending `image_url` when the template expects `image_file` causes `jinja2.exceptions.UndefinedError: 'image_file' is undefined`.

### Common Video API Failure Patterns

| Error | Meaning | Root Cause | Fix |
|-------|---------|------------|-----|
| `'image_file' is undefined` | Jinja2 template variable missing | API sent `image_url` but uapiio expects `image_file` | Use exact field names from uapiio.input_fields |
| `ERROR:400, message='Bad Request', url='https://...'` | Upstream API rejected request | Parameter format mismatch (duration, resolution, model name) | Check upstream provider's API docs for exact parameter format |
| `ERROR:404, url='.../ent/v2/ent/v2/reference2video'` | Duplicated path segment | uapi.path template has wrong base path (includes prefix that's already in baseurl) | Fix uapi.path to remove duplicate prefix |
| `没有找到模型` (model not found) | llm lookup failed | Model missing from `llm` table, or `llm_api_map` has no entry for this catelogid | Add llm record and llm_api_map entries |
| `ERROR:400` from dashscope/volces/vidu | Upstream 400 | Usually parameter value issue (e.g., duration must be integer not "5s") | Check upstream API spec for exact value format |
| `400 Bad Request` from Seedance r2v | Missing required param | `ratio` parameter is required by Volcengine Ark API despite being documented as optional | Always include `ratio` (e.g., `"16:9"`) for Seedance Ref2V |
| `Parameters error` from Vidu r2v (all attempts) | Proxy template mismatch | uapi.data template structure doesn't match Vidu API spec | Check uapi.data template and uapiio.input_fields alignment |
| Status inconsistency across providers | Non-error quirk | Seedance returns `"CREATED"`, HappyHorse returns `"PENDING"`, Vidu returns `"created"` | Normalize to uppercase when comparing status values |

### Async Task Polling After Submission

When a video generation request returns a non-FAILED status with a `taskid`, poll for completion:

```python
# Poll task status
resp = requests.get(f"{BASE}/tasks?taskid={taskid}", headers=H, timeout=30)
task_data = parse_response(resp)
status = task_data.get("status", "")
# Statuses: PENDING → RUNNING → SUCCEEDED/FAILED
```

The llmage backend also auto-polls via `query_task_status()` background task (configured by `query_period` in `llm_api_map`).

See `references/video-api-debugging.md` for full debugging workflow and failure classification.

## Error Quick-Reference

When an API test fails, the error message tells you where in the pipeline the problem is:

| Error | Meaning | Root Cause |
|-------|---------|------------|
| `insufficient_quota` | Routing works, upstream accepted | Account balance is zero, needs top-up |
| `Parameters error` | Routed to upstream but params wrong | Model missing from `llm_api_map` table, or params don't match upstream API spec |
| `400 Missing required parameter` | Model found, request incomplete | Missing required field in request body |
| `403` | Auth rejected | Invalid or expired apikey |
| Model not found (404-like) | No matching record | Model not in `llm` table or `llm_api_map` has no entry for this catelogid |

**Diagnostic Steps (use grep on SQL dump at ~/db/prod.*.sql)**:

1. **Check `llm` table** — verify model exists with correct supplier:
   ```bash
   grep -oP "'<model_name>'[^)]*" ~/db/prod.*.sql
   ```

2. **Check `llm_api_map`** — verify the model has an API mapping:
   ```bash
   grep -oP "INSERT INTO \`llm_api_map\`[^;]*" ~/db/prod.*.sql | grep -oP "'<llm_id>[^)]*"
   ```

3. **Common Fix**: If a model is in `llm` but NOT in `llm_api_map`, INSERT a mapping record. Reference an existing working model's mapping as a template.

**`Parameters error` is misleading** — it often means the model has no `llm_api_map` entry, not that your request params are wrong. Always check the mapping table first.

## Common Pitfalls

### 1. Using Wrong API Key for Environment
- **Symptom**: All 46 test points fail with 403
- **Cause**: API key is for test environment, not production
- **Fix**: Use production API key for `token.opencomputing.cn`, test API key for test environment

### 2. Not Mapping Models to Correct Catalog
- **Symptom**: Model returns 400 "model not found" even though it's in `/v1/models`
- **Cause**: Testing with wrong `catelogid` parameter
- **Fix**: Cross-reference database to get correct catalog ID for each model

### 3. Assuming /v1/models Validates Full Access
- **Symptom**: `/v1/models` returns 45 models, but all fail with 403
- **Cause**: `/v1/models` doesn't call `get_user()`, so doesn't validate user permissions
- **Fix**: Understand that `/v1/models` only lists models, doesn't validate access

### 4. Large SQL Dump Causing OOM
- **Symptom**: Python killed with exit code 137 when parsing full SQL dump
- **Cause**: Loading entire 400MB SQL file into memory
- **Fix**: Use `grep` to extract specific INSERT statements first, then parse smaller files

### 5. Missing catelogid Parameter
- **Symptom**: 400 error "missing catelogid"
- **Cause**: Video/image endpoints require `catelogid` to route to correct model handler
- **Fix**: Always include `catelogid` for non-text endpoints

### 8. Full Endpoint Smoke Test (22 endpoints)

After any deployment or config change, verify ALL /llmage/v1/* endpoints return non-500 status codes. This catches import errors, cache listener failures, and DB pool exhaustion before the user reports them.

```bash
#!/bin/bash
# curl test for all /llmage/v1/* endpoints — pass/fail by HTTP code only
HOST="http://localhost:9180"
KEY="$(decrypt_api_key)"

endpoints=(
  "chat/completions:POST"  "image/generations:POST"  "video/generations:POST"
  "audio/speech:POST"      "audio/transcriptions:POST" "music/generations:POST"
  "models:GET"             "pricing:GET"              "tasks:GET"
  "pipeline/submit:POST"   "media/asr-transcribe:POST"
  "media/demucs-separate:POST" "media/face-compare:POST"
  "media/face-detect:POST"    "media/face-recognize:POST"
  "media/merge-video:POST"    "media/realesrgan-upscale:POST"
  "media/rvc-convert:POST"    "media/songrate-evaluate:POST"
  "media/subtitle-render:POST" "media/synth-generate:POST"
  "media/video-eval-evaluate:POST"
)

for item in "${endpoints[@]}"; do
  ep="${item%%:*}" method="${item##*:}"
  url="$HOST/llmage/v1/$ep"
  [ "$method" = "POST" ] && body='{"model":"test"}' || body=""
  code=$(curl -s -o /dev/null -w "%{http_code}" -X "$method" \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    ${body:+-d "$body"} "$url" 2>/dev/null)
  [ "$code" = "500" ] && echo "FAIL $code $ep" || echo "OK   $code $ep"
done
```

**Key principle**: Test ALL endpoints, not just the one you changed. A single import error in a shared module breaks everything. 500 means crash — 401/403/400 are all fine (they prove the endpoint route is alive).

当上游返回 `400 Bad Request` 或 `"没找到模型"` 但模型确实注册了，检查 uapi data 模板：

- `{{model}}` 发送中文名而非 `llm.model` 英文标识符 → 硬编码英文名
- `{key:{{value}}}` 缺少 JSON 引号 → 改为 `{"key":"{{value}}"}`
- Async submit 响应缺少 `status` 字段 → 添加 `"status":"PENDING"`
- GPU longtasks task_id 不匹配 → submit handler 使用 longtasks 返回值

详见 `references/uapi-template-pitfalls.md`。

### 7. MariaDB Collation 冲突

JOIN 报 `Illegal mix of collations` 时使用诊断 SQL 定位，用 `ALTER TABLE ... CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci` 修复。xls2ddl 标准为 `utf8mb4_unicode_ci`。详见 `references/mariadb-collation-fix.md`。

## Debugging Checklist

When models fail with 403:

1. **Verify API key is for correct environment**
   ```bash
   # Test token on models endpoint first
   curl -s "https://token.opencomputing.cn/llmage/v1/models" \
     -H "Authorization: Bearer YOUR_KEY" | head -20
   ```

2. **Check if API key is registered in downapikey**
   ```sql
   -- Find the user for a given encrypted API key
   SELECT u.username, a.dappid, a.enabled_date, a.expired_date
   FROM downapikey a JOIN users u ON a.userid=u.id
   WHERE a.apikey = '<encrypted_key>';
   ```
   See `references/auth-debugging.md` for encoding/decoding stored keys.

3. **Verify model is published**
   ```sql
   SELECT * FROM llm WHERE model = 'qwen3-max' AND status = 'published';
   ```

4. **Check catalog mapping exists**
   ```sql
   SELECT m.*, c.name as catelog_name, l.model FROM llm_api_map m
   JOIN llmcatelog c ON m.llmcatelogid = c.id
   JOIN llm l ON m.llmid = l.id
   WHERE l.model = 'qwen3-max';
   ```

5. **Test with verbose curl**
   ```bash
   curl -v "https://token.opencomputing.cn/llmage/v1/chat/completions" \
     -H "Authorization: Bearer YOUR_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"qwen3-max","messages":[{"role":"user","content":"Hi"}]}'
   ```

## Environment URLs

- **Production**: `https://token.opencomputing.cn/llmage/v1` (note: `opencomputing`, not `oppencomputing`)
- **Test**: Check your environment-specific URL
- **API docs**: See `~/repos/llmage/docs/API.md` for full endpoint reference

## High-Concurrency Tuning

### 500 errors increase with concurrency — DB connection pool exhaustion

Each llmage request hits the DB 2-3 times. The sqlor connection pool hard caps:

| Limit | Default | Location |
|-------|---------|----------|
| Global semaphore | 400 | `sqlor/dbpools.py:29` `_GLOBAL_CONN_LIMIT` |
| Per-pool max | 100 | `sqlor/dbpools.py:59` `SqlorPool(maxconn=100)` |
| MariaDB | ~151 | `/etc/mysql/mariadb.conf.d/50-server.cnf` |

2000 concurrent = ~6000 DB operations competing for 400 slots → 500 errors cascade. Raise limits + add model caching.

See `sage-platform` skill, section "Sage/llmage High-Concurrency Tuning" for full fix.

## Load Testing & Performance Testing

For stress testing, load testing, and performance benchmarking (TPS, TTFT, response time, concurrency, RPS), see `references/llm-load-testing.md`. Key points:

- **Use worker pool pattern** (persistent async workers), NOT batch mode
- **Escalating concurrency groups**: 10 → 50 → 100 → 200 → 500 → 1000
- **Measure**: TTFT (first token latency), TPS (tokens/sec), response time, RPS
- **Tool redaction pitfall**: Construct API keys at runtime from char arrays to avoid Hermes tools breaking syntax

### Performance Profiling (In-Process Timing)

To identify CPU hotspots inside the request path, instrument the running server with a lightweight timing module. The technique:

1. **Create a timing module** (`timing.py`) that monkey-patches `json.loads`/`json.dumps` and provides a `@timed` decorator for async functions. Use a file-based toggle (`touch /tmp/sage_profile_enable`) instead of env vars — env vars don't propagate through start scripts.

2. **Insert decorators** on key functions: auth (`bearer_auth`, `get_apikey_user`, `objcheckperm`), balance (`reserve_balance`, `finalize_balance`), inference (`uapi_request`, `sync_uapi_request`, `_inference_generator`), accounting (`llm_charging`, `llm_accounting`).

3. **Critical pitfall — site-packages vs pkgs**: Sage modules may load from BOTH `site-packages` (installed editable) and `pkgs/` (source). Always check `module.__file__` to determine which copy is active before editing. Modifying the wrong copy = probes silently don't fire.

4. **Critical pitfall — sys.path insertion**: Adding `sys.path.insert(0, "/d/apitest/sage/pkgs")` to import a helper module can **shadow editable installs**. For example, `pkgs/llmage/__init__.py` may be empty while the editable-installed version has content — inserting pkgs at position 0 causes Python to find the empty version, breaking all imports from that package. Prefer copying the helper module into site-packages instead.

5. **Start clean**: Kill ALL stale workers before restarting. Multiple `start.sh` runs accumulate workers; `pgrep -f "app/sage.py"` reveals the true count. Kill old PIDs explicitly (`kill -9 <pid>`) before starting fresh.

Full profiling script: `references/perf-profiling-timing.py`. Load-test script with CPU monitoring: `templates/llm-stress-cpu.py`.

**Known bottleneck (2026-08-05, 4-core tokentest)**: `objcheckperm` permission JOIN query (`rolepermission` ⋈ `userrole`) takes **128ms p50 / 547ms max** — 77% of total request time. API key lookup (`downapikey` SQL) is only ~2ms. JSON parsing is <0.1ms. Optimize permission caching before anything else.

**Two implementation patterns:**

| Pattern | File | When to Use |
|---------|------|-------------|
| Persistent worker pool | `references/llm-load-testing.md` | Sustained max-throughput testing; TPS/SLA measurement |
| Slot-filling (simpler) | `templates/load-test-stream.py` | Quick TTFB/QPM baseline; per-minute breakdown; easy to modify |
| **Enhanced error detection** | `templates/concurrent-load-test-v2.py` | When you need per-error-type breakdown (HTTP status, stream errors, connection errors); samples error details; ideal for debugging failures under load |

Both patterns use unique prompts per request to avoid caching effects. The v2 template adds comprehensive error classification: HTTP status code bucketing, stream content validation (data: prefix, [DONE] signal, JSON parse errors, API error fields), connection-level exception subclassing (timeout vs reset vs disconnected vs SSL), and error sampling for root-cause analysis.

## Related Skills

- `llmage`: Core llmage module architecture and implementation
- `api-testing`: General API testing patterns and best practices
- `database-debugging`: SQL dump parsing and schema analysis

## References

- llmage API documentation: `~/repos/llmage/docs/API.md`
- llmage source code: `~/repos/llmage/wwwroot/v1/`
- Database schema: See `~/db/prod-*.sql` or `~/db/test-*.sql` dumps
- **Load testing patterns**: `references/llm-load-testing.md`
- **Performance profiling (in-process timing)**: `references/perf-profiling-timing.py`
- **CPU-aware load test template**: `templates/llm-stress-cpu.py`
- **Auth debugging (downapikey, RC4, DB creds)**: `references/auth-debugging.md`
- **Balance check & pricing debug flow**: `references/llmage-balance-check-flow.md`