yumoqing/pipeline_core

Fork 0

yumoqing 1abaa4ea2e feat: 技能库(skills_library 213个Hermes技能)+skill_pack技能集安装能力+ocai-h5-dev技能集(37个应用/模块开发技能)

2026-08-16 15:17:43 +08:00

54 KiB

Raw Blame History

name

description

API Load Testing

Design and execute HTTP API stress tests with sustained concurrency, streaming metrics, and group-based load profiles.

When to Use

User asks to "pressure test", "stress test", or "benchmark" an API
Need to measure throughput (RPS), latency (TTFT, response time), or concurrent connection limits
Testing streaming APIs (SSE, WebSocket) with token-level metrics
Comparing performance across different concurrency levels

Architecture: Worker Pool vs Batch Mode

CRITICAL: Use worker pool pattern, NOT batch mode.

Worker Pool Pattern (Correct)

# Each worker completes a request, immediately sends the next
async def worker(session, prompts, metrics, stop_event, worker_id):
    while not stop_event.is_set():
        prompt = prompts[worker_id % len(prompts)]
        await do_one_request(session, prompt, metrics)

Maintains constant concurrency (N workers = N in-flight requests)
Workers self-throttle based on response time
RPS scales with backend capacity

Batch Mode for Burst Testing

While worker pool is the default for sustained load testing, batch mode is appropriate when testing burst capacity or measuring "all requests complete before next batch" semantics.

When to Use Batch Mode

Testing how system handles a sudden burst of N simultaneous requests
Measuring total time for N concurrent requests to complete
User explicitly requests "wait for all requests before starting next group"
Simpler metrics: just count success/fail and measure total time

Implementation Pattern

async def test_batch(session, concurrency, prompts, results):
    """Send N requests simultaneously, wait for ALL to complete."""
    tasks = []
    for i in range(concurrency):
        prompt = prompts[i % len(prompts)]
        task = asyncio.create_task(send_request(session, i, prompt, results))
        tasks.append(task)
    
    # CRITICAL: Wait for ALL tasks to complete (success or failure)
    await asyncio.gather(*tasks, return_exceptions=True)
    
    # Now results[] contains all outcomes
    success = sum(1 for r in results if r["success"])
    errors = len(results) - success
    return success, errors

Anti-pattern: Incomplete Wait

# WRONG: Using timeout that expires before all requests complete
try:
    await asyncio.wait_for(asyncio.gather(*tasks), timeout=60)
except TimeoutError:
    pass  # Some requests still in-flight!

# This contaminates the next batch with leftover requests

Rule: If using batch mode, always use await asyncio.gather(*tasks, return_exceptions=True) without timeout (or with very generous timeout like 300s).

When NOT to Use Batch Mode

Measuring sustained throughput over time (use worker pool)
Comparing RPS across concurrency levels (batch artificially limits RPS)
Long-running tests where you want continuous load (worker pool self-throttles)

Summary: Worker pool for sustained load, batch mode for burst testing. Choose based on what you're measuring.

Streaming Metrics Collection

For streaming APIs (SSE, stream=true), measure:

TTFT (Time To First Token)

start = time.monotonic()
ttft = None

async for line in response.content:
    # Parse SSE/data line
    if content := extract_content(line):
        if ttft is None:
            ttft = (time.monotonic() - start) * 1000  # ms

TPS (Tokens Per Second)

token_count = 0
async for line in response.content:
    if content := extract_content(line):
        token_count += len(content)  # WARNING: counts characters, not tokens

duration_sec = (time.monotonic() - start) / 1000
tps = token_count / duration_sec

Pitfall: len(content) counts characters, not tokens. For accurate TPS:

Use tokenizer (e.g., tiktoken) if available
Or document that TPS is "characters per second" (typically 2-3x higher than true tokens)
For comparison tests, character count is acceptable if consistent

Response Time

Total time from request start to stream completion:

total_ms = (time.monotonic() - start) * 1000

Group-Based Test Design

Test multiple concurrency levels to find breaking points:

GROUPS = [
    {"concurrency": 10, "duration_sec": 300, "prompts": [...]},
    {"concurrency": 50, "duration_sec": 300, "prompts": [...]},
    {"concurrency": 100, "duration_sec": 300, "prompts": [...]},
    {"concurrency": 200, "duration_sec": 300, "prompts": [...]},
    {"concurrency": 500, "duration_sec": 300, "prompts": [...]},
    {"concurrency": 1000, "duration_sec": 300, "prompts": [...]},
]

Key practices:

5 minutes per group (300s) gives stable metrics
10-second rest between groups (let server recover)
Vary prompts to avoid cache effects
Use asyncio.Event() to signal group end, not hard timeouts

Metrics to Report

For each group:

RPS: requests per second (throughput)
Error rate: percentage of failed requests
TTFT: p50, p95, p99 (first token latency)
TPS: p50, p95 (throughput per request)
Response time: p50, p95 (total duration)
Error breakdown: HTTP 4xx/5xx, timeouts, connection errors

Threshold checks:

Pass/fail against requirements (e.g., "TTFT ≤ 200ms")
Identify breaking point (concurrency where error rate > 10%)

Common Pitfalls

1. Database Connection Exhaustion at High Concurrency

When load testing triggers backend code that opens DB connections, high concurrency can exceed the database's max_connections. Two distinct root causes:

Root Cause A: test_sqlor() Validation Overhead (CRITICAL)

Symptom: MariaDB Threads_connected reaches max_connections (e.g., 501/500), hundreds of Sleep connections, "Too many connections" errors.

Root cause: SqlorPool.context() calls test_sqlor() on every idle connection before reuse:

# WRONG — N×M validation queries under high concurrency:
for s in sqlors:
    if not s.used:
        flg = await self.test_sqlor(s.sqlor)  # SELECT 1 on every idle connection

What happens:

1000 concurrent requests × each testing N idle connections = N×1000 unnecessary SELECT 1 queries
Race conditions: multiple requests test the same connection simultaneously
Network timeouts under load cause healthy connections to be marked as "failed" and deleted
Excessive connection creation → connection pileup

Fix: Remove test_sqlor() validation, take first available connection:

# CORRECT — no validation:
for s in sqlors:
    if not s.used:
        yielded_sqlor = s
        break  # Take first available without testing

Verification: After fixing, Threads_connected should stay well below max_connections during load testing.

Root Cause B: Multiple Connection Pools Aggregate Beyond DB Limit

Symptom: Even after fixing test_sqlor(), still hitting connection limits with multiple databases.

Root cause: sqlor creates one SqlorPool per database, each with maxconn=100. Multiple pools can aggregate beyond DB limit:

5 databases × 100 connections each = 500 connections
MariaDB max_connections = 500, but Threads_connected reaches 501+

Solution: Add global connection limit in sqlor/dbpools.py:

_GLOBAL_CONN_LIMIT = 400  # below max_connections=500
_global_conn_sema = asyncio.Semaphore(_GLOBAL_CONN_LIMIT)

async def _new_sqlor(self):
    await _global_conn_sema.acquire()  # Global limit
    try:
        sqlor = await self.create_func()
        # ... create connection
    except Exception:
        _global_conn_sema.release()
        raise

async def _del_sqlor(self, sor):
    # ... close connection
    _global_conn_sema.release()  # Release global slot

Root Cause C: asyncio.create_task() Reference Leaks

Symptom: Background tasks (not t2t calls) hold connections indefinitely.

Root cause: Background asyncio.create_task() calls lose references — if GC collects tasks mid-execution, their DB connections leak:

# BAD: task reference lost, connections never close
asyncio.create_task(long_running_task())

# GOOD: save reference with auto-cleanup
_background_tasks = set()
task = asyncio.create_task(long_running_task())
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)

Note: This only affects code that spawns background tasks (e.g., async inference polling). Pure synchronous/streaming t2t calls don't use create_task() and are not affected by this issue.

Root Cause D: Connection Pool Health Check Snowball (CRITICAL — 2026-06-28)

Symptom: Under 200 concurrent requests, 57% of application logs are "discarding dead connection" (51,813 lines in 3 minutes). Client P50 latency = 4595ms, but vLLM inference is <200ms. Each request wastes ~100ms on DB health checks.

Root cause: SqlorPool.context() calls _check_alive() on every idle connection before reuse:

# WRONG — full DB round-trip for every idle connection:
for s in sqlors:
    if not s.used:
        ok = await self._check_alive(s.sqlor)  # SELECT 1 + transport check
        if ok:
            yielded_sqlor = s
            break

What happens under high concurrency:

200 concurrent requests × each checking N idle connections = N×200 health checks
Connections used 2 seconds ago still get full DB round-trip (wasteful)
Network timeouts under load cause healthy connections to fail health check and be discarded
Creates feedback loop: more discards → more new connections → more health checks

Fix: Add time-based threshold to skip health checks on recently-used connections:

# CORRECT — skip check for warm connections:
CHECK_ALIVE_THRESHOLD = 30  # seconds

for s in sqlors:
    if not s.used:
        idle_time = time.time() - s.use_at
        if idle_time < self.CHECK_ALIVE_THRESHOLD:
            yielded_sqlor = s  # Trust recently-used connection
            break
        ok = await self._check_alive(s.sqlor)  # Only check old connections
        if ok:
            yielded_sqlor = s
            break

Verification: After fix, "discarding dead connection" logs should drop from 51,813 to <100 per 3-minute window. P50 latency should improve by ~100-200ms.

Key insight: Health checks are expensive under load. Recently-used connections (idle <30s) are very likely still healthy — skip the round-trip. Only check connections that have been idle long enough to potentially be closed by the server.

Diagnostic Methodology — CRITICAL

Do NOT assume a root cause before tracing the FULL call chain under the actual load test scenario. A common mistake is to see "connection leak" and immediately fix the most obvious suspect (e.g., background asyncio.create_task() calls) without confirming those code paths are actually exercised during the test.

Required steps before fixing:

Confirm what the test actually calls: Ask "what endpoints/operations are in the test?" — e.g., "all t2t (text-to-text) synchronous/streaming, no background tasks"
Trace the full call chain for that specific scenario: Entry point → DB operations → network calls → DB release. Map every async with get_sor_context() or db.sqlorContext() block
Check for "DB connection held during network IO" anti-pattern: Are there any async with db.sqlorContext() blocks that CONTAIN network calls (HTTP requests, LLM API calls) inside them? This is the classic anti-pattern that causes connection pileup under high concurrency. If found, the fix is to release the DB connection BEFORE the network call.
Only then identify the real root cause — often it's something you wouldn't suspect from static reading (e.g., test_sqlor() validation overhead under concurrency)

Anti-pattern: premature diagnosis

User: "501 Sleep connections, Too many connections"
Agent: "Must be asyncio.create_task() leaks!"  ← WRONG
Agent: *fixes asyncinference.py*
User: "Pressure test has no background tasks, all t2t — you didn't find the real leak point"

The real cause was SqlorPool.context() calling test_sqlor() on every idle connection (N×M queries). Would have been obvious if we traced the t2t call chain first.

Finding (llmage codebase): In t2t synchronous/streaming calls, there is NO "DB connection held during network IO" anti-pattern. Each async with get_sor_context() block is a short, isolated pure-DB operation. The LLM API call happens AFTER all DB connections are released. The connection pileup was caused by connection pool internals (test_sqlor()), not application code.

Verification: Monitor SHOW STATUS LIKE 'Threads_connected' during high-concurrency groups. Should stabilize after fixing all three root causes.

2. Tool Output Redaction Breaks API Keys in Scripts

Hermes write_file redacts API keys inline when writing scripts, producing code that sends masked strings as credentials. Terminal echo also masks keys in output.

Symptom: Load test returns 100% HTTP 403 with "You don't have access to this model" — looks like rate limiting, but actually the key in the script is *** or [REDACTED].

Best fix — read key from config at runtime (avoids redaction entirely):

import yaml, os
config_path = os.path.expanduser('~/.hermes/config.yaml')
with open(config_path) as f:
    config = yaml.safe_load(f)
api_key = config['custom_providers'][0]['api_key']
headers = {'Authorization': 'Bearer ' + api_key}

Fallback — construct key at runtime:

parts = ['sk','-','abc','123']
key = ''.join(parts)

Do NOT hardcode the key string in write_file content or echo commands — it will be masked before hitting disk.

2. Batch Mode Masquerading as Concurrency

# This is batch mode, not true concurrency:
semaphore = asyncio.Semaphore(100)
async with semaphore:
    await request()  # Only 100 concurrent, but batch waits for all

Fix: Use worker pool with persistent workers, not semaphore-limited tasks.

3. Ignoring Stream Errors

# BAD: Silently skip errors
async for line in response.content:
    if not line.startswith("data:"):
        continue  # Missed errors

# GOOD: Check for error messages
async for line in response.content:
    if line.startswith("data:"):
        data = json.loads(line[5:])
        if "error" in data:
            metrics.record_error(data["error"])
            break

4. Character vs Token Confusion

When reporting TPS, always clarify:

"TPS (characters)" if using len(content)
"TPS (tokens)" if using tokenizer
Include both in detailed reports if possible

5. Not Flushing Output

Progress reports get buffered, appear after test completes:

# BAD: Output appears late
print(f"[{elapsed}s] req={count}")

# GOOD: Flush immediately
import sys
print(f"[{elapsed}s] req={count}")
sys.stdout.flush()

6. Misdiagnosing Backend Layer Saturation as Application-Layer Problem

Symptom: Load test shows high RPS (100-1000+/s) but near-100% error rate at high concurrency (200+), with very high TTFT (10-35 seconds). Application-layer fixes (DB connection pools, code optimization) have no effect.

Root cause: The bottleneck is the inference backend (vLLM, TGI, Ollama), not the application layer (sage, sqlor, database). LLM inference backends have limited concurrent request capacity — when overwhelmed, requests queue and timeout.

Diagnostic pattern (LLM API load testing):

Group 1 (10 concurrent):  RPS=0.6,  Error=7%,   TTFT=817ms    ← Backend handling OK
Group 2 (50 concurrent):  RPS=1.2,  Error=43%,  TTFT=1023ms   ← Backend starting to saturate
Group 3 (100 concurrent): RPS=3.9,  Error=92%,  TTFT=7071ms   ← Backend saturated
Group 4 (200 concurrent): RPS=122,  Error=99%,  TTFT=2752ms   ← Backend overwhelmed
Group 5 (500 concurrent): RPS=713,  Error=99%,  TTFT=4689ms   ← Requests timing out in queue
Group 6 (1000 concurrent): RPS=1053, Error=100%, TTFT=35240ms  ← Complete saturation

Key indicators:

RPS increases with concurrency (requests are being sent)
Error rate approaches 100% at high concurrency (requests fail)
TTFT explodes (10-35 seconds = requests stuck in inference queue)
Success count stays low (e.g., only 6-9 successful requests out of 379k)
Application-layer metrics are fine (DB connections normal, no OOM)

What NOT to do:

Don't fix DB connection pools if they're not the bottleneck
Don't optimize application code if inference is the constraint
Don't assume "connection leak" when the real issue is backend capacity

What to do:

Identify the bottleneck layer: Check inference backend logs (vLLM/TGI) for queue depth, OOM, timeout errors
Verify inference capacity: Single GPU instance of Qwen3-0.6B may only handle 10-20 concurrent requests
Scale inference backend: Add more GPU instances, use request queuing/load balancing
Adjust test expectations: If backend can only handle 50 concurrent, don't test at 1000 concurrent

Anti-pattern: Spending days fixing application-layer code (DB pools, connection management) when the real bottleneck is a single GPU instance that can't handle more than 20 concurrent LLM requests. Always identify the bottleneck layer FIRST before optimizing.

Verification: If fixing application-layer code doesn't improve high-concurrency error rates, the bottleneck is likely the inference backend. Check inference logs and scale accordingly.

6a. Finally-Block Double-INSERT in Async Generators (Streaming API)

Symptom: Every successful streaming request hits HTTP 500 with IntegrityError(1062, "Duplicate entry for PRIMARY") in server logs. Error rate ≈ success count — the same number of records succeed as fail because each success triggers a duplicate INSERT.

Root cause: In async generators, finally blocks fire after try completes successfully. If try already called write_llmusage() (INSERT), the finally block unconditionally calls it again with the same primary key.

Fix: Guard with and llmusage.status != 'SUCCEEDED':

# Before (broken):
finally:
    if llmusage and llmusage.get('id') == luid:
        await write_llmusage(llmusage)  # ← unconditional re-INSERT

# After (fixed):
finally:
    if llmusage and llmusage.get('id') == luid and llmusage.status != 'SUCCEEDED':
        await write_llmusage(llmusage)

Real-world: llmage/llmclient.py — 1350+ IntegrityErrors in 180s at 200 concurrent, dropped to 0 after fix. HTTP 500 reduced 35%. See references/finally-double-insert-async-generator.md.

6b. MoE Reasoning Models Unsuitable for High-Concurrency Stress Tests

Symptom: TTFT >30s, avg tokens/response >1000, quota exhausts in <100 requests, 60-89% error at 200 concurrent regardless of infrastructure fixes.

Root cause: Models like qwen3.6-35b-a3b generate ~900 reasoning tokens before answering. Each "hi" → ~1100 tokens total. 200 concurrent × 30-50s response = massive queue & timeout.

Fix: Use smaller non-reasoning models for load testing (e.g., Qwen3-0.6B). See references/reasoning-model-load-test-limits.md.

6c. Nested Try/Except + Finally — UnboundLocalError in Async Generators

Symptom: Streaming inference crashes with UnboundLocalError: local variable 'llmusage' referenced before assignment in the finally block. Happens intermittently — only when the inner try's uapi.call() raises before llmusage is initialized.

Root cause: Nested try/except structure in async generator:

# asyncinference.py — BROKEN
try:                          # outer try
    try:                      # inner try
        b = await uapi.call(...)
    except Exception:         # inner except
        yield error; return   # ← returns WITHOUT initializing llmusage!
    llmusage = DictObject()   # ← only reached if inner try succeeds
    ...
except Exception:              # outer except
    llmusage = DictObject()   # ← only for outer exceptions
finally:
    await write_llmusage(llmusage)  # 💥 UnboundLocalError

The inner except (submission failure) returns early without llmusage being assigned. The outer finally then references the unbound variable.

Fix — two changes:

# 1. Initialize before the outer try
llmusage = None

try:
    ...
finally:
    # 2. Guard the finally
    if llmusage is not None:
        await write_llmusage(llmusage)

Audit pattern: Search all async generators for finally blocks that reference variables assigned inside try/except:

grep -n 'finally:' **/*.py | while read line; do
    # Check if the finally references a variable not initialized before the outer try
done

Real-world: llmage/asyncinference.py line 166 — triggered when uapi.call() submission fails in the inner try. Fixed by adding llmusage = None before outer try + if llmusage is not None guard. Verified syncinference.py and llmclient.py (in the same repo) did NOT have this bug — syncinference uses single-level try and llmclient already had the = None initialization.

7. HTTP Client Resource Leaks (SSL Contexts, Connection Pools)

Symptom: Under high concurrency (100+ parallel requests), system memory grows continuously. No obvious memory leak in application code, but RSS increases steadily during load test.

Root cause: HTTP client libraries creating expensive resources per-instance instead of sharing:

# WRONG — each client instance loads full CA cert chain (~5-10KB):
class StreamHttpClient:
    def __init__(self):
        self.ssl_context = ssl.create_default_context(cafile=certifi.where())  # Expensive!
# 100 concurrent requests × 100 client instances = 100 SSL contexts = 500KB-1MB leak

What happens:

Each StreamHttpClient() calls ssl.create_default_context(cafile=certifi.where())
This loads the full CA certificate chain into memory (5-10KB per context)
Under high concurrency, hundreds of instances create hundreds of duplicate SSL contexts
Memory grows linearly with request count, never released

Fix: Use module-level singleton for expensive resources:

# CORRECT — shared SSL context across all instances:
_SHARED_SSL_CONTEXT = None
def _get_shared_ssl_context():
    global _SHARED_SSL_CONTEXT
    if _SHARED_SSL_CONTEXT is None:
        _SHARED_SSL_CONTEXT = ssl.create_default_context(cafile=certifi.where())
    return _SHARED_SSL_CONTEXT

class StreamHttpClient:
    def __init__(self):
        self.ssl_context = _get_shared_ssl_context()  # Reuse singleton

Verification: Create 20+ client instances and check unique SSL context IDs:

clients = [StreamHttpClient() for _ in range(20)]
contexts = set(id(c.ssl_context) for c in clients)
assert len(contexts) == 1, f"Expected 1 shared context, found {len(contexts)}"

Common resources that leak this way:

SSL contexts (ssl.create_default_context())
aiohttp TCPConnector instances
Large config files loaded per-instance
Database connection pools (if not using singleton pattern)

Diagnostic pattern for memory leaks under load:

Monitor RSS during load test (ps aux | grep python)
If memory grows linearly with request count (not stabilizing), suspect per-instance resource creation
Trace the request path to find __init__() methods that create expensive objects
Check for ssl.create_default_context(), aiohttp.ClientSession(), file reads, etc.
Convert to module-level singletons or shared pools

Rule: Any resource that takes >1ms to create or uses >1KB of memory should be shared across instances, not created per-instance.

8. Dead DB Connections After Backend OOM — Restart Upstream Before Re-Testing

Symptom: After fixing inference backend (e.g., restarting vLLM after CUDA OOM), the upstream API service (sage.py/token server) still throws TCPTransport closed=True errors on every DB query:

RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x...>; the handler is closed

Root cause: During the OOM event, the inference backend was unresponsive → HTTP requests from the upstream service timed out → the upstream's async HTTP client closed connections → the upstream's DB connection pool still holds these now-dead MySQL connections. Subsequent requests reuse the dead connections and fail.

Fix: Restart the upstream service (e.g., ./stop.sh && ./start.sh on the token/sage server) to flush the connection pool. Without this, re-running the load test will show 100% errors even though the backend is now healthy.

Diagnostic checklist before re-running a load test after any backend incident:

Confirm backend is healthy (no OOM, processes running, responding to ad-hoc requests)
Confirm upstream service's DB pool is fresh (restart if any OOM/timeout happened during previous test)
Send 1 ad-hoc test request through the full chain before launching the full load test

Anti-pattern: Re-running the load test immediately after fixing the backend, without restarting the upstream service. The test will fail with DB errors that look like a new bug but are just stale connections.

9. Distinguishing 403 Sources: Gateway vs Rate Limit vs Invalid Key

When requests return 403, determine which layer rejects them:

Layer	Response body	Cause
Nginx `blocked_ip`	Plain 403, no JSON body	IP in blacklist
Gateway (invalid/masked key)	`{"error": {"message": "You don't have access to this model", "code": "model_not_found"}}`	Key is wrong, masked, or redacted
Gateway (rate limit)	Same JSON as above	Key is valid but throttled

Diagnostic steps:

Single curl with known-good key → 200: key works, problem is concurrency
Single curl with key from script → 403: script has masked/redacted key (see Pitfall #2)
All concurrent requests → 403, single curl → 200: rate limiting or masked key
Bypass gateway, hit backend directly: curl http://localhost:9089/v1/chat/completions — if 200, problem is gateway layer

Real-world case: write_file redacted API key to ***, stress test sent masked key, all 2500 requests returned 403. Looked like rate limiting but was invalid credentials. Fix: read key from config.yaml at runtime.

Multi-GPU vLLM Deployment: Instance Count per GPU

Symptom: Load test shows near-100% errors at high concurrency. Backend logs show:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB.
GPU 0 has a total capacity of 23.52 GiB of which 384.00 KiB is free.

OOM happens during KV cache expansion:

self.keys = torch.cat([self.keys, key_states], dim=-2)  # can't allocate even 2 MiB

Root cause: Multiple vLLM instances on the same GPU without --max-model-len and --max-num-seqs limits. Default values try to pre-allocate KV cache for 256 concurrent sequences × model max length (e.g., 32K for Qwen3), which exhausts GPU memory when multiple instances activate simultaneously.

Deployment Pattern: 1 Instance per GPU (Recommended for Stability)

When to use: Production deployments prioritizing stability over instance count. Each GPU has full memory for one instance with maximum utilization.

#!/bin/bash
# start_single.sh - One instance per GPU, ports 9089-9096
GPU_UTIL=0.90  # Higher utilization when only 1 instance

for gpu in {0..7}; do
    port=$((9089 + gpu))
    CUDA_VISIBLE_DEVICES=$gpu vllm serve /d/models/Qwen/Qwen3-0.6B \
        --port $port \
        --gpu-memory-utilization $GPU_UTIL \
        --served-model-name Qwen3-0.6B \
        > logs/gpu${gpu}_p${port}.log 2>&1 &
done

Advantages:

No memory contention between instances
Higher --gpu-memory-utilization (0.90 vs 0.45)
Simpler troubleshooting (1 instance per GPU)
Full GPU memory available for KV cache

Memory allocation (24GB GPU example):

Model weights: ~1.2 GB (Qwen3-0.6B)
Available for KV cache: ~20.8 GB
No need to limit --max-model-len or --max-num-seqs as aggressively

Deployment Pattern: 2 Instances per GPU (Higher Throughput, Lower Stability)

When to use: Testing/development where you want more total instances but accept lower per-instance memory.

#!/bin/bash
# start_dual.sh - Two instances per GPU, ports 9089-9104
GPU_UTIL=0.45  # Split memory between 2 instances

for gpu in {0..7}; do
    port1=$((9089 + gpu * 2))
    port2=$((9090 + gpu * 2))
    CUDA_VISIBLE_DEVICES=$gpu vllm serve /d/models/Qwen/Qwen3-0.6B \
        --port $port1 \
        --gpu-memory-utilization $GPU_UTIL \
        --max-model-len 2048 \
        --max-num-seqs 32 \
        > logs/gpu${gpu}_p${port1}.log 2>&1 &
    CUDA_VISIBLE_DEVICES=$gpu vllm serve /d/models/Qwen/Qwen3-0.6B \
        --port $port2 \
        --gpu-memory-utilization $GPU_UTIL \
        --max-model-len 2048 \
        --max-num-seqs 32 \
        > logs/gpu${gpu}_p${port2}.log 2>&1 &
done

Critical parameters for 2-instances-per-GPU:

--max-model-len 2048 (or appropriate for your prompt length) — limits KV cache per sequence
--max-num-seqs 32 (or 64) — limits concurrent sequences per instance
Without these, vLLM tries to allocate KV cache for 256 seqs × 32K tokens = impossible under load

Memory allocation (24GB GPU example):

Model weights: ~1.2 GB × 2 = 2.4 GB
Available for KV cache: ~21.6 GB ÷ 2 = ~10.8 GB per instance
Rule of thumb: max-model-len × max-num-seqs ≤ 65536 (e.g., 2048 × 32)

Choosing Between Patterns

Scenario	Pattern	Instances	Utilization	Stability
Production	1 per GPU	8	0.90	High
Load testing	2 per GPU	16	0.45	Medium
Development	1 per GPU	8	0.90	High

Diagnostic: Run nvidia-smi to check per-GPU memory usage. If each instance uses 1.5-2 GB (model weights) and GPU has 24 GB total, calculate remaining for KV cache.

⚠️ CRITICAL: Use nvidia-smi, not ps, to diagnose GPU allocation

Anti-pattern: Running ps -ef | grep vllm and seeing 40 processes, then assuming all 40 are on one GPU. This leads to wrong conclusions like "40 workers fighting for 24GB = OOM".

Reality: ps doesn't show CUDA_VISIBLE_DEVICES. The 40 processes could be 8 GPUs × 2 instances each (16 processes) + worker threads, or 1 GPU × 40 instances. You cannot tell from ps alone.

Correct diagnostic workflow:

Run nvidia-smi to see actual GPU allocation:

GPU 0: 2 × vLLM (1824 MiB each) = 3.6 GB / 24 GB
GPU 1: 2 × vLLM (1824 MiB each) = 3.6 GB / 24 GB
...
GPU 7: 2 × vLLM (1824 MiB each) = 3.6 GB / 24 GB

This shows 8 GPUs × 2 instances = 16 instances total, not "40 processes on one GPU"
Now you know the real constraint: each GPU has ~20 GB for KV cache across 2 instances

Why this matters: Misdiagnosing GPU allocation leads to wrong fixes. If you think 40 processes share one GPU, you'd kill 38 of them. But if 8 GPUs × 2 instances is the intended deployment, killing processes breaks the load balancer. Always verify with nvidia-smi first.

9. vLLM Model Naming: Full Path vs Short Name

Symptom: Load test shows 100% error rate with "Model not found" errors, even though the endpoint is responding and curl localhost:9089/v1/models returns valid data.

Root cause: When serving a local model path (e.g., /d/models/Qwen/Qwen3-0.6B), vLLM uses the full path as the model ID, not just the last segment:

# What vLLM returns:
curl localhost:9089/v1/models
{"data": [{"id": "/d/models/Qwen/Qwen3-0.6B", ...}]}

# What the test script sends (WRONG):
curl ... -d '{"model": "Qwen3-0.6B"}'
{"error": "The model `Qwen3-0.6B` does not exist."}

# What works:
curl ... -d '{"model": "/d/models/Qwen/Qwen3-0.6B"}'

Fix in test script: Update the MODEL constant to match the full path:

MODEL = "/d/models/Qwen/Qwen3-0.6B"  # Not just "Qwen3-0.6B"

Fix in vLLM startup: Use --served-model-name to override:

vllm serve /d/models/Qwen/Qwen3-0.6B --served-model-name Qwen3-0.6B

Diagnostic: When load test shows 100% errors with high RPS (requests are reaching the server), check:

curl <endpoint>/v1/models — what does it return as the model ID?
Compare with the model field in your test script's payload
If they don't match exactly, that's the issue

Anti-pattern: Assuming "model not found" means the server is down or misconfigured. The server is working fine — it just expects a different model name than you're sending.

10. Load Balancer Hitting Dead Backends: Verify All Instances Before Testing

Symptom: Load test shows mixed errors (403, 502) even though some requests succeed. Single curl tests work fine, but concurrent requests fail intermittently or consistently.

Root cause: Multiple vLLM instances deployed behind nginx load balancer, but some instances:

Failed to start (ports not listening)
Started with wrong model name (missing --served-model-name)
Crashed during model loading

What happens:

nginx upstream has 24 servers configured (e.g., ports 9089-9112)
Only 22 instances are actually running with correct model names
Load balancer sends requests to all 24 upstreams
Requests hitting dead/misconfigured backends return 403 or 502

Diagnostic checklist before running load tests:

Count running processes vs expected:

ps aux | grep vllm.entrypoints | grep -v grep | wc -l
# Should match expected count (e.g., 24 for 8 GPUs × 3 instances)

Count listening ports vs expected:

ss -tlnp | grep -E '908[0-9]|909[0-9]|910[0-9]|911[0-2]' | wc -l

Verify model name on ALL instances:

for port in $(seq 9089 9112); do
    model=$(curl -s http://127.0.0.1:$port/v1/models | python3 -c 'import sys,json; print(json.load(sys.stdin)["data"][0]["id"])' 2>/dev/null)
    [ "$model" = "Qwen3-0.6B" ] || echo "BAD: Port $port has model: $model"
done

Check for missing ports:

for p in $(seq 9089 9112); do
    ss -tlnp | grep -q :$p || echo "MISSING: $p"
done

Fix: Restart failed instances, ensure all use --served-model-name Qwen3-0.6B.

Anti-pattern: Running load test immediately after deployment without verifying all instances. Single curl tests work because they hit a working backend, but load tests expose the dead ones. Always verify the full instance count and model names before stress testing.

Reference Architecture

See templates/worker-pool-async.py for a complete async worker-pool load test script with TTFT/completion metrics, multi-group comparison, and per-minute throughput breakdown.

Key pattern: each request gets unique prompt via incrementing counter (f"请用一句话介绍你自己，编号{idx}") to defeat LLM prefix caching.

Group Transition: Wait for All Requests to Complete

User requirement: Before starting the next group, ALL in-flight requests from the current group must finish (succeed or fail). Do NOT move on while requests are still pending.

Why: Partial results from a previous group contaminating the next group's metrics make the data unreliable. If 500 requests are still in-flight when group 2 starts, the server is handling group-1 leftovers + group-2 fresh load simultaneously.

Implementation pattern:

# After setting stop_event, wait for ALL workers with generous timeout
stop_event.set()

# Generous timeout: at least 2x the longest expected response time
# For high-concurrency groups with slow backends, 300s is safer than 60s
try:
    await asyncio.wait_for(asyncio.gather(*tasks, return_exceptions=True), timeout=300)
except asyncio.TimeoutError:
    # Log warning but don't proceed until truly done
    print(f"  WARNING: some requests still in-flight after 300s, cancelling...")
    for t in tasks:
        t.cancel()
    await asyncio.gather(*tasks, return_exceptions=True)

# Verify no active workers before proceeding
_, _, _, active = metrics.snapshot()
if active > 0:
    print(f"  WARNING: {active} workers still active")

Anti-pattern: Using a 60-second timeout then proceeding regardless. At 1000 concurrency with a slow backend, individual requests can take 30-60+ seconds. The timeout fires mid-flight and the next group starts with a polluted server.

nginx Upstream Load Balancing for Multiple vLLM Instances

When deploying multiple vLLM instances (e.g., 1 per GPU × 8 GPUs), use nginx upstream with least_conn:

upstream qwen3_backend {
    least_conn;
    server 127.0.0.1:9089;
    server 127.0.0.1:9090;
    server 127.0.0.1:9091;
    server 127.0.0.1:9092;
    server 127.0.0.1:9093;
    server 127.0.0.1:9094;
    server 127.0.0.1:9095;
    server 127.0.0.1:9096;
}

location ^~ /qwen3-6b/ {
    proxy_pass http://qwen3_backend/;
    # ... standard headers
}

least_conn is preferred over round_robin for LLM workloads because request durations vary wildly (short prompts vs long generation).

Pitfall: nginx IP Blacklist Blocks External Load Testing

Symptom: curl https://api.example.com/v1/chat/completions returns 403 Forbidden from outside the server, but works fine from localhost.

Root cause: nginx config has $blocked_ip variable that returns 403 for IPs not on a whitelist. Load testing from external machines (including the agent's host) hits the blacklist.

Fix: Run the load test script on the GPU server itself via SSH:

# Upload script to server
scp stress_test.py user@gpu-server:/tmp/

# Run remotely
ssh user@gpu-server "python3 /tmp/stress_test.py"

Diagnostic: If all requests fail with http_403 errors, check nginx config:

grep -r 'blocked_ip' /etc/nginx/

Anti-pattern: Changing the test URL to localhost to bypass the issue — this skips testing the actual production nginx chain. If the goal is testing the full stack (nginx → upstream → vLLM), run from the server with the production URL.

11. 403 Layer Isolation in Multi-Hop API Chains

Symptom: Requests return 403, but it's unclear which layer is rejecting — the auth/token proxy, the load balancer, or the backend service.

Diagnostic methodology — test each hop independently, from outside in:

# Step 1: Token endpoint without auth (expect 401, confirms endpoint reachable)
curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/api/v1/models

# Step 2: Token endpoint with auth, known-working model
curl -s -w "\n---HTTP_CODE:%{http_code}---" 'https://proxy.example.com/api/v1/chat/completions' \
  -H 'Authorization: Bearer <key>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"known-good-model","messages":[{"role":"user","content":"hi"}],"stream":false,"max_tokens":10}'

# Step 3: Token endpoint with auth, problematic model
curl -s -w "\n---HTTP_CODE:%{http_code}---" 'https://proxy.example.com/api/v1/chat/completions' \
  -H 'Authorization: Bearer <key>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"problematic-model","messages":[{"role":"user","content":"hi"}],"stream":false,"max_tokens":10}'

# Step 4: Backend direct (SSH to server, bypass proxy)
ssh user@server 'curl -s -w "\n---HTTP_CODE:%{http_code}---" http://localhost:9089/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"problematic-model\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"stream\":false,\"max_tokens\":10}"'

Interpretation:

Step 2 fails → token/auth layer issue (bad key, IP blacklist, expired token)
Step 2 OK, Step 3 fails → model-specific issue (model not registered, wrong name, backend down)
Step 3 OK, Step 4 fails → proxy routing issue (proxy can't reach backend)
Step 4 OK but Step 3 fails → proxy misconfiguration (wrong upstream, missing route)

Anti-pattern: Assuming 403 is always from the auth layer. In multi-hop chains, any layer can return 403. Always isolate by testing each hop.

12. Quick Sync Load Test with ThreadPoolExecutor

For quick one-off tests (1-5 minutes, ≤200 concurrency), sync requests + ThreadPoolExecutor is simpler than async setup:

import requests, time, statistics
from concurrent.futures import ThreadPoolExecutor, as_completed
from collections import Counter

def make_request():
    start = time.time()
    try:
        r = requests.post(URL, headers=HEADERS, json=PAYLOAD, timeout=30)
        latency = (time.time() - start) * 1000
        return ('success' if r.status_code == 200 else 'error', latency,
                None if r.status_code == 200 else f"HTTP {r.status_code}")
    except Exception as e:
        return ('error', (time.time() - start) * 1000, str(e))

# Maintain constant concurrency by refilling completed futures
futures = [executor.submit(make_request) for _ in range(CONCURRENCY)]
while time.time() - start_time < DURATION:
    done = [f for f in futures if f.done()]
    for f in done:
        # collect metrics
        futures.remove(f)
    while len(futures) < CONCURRENCY:
        futures.append(executor.submit(make_request))
    time.sleep(0.01)

When to use sync: Quick smoke tests, no streaming, simple POST APIs. When to use async: Streaming, >500 concurrency, long-running tests, TTFT measurement.

Pitfall: requests + threads use 1 thread per connection. At 1000+ concurrency, thread overhead dominates. Switch to async at that point.

13. Server-Side Log Analysis: Read Logs Before Making Claims

User correction: "分析性能问题必须先取服务端日志再做推论，禁止仅凭客户端数据猜测瓶颈"

Anti-pattern: Running a load test, seeing high P50 latency, and immediately concluding "the bottleneck is X" without reading actual server-side application logs. This leads to contradictory conclusions and wasted debugging time.

Required diagnostic sequence:

Run the load test → collect client-side metrics (RPS, P50, error rate, error types)
Get server-side application logs for the test time window → the authoritative source
Analyze logs → identify the actual bottleneck layer and root cause
Cross-validate → ensure client metrics are consistent with server-side findings

User mandate — "拿测试数据说话": Never present conclusions without actual measured data. Speculative claims about expected improvement are unacceptable. Run the test, collect data, present numbers.

Required diagnostic sequence:

Run the load test → collect client-side metrics (RPS, P50, error rate, error types)
Get server-side application logs for the test time window → the authoritative source
Analyze logs → identify the actual bottleneck layer and root cause
Cross-validate → ensure client metrics are consistent with server-side findings

Real-world case (token.opencomputing.cn, 200 concurrent):

Client: P50=4595ms, 98% success, 33 RPS, 89 connection timeouts
Without logs, agent guessed: "llmage is a single-point bottleneck" — WRONG
Server logs revealed: 51,813 "discarding dead connection" lines in 3 minutes (57% of all logs)
Each request discarded ~12 dead DB connections before proceeding
Root cause: sqlor _check_alive() snowball, NOT llmage, NOT vLLM
Fix: optimize connection pool health check strategy, not gateway scaling

How to get server-side logs:

Method 1: SSH and read log files directly

~/logs/sage.log              # Main application log
~/logs/backend_accounting.log # Accounting backend log
grep '2026-06-28 23:1[7-9]' ~/logs/sage.log | wc -l
grep '2026-06-28 23:1[7-9]' ~/logs/sage.log | grep -c 'discarding dead connection'

Method 2: Use bugfix module API (when available on the platform)

# User says "用bugfix从服务端获得日志" — the bugfix module exposes log retrieval
# The exact endpoint varies; ask the user or check platform docs

Real-world case (token.opencomputing.cn, 2026-06-28, 200 concurrent):

Client: P50=3952ms, 97.6% success, 34.1 RPS
Server logs (sage.log): 67,026 total lines, 26,852 "discarding dead connection" (40%)
Before sqlor fix: 57% discarding → after fix: 40% (48% reduction)
vLLM engine logs: all 24 instances Running 0-2, Waiting 0 — NOT the bottleneck
Remaining 3300ms P50 gap: token平台 llmage HTTP client connection pool queuing

What to look for in logs:

Error counts by type (connection errors, timeouts, exceptions)
Per-request processing chain (timestamp gaps = time spent in each step)
Resource contention signals ("discarding dead connection", "pool exhausted", "queue full")
Request distribution across backend instances (imbalance = load balancer issue)

Consistency check: If you analyze backend engine logs (e.g., vLLM) and see "Running 0-2 reqs, Waiting 0", but client latency is 4.5s, the bottleneck is NOT the backend — it's the middleware layer between client and backend. But verify this with application logs, not just inference.

14. Token Display Truncation by read_file/cat — `...` is NOT Literal

Symptom: All requests return 403 "You don't have access to this model" for every model, even models that appear in /v1/models. The API key looks valid in cat/read_file output but consistently fails.

Root cause: read_file and cat truncate long tokens in their display output, replacing middle characters with literal .... Example: the actual token V9J41PngWBUU6gdHWJWDJ displays as V9J41P...JWDJ. You then hardcode the truncated version and it's wrong.

Diagnostic — use hexdump to see real bytes:

grep 'Authorization' ~/curl/oc1.curl | sed 's/.*Bearer //' | tr -d '\\"' | xxd
# Output shows actual bytes — no '...' present

Fix: Always verify a token's raw bytes when it comes from read_file/cat:

# Wrong: assume displayed text is complete
TOKEN = "V9J41P...JWDJ"  # '...' is literal in this string!

# Right: extract from hex or original source
TOKEN = "V9J41PngWBUU6gdHWJWDJ"  # Actual 21-char token

Anti-pattern: Copying the display-truncated ... token into code and testing 10 different models trying to find one that "works" — when the issue is the key itself, not model access.

15. API Gateway Health Check Creates Phantom Bottleneck

Symptom: Load test through an API gateway (e.g., llmage) shows intermittent "No healthy upstream" errors and severe load imbalance. Some runs show 91% error rate, others show 98% success but P50 latency 10x higher than expected.

Root cause: The gateway's health check periodically probes backend instances. Under load, health probes timeout or fail, causing the gateway to mark healthy instances as unhealthy. Traffic concentrates on the few instances still marked healthy, creating a feedback loop: more load → slower health responses → more instances marked unhealthy → even more concentrated load.

Real-world case (24 vLLM instances behind llmage gateway):

Port 9089 received 3840 requests, other 23 instances received ~180 each (21:1 ratio)
vLLM engine logs showed Running 0-2 reqs, Waiting 0 — instances were nearly idle
Client measured P50=4595ms, but vLLM actual inference was <200ms
4.4 second gap = gateway queuing/routing overhead

Diagnostic methodology — three-layer comparison:

# Layer 1: Client-side metrics (from load test script)
# RPS, P50, P99, error rate — these measure the FULL chain

# Layer 2: Per-instance request distribution (from backend logs)
for f in /path/to/logs/gpu*_inst*_p*.log; do
    port=$(basename $f | grep -oP 'p\K[0-9]+')
    count=$(grep -c 'POST /v1/chat/completions' $f)
    echo "$port: $count"
done | sort -t: -k1 -n
# If one instance has 10x+ requests vs others → gateway imbalance

# Layer 3: Backend engine throughput (from engine logs)
grep 'Engine 000' /path/to/logs/instance.log | grep '22:3[0-6]'
# Look for Running/Waiting counts and generation throughput
# If Running ≤ 3 and Waiting = 0 → backend is NOT the bottleneck

Interpretation matrix:

Client latency	Backend Running	Bottleneck
High (seconds)	0-2, Waiting 0	Gateway/middleware
High (seconds)	20+, Waiting >0	Backend saturated
Low (<500ms)	0-1, Waiting 0	No bottleneck, healthy

Fix options:

Tune gateway health check: longer timeout, higher failure threshold, longer interval
Add gateway-level caching or connection pooling
Scale gateway horizontally (multiple gateway instances)
Reduce max-num-seqs on backends so health probes complete faster under load

Anti-pattern: Seeing high P50 latency and immediately trying to optimize the backend. If backend engine logs show idle instances (Running 0-2, Waiting 0), the bottleneck is the middleware/gateway layer, not the backend.

16. CPU-Bound Middleware Under I/O-Heavy Load — The "Why Is CPU at 100%?" Signal

Symptom: API gateway/middleware server (e.g., Sage, token platform) shows CPU near 100% during load test, but the server's primary job is proxying requests to LLM backends. Since await on network I/O should leave the event loop idle, high CPU indicates the server is burning cycles on local work it shouldn't be doing.

User insight (critical diagnostic question): "如果服务器在等 LLM 服务器响应，CPU 不该这么重的负荷，而应该在网络和 IO" — this is correct. An async Python gateway proxying to external backends should be mostly I/O-bound. CPU at 100% signals a local bottleneck.

Case study: 4-core server with NO GPU, proxying streaming requests to external model API. CPU saturated at just 30 concurrent connections (TTFB degraded 7.4x vs baseline). Root cause: per-chunk json.loads() in the streaming proxy loop, NOT inference (model runs elsewhere). Full analysis in references/streaming-proxy-cpu-bottleneck.md.

What to look for (in priority order):

Finally-block double-writes in streaming generators: If every successful request calls write_usage_record() in the try block AND again unconditionally in the finally block, each request triggers an IntegrityError (duplicate key) that burns CPU on exception construction + DB round-trip. Example: llmage/llmclient.py lines 94 + 130 — write_llmusage() called in try (success) then again in finally (unconditional re-INSERT). Fix: guard finally with and record.status != 'SUCCEEDED'.
Uncached DB lookups on every request: Count how many DB operations happen per request in the hot path. Each DB operation = asyncio semaphore contention + network round-trip. At 2000 concurrent with 5-6 DB ops per request = 10,000-12,000 competing operations. Example llmage chat/completions call chain: get_user() → checkCustomerBalance() → get_llm() (3-table JOIN) → write_llmusage() × 2 (finally bug). See references/llmage-chat-completions-db-trace.md.
Connection pool health check snowball: Even with a CHECK_ALIVE_THRESHOLD, under sustained load connections age past the threshold and trigger health checks, which are DB round-trips. At scale, thousands of health checks per second compete for pool slots.
N+1 cache-miss cascades: If cache TTL is short relative to request duration, cache entries expire mid-flight and trigger fresh DB lookups for in-flight requests.

Diagnostic workflow:

Step 1: Check server CPU (top/htop) → near 100%? → local bottleneck, not backend
Step 2: Trace 1 request's full DB call chain (read the DSPY + all imported functions)
Step 3: Count DB operations per request × concurrency → compare to pool limits
Step 4: Check for "write then re-write" patterns in finally/cleanup blocks
Step 5: Check for uncached lookups that could be cached (model config, balance, pricing)

Real-world case (token.opencomputing.cn, Sage/llmage, 2000 concurrent):

CPU near 100%, high failure rate, duplicate llmusage inserts
Root cause #1: llmclient.py finally block double-INSERT on write_llmusage()
Root cause #2: checkCustomerBalance() hits pricing DB on every request (no middleware cache)
Root cause #3: get_llm() → 3-table JOIN on every request despite model config rarely changing
Total: ~6 DB ops per request, 2 redundant (one in finally, one uncached)
Full DB call-chain trace: references/llmage-chat-completions-db-trace.md

Fix checklist:

Add and record.status != 'SUCCEEDED' guard to all streaming-generator finally blocks
Add process-level or Redis cache for hot-path lookups (model config, balance, pricing)
Raise global connection pool limits to handle peak concurrency
Verify nginx keepalive to backend is configured (avoids TCP handshake overhead)

CPU Monitoring via /proc/stat

For load tests where CPU saturation is the suspected bottleneck, sample CPU usage concurrent with the test. No external tools needed — parse /proc/stat directly:

async def cpu_sampler(stop_event):
    samples = []
    while not stop_event.is_set():
        with open("/proc/stat") as f:
            parts = f.readline().split()
        total = sum(int(x) for x in parts[1:])
        idle = int(parts[4])
        samples.append((time.time(), total, idle))
        await asyncio.sleep(1)
    return samples

Compute per-second CPU usage from consecutive samples: usage = 100 * (1 - delta_idle / delta_total).

Full integrated script with CPU sampling: scripts/streaming-proxy-loadtest.py.

For streaming proxy load tests with system resource monitoring, use templates/stream-stress-cpu-net-io.py which adds:

CPU sampling via /proc/stat (per-second, no external tools)
Network I/O via /proc/net/dev (RX/TX KB/s)
Disk IOPS via /proc/diskstats (read/write IOPS)
Error classification: timeout vs HTTP status codes
JSON report output: per-concurrency files for comparison

Run at 30/60/100 concurrency to find the saturation point where TTFB degrades and failures begin.

For a complete standalone script with worker-pool, real-time 5-second progress reporter, error classification, system monitoring, and JSON report output, copy scripts/sustained-worker-pool-stress.py — edit the URL/API_KEY/MODEL/PROMPTS at the top and run with python3 script.py <concurrency> <duration_sec>.

Verification

After test completes:

Check error rate trend — should increase with concurrency
Check RPS trend — should plateau or drop at breaking point
Check TTFT trend — should increase with load
Verify no systematic errors (all timeouts vs mixed 5xx)

For HTTP API testing: use httpx or aiohttp (async), not requests (sync)
For WebSocket testing: see websocket-load-testing skill (if exists)
For database load testing: use connection pool metrics, not query count

54 KiB Raw Blame History Unescape Escape

API Load Testing

When to Use

Architecture: Worker Pool vs Batch Mode

Worker Pool Pattern (Correct)

Batch Mode for Burst Testing

When to Use Batch Mode

Implementation Pattern

Anti-pattern: Incomplete Wait

When NOT to Use Batch Mode

Streaming Metrics Collection

TTFT (Time To First Token)

TPS (Tokens Per Second)

Response Time

Group-Based Test Design

Metrics to Report

Common Pitfalls

1. Database Connection Exhaustion at High Concurrency

Root Cause A: test_sqlor() Validation Overhead (CRITICAL)

Root Cause B: Multiple Connection Pools Aggregate Beyond DB Limit

Root Cause C: asyncio.create_task() Reference Leaks

Root Cause D: Connection Pool Health Check Snowball (CRITICAL — 2026-06-28)

Diagnostic Methodology — CRITICAL

2. Tool Output Redaction Breaks API Keys in Scripts

2. Batch Mode Masquerading as Concurrency

3. Ignoring Stream Errors

4. Character vs Token Confusion

5. Not Flushing Output

6. Misdiagnosing Backend Layer Saturation as Application-Layer Problem

6a. Finally-Block Double-INSERT in Async Generators (Streaming API)

6b. MoE Reasoning Models Unsuitable for High-Concurrency Stress Tests

6c. Nested Try/Except + Finally — UnboundLocalError in Async Generators

7. HTTP Client Resource Leaks (SSL Contexts, Connection Pools)

8. Dead DB Connections After Backend OOM — Restart Upstream Before Re-Testing

9. Distinguishing 403 Sources: Gateway vs Rate Limit vs Invalid Key

Multi-GPU vLLM Deployment: Instance Count per GPU

Deployment Pattern: 1 Instance per GPU (Recommended for Stability)

Deployment Pattern: 2 Instances per GPU (Higher Throughput, Lower Stability)

Choosing Between Patterns

9. vLLM Model Naming: Full Path vs Short Name

10. Load Balancer Hitting Dead Backends: Verify All Instances Before Testing

Reference Architecture

Group Transition: Wait for All Requests to Complete

nginx Upstream Load Balancing for Multiple vLLM Instances

Pitfall: nginx IP Blacklist Blocks External Load Testing

11. 403 Layer Isolation in Multi-Hop API Chains

12. Quick Sync Load Test with ThreadPoolExecutor

13. Server-Side Log Analysis: Read Logs Before Making Claims

14. Token Display Truncation by read_file/cat — ... is NOT Literal

15. API Gateway Health Check Creates Phantom Bottleneck

16. CPU-Bound Middleware Under I/O-Heavy Load — The "Why Is CPU at 100%?" Signal

CPU Monitoring via /proc/stat

Verification

Related

54 KiB

Raw Blame History

14. Token Display Truncation by read_file/cat — `...` is NOT Literal