yumoqing/pipeline_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pipeline_core/skills_library/all/vllm-multi-instance-gpu/SKILL.md

12 KiB
Raw Blame History

name description version metadata
vllm-multi-instance-gpu Run multiple independent vLLM instances per GPU for small models. Covers memory budgeting, sequential startup, orphan process cleanup, and Nginx load balancing. 1.0.0
hermes
tags
vLLM
multi-instance
GPU sharing
inference
deployment
Nginx
load-balancing

vLLM Multi-Instance Per GPU Deployment

When to use

  • Small models (≤7B params) that don't need tensor parallelism
  • Want to maximize throughput by running N independent instances per GPU
  • Each instance serves its own request queue independently (no inter-instance communication)

Architecture

GPU 0: [vLLM inst1 :9089] [vLLM inst2 :9090] [vLLM inst3 :9091]
GPU 1: [vLLM inst1 :9092] [vLLM inst2 :9093] [vLLM inst3 :9094]
...
         ↓ Nginx upstream (least_conn)
    Single entry point :10443
  • Each instance gets CUDA_VISIBLE_DEVICES=<gpu_id> (single GPU)
  • Each instance binds a unique port — vLLM does NOT support SO_REUSEPORT
  • Nginx upstream block with least_conn load-balances across all instances

Memory budget

For N instances per GPU:

gpu_memory_utilization_per_instance = 0.90 / N

Example (24GB GPU, 0.6B model, 3 instances):

  • Per instance: 0.30 × 24GB = 7.2GB
  • Model weights (0.6B bf16): ~1.2GB
  • KV Cache: ~6GB (enough for max_model_len=2048, max_num_seqs=32)

Startup script (sequential launch required)

#!/bin/bash
set -e

# Kill ALL vllm processes including orphan EngineCore children
pkill -9 -f vllm.entrypoints 2>/dev/null || true
pkill -9 -f qwen3_vllm_server 2>/dev/null || true
pkill -9 -f VLLM::EngineCore 2>/dev/null || true
pkill -9 -f "vllm.v1.engine.core" 2>/dev/null || true
sleep 3

mkdir -p logs

port=9089
for gpu in 0 1 2 3 4 5 6 7; do
    for i in 1 2 3; do
        CUDA_VISIBLE_DEVICES=$gpu python \
          -m vllm.entrypoints.openai.api_server \
          --model /path/to/model \
          --served-model-name Qwen3-0.6B \
          --port $port \
          --max-model-len 2048 \
          --max-num-seqs 32 \
          --gpu-memory-utilization 0.30 \
          > logs/gpu${gpu}_inst${i}_p${port}.log 2>&1 &
        echo "GPU $gpu instance $i on port $port (pid=$!)"
        sleep 5  # ← CRITICAL: stagger to avoid memory race
        port=$((port + 1))
    done
done

Pitfalls

1. Orphan EngineCore processes survive pkill

pkill -f vllm.entrypoints kills the API server parent but the VLLM::EngineCore child (separate multiprocessing worker) survives and keeps GPU memory locked. New instances then fail:

ValueError: Free memory on device cuda:0 (1.38/23.52 GiB) on startup is less
than desired GPU memory utilization (0.3, 7.05 GiB).

Fix: Explicitly kill EngineCore processes. Verify GPU is clear:

nvidia-smi --query-compute-apps=pid --format=csv,noheader | wc -l  # must be 0

2. Simultaneous startup causes memory race

When multiple instances start at the same time on the same GPU, they race during memory profiling. Results in:

AssertionError: Error in memory profiling. Initial free memory 15.48 GiB,
current free memory 15.76 GiB.

Or: No available memory for the cache blocks or KV cache too small errors.

Fix: sleep 5 between each instance launch. For 24 instances ≈ 2 min total.

3. Old log files cause confusion

After restart, old log files still contain errors from previous failed runs. Status-check scripts grep for errors and report false positives from stale files.

Fix: rm -f logs/*.log before restart, or check only files matching current PIDs.

4. SSH rate limiting on remote servers

GPU servers may reset SSH connections if you connect too rapidly (multiple SSH calls in quick succession). Manifests as kex_exchange_identification: read: Connection reset by peer.

Fix: Batch operations into a single SSH session via ssh host 'bash -s' << 'SCRIPT', or add sleep 15-60 between SSH commands.

5. nvidia-smi shows fewer processes than expected

If you launched 24 but nvidia-smi shows 8, the extra 16 silently failed. Always check logs after startup.

6. Nginx upstream has fewer servers than running instances (silent waste)

Symptom: "Why is performance worse with 24 instances than with 8?" — RPS is low, latency is high, but ps aux shows all 24 vLLM processes running and nvidia-smi shows all GPUs loaded.

Root cause: Nginx upstream block lists fewer servers than actual instances. Extra instances consume GPU memory but receive zero traffic.

Real-world case: 24 Qwen3-0.6B instances (9089-9112) running on 8 GPUs, but qwen3_backend upstream only had 8 servers (9089-9096). The other 16 instances sat idle — GPU memory wasted, no throughput contribution.

Diagnostic — compare running instances vs upstream servers:

# Count running vLLM instances
ps aux | grep vllm.entrypoints | grep -v grep | wc -l   # e.g., 24

# Extract ports from running processes
ps aux | grep vllm.entrypoints | grep -v grep | grep -oP '\-\-port \K\d+' | sort -n

# Count servers in nginx upstream
grep -c 'server 127.0.0.1:' /etc/nginx/sites-enabled/t2t  # e.g., 8

# Show actual upstream block
grep -A 30 'upstream' /etc/nginx/sites-enabled/t2t | head -30

If instance count > upstream server count, traffic only hits a subset. Fix: add all instance ports to the upstream block, then sudo kill -HUP $(pgrep -f "nginx: master").

7. Model name mismatch causes 404 errors

vLLM registers models by their full path (e.g., /d/models/Qwen/Qwen3-0.6B) unless you specify --served-model-name. If your nginx route strips the path prefix (e.g., /qwen3-6b/v1/chat/completions) but the test request uses a different model name, vLLM returns 404 because it doesn't recognize the model identifier.

Fix: Always add --served-model-name <name> and ensure:

  1. Nginx proxy_pass strips the path prefix (trailing /)
  2. Test request uses the exact --served-model-name value
  3. The model name in the request body matches --served-model-name

8. Nginx blocked_ip list blocks stress test traffic

Load testing from a specific IP may trigger Nginx's $blocked_ip check, causing all requests to return 403 before reaching vLLM. The test appears to "fail" but actually never hits the backend.

Fix:

  • Check if test IP is in the blocked list: sudo grep -A10 'geo.*blocked\|map.*blocked' /etc/nginx/nginx.conf
  • Temporarily whitelist test IP or disable blocking during tests
  • Monitor nginx error log: sudo tail -f /var/log/nginx/error.log
  • Verify with direct curl to vLLM port (bypass nginx): curl http://localhost:9089/health

9. API gateway rate limiting causes 403 under high concurrency

When testing through an API gateway (e.g., llmage at token.opencomputing.cn), the gateway itself may enforce per-key rate limits. Single curl requests succeed, but 200 concurrent requests all return 403 with "You don't have access to this model" — even though the model IS in /v1/models.

Previous stress test data shows the pattern:

  • 10 concurrent: 9% error rate
  • 50 concurrent: 19% error rate
  • 100 concurrent: 65% error rate
  • 200 concurrent: 99% error rate (effectively blocked)

This is not a vLLM issue — it's gateway-level throttling.

Fix:

  • Test directly against vLLM ports first to isolate: curl http://localhost:9089/v1/chat/completions
  • If gateway rate limits, either: (a) use multiple API keys with round-robin, (b) lower concurrency, (c) contact gateway admin to raise limits
  • Distinguish 403 sources: nginx blocked_ip returns plain 403; llmage gateway returns {"error": {"message": "You don't have access to this model", "code": "model_not_found"}}

10. Stress test RPS is misleading when all requests fail

If 100% of requests return 403/404, the "RPS" metric reflects nginx rejection speed, not vLLM throughput. Check error codes before trusting performance numbers.

Fix: Always inspect error distribution in test results:

if stats["ok"] == 0:
    print("WARNING: All requests failed — RPS is meaningless")

Nginx upstream config

upstream vllm_backend {
    least_conn;
    server 127.0.0.1:9089;
    server 127.0.0.1:9090;
    ...
    server 127.0.0.1:9112;  # 24 instances
}

Generate programmatically:

servers = '\n'.join(f'    server 127.0.0.1:{p};' for p in range(9089, 9089 + num_instances))

11. Qwen3 chat template jinja2 warnings flood logs

Qwen3-0.6B's chat template triggers harmless jinja2/sandbox.py:get value error messages on every request. These fill logs with noise but don't affect inference correctness.

Impact on diagnostics: grep -c 'POST' log still works for request counting, but grep 'error\|ERROR' log will match thousands of false positives. Use grep 'ERROR\|Traceback\|CUDA' log to filter real errors.

12. Gateway-layer health checks cause load imbalance across Nginx upstream

Symptom: Even though Nginx least_conn is configured and all 24 instances are healthy, one instance receives 20x more requests than others (e.g., 9089 gets 3840 requests, others get ~180).

Root cause: When an API gateway (llmage) sits between the client and Nginx, the gateway's health check probes can fail under load. The gateway may bypass Nginx entirely for some requests or implement its own upstream selection that doesn't use least_conn.

Diagnostic: Count requests per instance from backend logs:

for f in /share/run/qwen3/logs/gpu*_inst*_p*.log; do
    port=$(basename $f | grep -oP 'p\K[0-9]+')
    count=$(grep -c 'POST /v1/chat/completions' $f)
    echo "$port: $count"
done | sort -t: -k1 -n

If distribution is highly uneven (>5:1 ratio), the issue is in the gateway layer, not Nginx. See api-load-testing skill, Pitfall #13.

Post-startup verification

# 1. All instances running
ps aux | grep 'vllm.entrypoints' | grep -v grep | wc -l   # expect 24

# 2. GPU processes visible
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | wc -l

# 3. All ports listening
ss -tlnp | grep -c '9089\|909[0-9]\|910[0-9]\|911[0-2]'

# 4. Spot-check health (bypass nginx, hit vLLM directly)
curl -s http://localhost:9089/health

# 5. Verify model name registration
curl -s http://localhost:9089/v1/models | python3 -m json.tool | grep id

Stress testing pitfalls

When running load tests against a vLLM cluster behind nginx or API gateway:

  1. Bypass nginx for direct testing: Test vLLM directly first (curl http://localhost:9089/v1/models) to isolate whether failures are in nginx routing, API gateway, or vLLM itself.

  2. Check nginx blocked_ip before testing: If the test machine IP is in the blocked list, all requests return 403 before reaching vLLM. The stress test "succeeds" (completes) but measures nginx rejection speed, not inference throughput.

  3. API gateway rate limiting: Gateways like llmage may enforce per-key rate limits. Single requests succeed but high concurrency triggers 403 for all requests (see Pitfall #9 above).

  4. API key in stress test script: Ensure the stress test script has the correct API key, not a placeholder. If nginx doesn't enforce auth, this won't matter; if it does, all requests fail with 401/403.

  5. Verify error codes and distinguish 403 sources: Before trusting RPS numbers, check the error distribution:

    • Nginx blocked_ip: plain 403, no JSON body
    • llmage gateway rate limit: {"error": {"message": "You don't have access to this model", "code": "model_not_found"}}
    • If 100% of requests fail, the "RPS" is meaningless — it's just how fast the rejection layer drops traffic.

  6. Model name in request body: The model field in the POST body must exactly match the --served-model-name value. Mismatched names cause 404 errors from vLLM.