---
name: vllm-multi-instance-gpu
description: "Run multiple independent vLLM instances per GPU for small models. Covers memory budgeting, sequential startup, orphan process cleanup, and Nginx load balancing."
version: 1.0.0
metadata:
  hermes:
    tags: [vLLM, multi-instance, GPU sharing, inference, deployment, Nginx, load-balancing]
---

# vLLM Multi-Instance Per GPU Deployment

## When to use

- Small models (≤7B params) that don't need tensor parallelism
- Want to maximize throughput by running N independent instances per GPU
- Each instance serves its own request queue independently (no inter-instance communication)

## Architecture

```
GPU 0: [vLLM inst1 :9089] [vLLM inst2 :9090] [vLLM inst3 :9091]
GPU 1: [vLLM inst1 :9092] [vLLM inst2 :9093] [vLLM inst3 :9094]
...
         ↓ Nginx upstream (least_conn)
    Single entry point :10443
```

- Each instance gets `CUDA_VISIBLE_DEVICES=<gpu_id>` (single GPU)
- Each instance binds a unique port — **vLLM does NOT support SO_REUSEPORT**
- Nginx `upstream` block with `least_conn` load-balances across all instances

## Memory budget

For N instances per GPU:

```
gpu_memory_utilization_per_instance = 0.90 / N
```

Example (24GB GPU, 0.6B model, 3 instances):
- Per instance: 0.30 × 24GB = 7.2GB
- Model weights (0.6B bf16): ~1.2GB
- KV Cache: ~6GB (enough for max_model_len=2048, max_num_seqs=32)

## Startup script (sequential launch required)

```bash
#!/bin/bash
set -e

# Kill ALL vllm processes including orphan EngineCore children
pkill -9 -f vllm.entrypoints 2>/dev/null || true
pkill -9 -f qwen3_vllm_server 2>/dev/null || true
pkill -9 -f VLLM::EngineCore 2>/dev/null || true
pkill -9 -f "vllm.v1.engine.core" 2>/dev/null || true
sleep 3

mkdir -p logs

port=9089
for gpu in 0 1 2 3 4 5 6 7; do
    for i in 1 2 3; do
        CUDA_VISIBLE_DEVICES=$gpu python \
          -m vllm.entrypoints.openai.api_server \
          --model /path/to/model \
          --served-model-name Qwen3-0.6B \
          --port $port \
          --max-model-len 2048 \
          --max-num-seqs 32 \
          --gpu-memory-utilization 0.30 \
          > logs/gpu${gpu}_inst${i}_p${port}.log 2>&1 &
        echo "GPU $gpu instance $i on port $port (pid=$!)"
        sleep 5  # ← CRITICAL: stagger to avoid memory race
        port=$((port + 1))
    done
done
```

## Pitfalls

### 1. Orphan EngineCore processes survive pkill

`pkill -f vllm.entrypoints` kills the API server parent but the `VLLM::EngineCore` child (separate multiprocessing worker) survives and **keeps GPU memory locked**. New instances then fail:

```
ValueError: Free memory on device cuda:0 (1.38/23.52 GiB) on startup is less
than desired GPU memory utilization (0.3, 7.05 GiB).
```

**Fix**: Explicitly kill EngineCore processes. Verify GPU is clear:
```bash
nvidia-smi --query-compute-apps=pid --format=csv,noheader | wc -l  # must be 0
```

### 2. Simultaneous startup causes memory race

When multiple instances start at the same time on the same GPU, they race during memory profiling. Results in:

```
AssertionError: Error in memory profiling. Initial free memory 15.48 GiB,
current free memory 15.76 GiB.
```

Or: `No available memory for the cache blocks` or KV cache too small errors.

**Fix**: `sleep 5` between each instance launch. For 24 instances ≈ 2 min total.

### 3. Old log files cause confusion

After restart, old log files still contain errors from previous failed runs. Status-check scripts grep for errors and report false positives from stale files.

**Fix**: `rm -f logs/*.log` before restart, or check only files matching current PIDs.

### 4. SSH rate limiting on remote servers

GPU servers may reset SSH connections if you connect too rapidly (multiple SSH calls in quick succession). Manifests as `kex_exchange_identification: read: Connection reset by peer`.

**Fix**: Batch operations into a single SSH session via `ssh host 'bash -s' << 'SCRIPT'`, or add `sleep 15-60` between SSH commands.

### 5. nvidia-smi shows fewer processes than expected

If you launched 24 but nvidia-smi shows 8, the extra 16 silently failed. Always check logs after startup.

### 6. Nginx upstream has fewer servers than running instances (silent waste)

**Symptom**: "Why is performance worse with 24 instances than with 8?" — RPS is low, latency is high, but `ps aux` shows all 24 vLLM processes running and `nvidia-smi` shows all GPUs loaded.

**Root cause**: Nginx upstream block lists fewer servers than actual instances. Extra instances consume GPU memory but receive zero traffic.

**Real-world case**: 24 Qwen3-0.6B instances (9089-9112) running on 8 GPUs, but `qwen3_backend` upstream only had 8 servers (9089-9096). The other 16 instances sat idle — GPU memory wasted, no throughput contribution.

**Diagnostic** — compare running instances vs upstream servers:
```bash
# Count running vLLM instances
ps aux | grep vllm.entrypoints | grep -v grep | wc -l   # e.g., 24

# Extract ports from running processes
ps aux | grep vllm.entrypoints | grep -v grep | grep -oP '\-\-port \K\d+' | sort -n

# Count servers in nginx upstream
grep -c 'server 127.0.0.1:' /etc/nginx/sites-enabled/t2t  # e.g., 8

# Show actual upstream block
grep -A 30 'upstream' /etc/nginx/sites-enabled/t2t | head -30
```

If instance count > upstream server count, traffic only hits a subset. Fix: add all instance ports to the upstream block, then `sudo kill -HUP $(pgrep -f "nginx: master")`.

### 7. Model name mismatch causes 404 errors

vLLM registers models by their full path (e.g., `/d/models/Qwen/Qwen3-0.6B`) unless you specify `--served-model-name`. If your nginx route strips the path prefix (e.g., `/qwen3-6b/v1/chat/completions`) but the test request uses a different model name, vLLM returns 404 because it doesn't recognize the model identifier.

**Fix**: Always add `--served-model-name <name>` and ensure:
1. Nginx `proxy_pass` strips the path prefix (trailing `/`)
2. Test request uses the exact `--served-model-name` value
3. The model name in the request body matches `--served-model-name`

### 8. Nginx blocked_ip list blocks stress test traffic

Load testing from a specific IP may trigger Nginx's `$blocked_ip` check, causing all requests to return 403 before reaching vLLM. The test appears to "fail" but actually never hits the backend.

**Fix**: 
- Check if test IP is in the blocked list: `sudo grep -A10 'geo.*blocked\|map.*blocked' /etc/nginx/nginx.conf`
- Temporarily whitelist test IP or disable blocking during tests
- Monitor nginx error log: `sudo tail -f /var/log/nginx/error.log`
- Verify with direct curl to vLLM port (bypass nginx): `curl http://localhost:9089/health`

### 9. API gateway rate limiting causes 403 under high concurrency

When testing through an API gateway (e.g., llmage at `token.opencomputing.cn`), the gateway itself may enforce per-key rate limits. Single curl requests succeed, but 200 concurrent requests all return 403 with `"You don't have access to this model"` — even though the model IS in `/v1/models`.

Previous stress test data shows the pattern:
- 10 concurrent: 9% error rate
- 50 concurrent: 19% error rate  
- 100 concurrent: 65% error rate
- 200 concurrent: 99% error rate (effectively blocked)

This is **not** a vLLM issue — it's gateway-level throttling.

**Fix**: 
- Test directly against vLLM ports first to isolate: `curl http://localhost:9089/v1/chat/completions`
- If gateway rate limits, either: (a) use multiple API keys with round-robin, (b) lower concurrency, (c) contact gateway admin to raise limits
- Distinguish 403 sources: nginx `blocked_ip` returns plain 403; llmage gateway returns `{"error": {"message": "You don't have access to this model", "code": "model_not_found"}}`

### 10. Stress test RPS is misleading when all requests fail

If 100% of requests return 403/404, the "RPS" metric reflects nginx rejection speed, not vLLM throughput. Check error codes before trusting performance numbers.

**Fix**: Always inspect error distribution in test results:
```python
if stats["ok"] == 0:
    print("WARNING: All requests failed — RPS is meaningless")
```

## Nginx upstream config

```nginx
upstream vllm_backend {
    least_conn;
    server 127.0.0.1:9089;
    server 127.0.0.1:9090;
    ...
    server 127.0.0.1:9112;  # 24 instances
}
```

Generate programmatically:
```python
servers = '\n'.join(f'    server 127.0.0.1:{p};' for p in range(9089, 9089 + num_instances))
```

### 11. Qwen3 chat template jinja2 warnings flood logs

Qwen3-0.6B's chat template triggers harmless `jinja2/sandbox.py:get value error` messages on every request. These fill logs with noise but don't affect inference correctness.

**Impact on diagnostics**: `grep -c 'POST' log` still works for request counting, but `grep 'error\|ERROR' log` will match thousands of false positives. Use `grep 'ERROR\|Traceback\|CUDA' log` to filter real errors.

### 12. Gateway-layer health checks cause load imbalance across Nginx upstream

**Symptom**: Even though Nginx `least_conn` is configured and all 24 instances are healthy, one instance receives 20x more requests than others (e.g., 9089 gets 3840 requests, others get ~180).

**Root cause**: When an API gateway (llmage) sits between the client and Nginx, the gateway's health check probes can fail under load. The gateway may bypass Nginx entirely for some requests or implement its own upstream selection that doesn't use least_conn.

**Diagnostic**: Count requests per instance from backend logs:
```bash
for f in /share/run/qwen3/logs/gpu*_inst*_p*.log; do
    port=$(basename $f | grep -oP 'p\K[0-9]+')
    count=$(grep -c 'POST /v1/chat/completions' $f)
    echo "$port: $count"
done | sort -t: -k1 -n
```

If distribution is highly uneven (>5:1 ratio), the issue is in the gateway layer, not Nginx. See `api-load-testing` skill, Pitfall #13.

## Post-startup verification

```bash
# 1. All instances running
ps aux | grep 'vllm.entrypoints' | grep -v grep | wc -l   # expect 24

# 2. GPU processes visible
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | wc -l

# 3. All ports listening
ss -tlnp | grep -c '9089\|909[0-9]\|910[0-9]\|911[0-2]'

# 4. Spot-check health (bypass nginx, hit vLLM directly)
curl -s http://localhost:9089/health

# 5. Verify model name registration
curl -s http://localhost:9089/v1/models | python3 -m json.tool | grep id
```

## Stress testing pitfalls

When running load tests against a vLLM cluster behind nginx or API gateway:

1. **Bypass nginx for direct testing**: Test vLLM directly first (`curl http://localhost:9089/v1/models`) to isolate whether failures are in nginx routing, API gateway, or vLLM itself.

2. **Check nginx blocked_ip before testing**: If the test machine IP is in the blocked list, all requests return 403 before reaching vLLM. The stress test "succeeds" (completes) but measures nginx rejection speed, not inference throughput.

3. **API gateway rate limiting**: Gateways like llmage may enforce per-key rate limits. Single requests succeed but high concurrency triggers 403 for all requests (see Pitfall #9 above).

4. **API key in stress test script**: Ensure the stress test script has the correct API key, not a placeholder. If nginx doesn't enforce auth, this won't matter; if it does, all requests fail with 401/403.

5. **Verify error codes and distinguish 403 sources**: Before trusting RPS numbers, check the error distribution:
   - Nginx `blocked_ip`: plain 403, no JSON body
   - llmage gateway rate limit: `{"error": {"message": "You don't have access to this model", "code": "model_not_found"}}`
   - If 100% of requests fail, the "RPS" is meaningless — it's just how fast the rejection layer drops traffic.

6. **Model name in request body**: The `model` field in the POST body must exactly match the `--served-model-name` value. Mismatched names cause 404 errors from vLLM.