---
name: async-db-connection-pool-reliability
description: Diagnose and fix async database connection pool issues — TCP transport closure, connection health checks, pool maintenance patterns.
tags: [database, async, connection-pool, aiomysql, uvloop, reliability]
version: 1.0.0
---

# Async Database Connection Pool Reliability

## When to Use
- RuntimeError about `TCPTransport closed` or `handler is closed` when using async database connections
- Database queries fail intermittently after idle periods
- Connection pool holds stale/dead connections
- aiomysql/asyncpg/aiosqlite connection timeout issues

## Core Problem Pattern

**Symptom**: `RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False>; the handler is closed`

**Root Cause**: Database server closes idle connections (TCP timeout), but client connection pool still holds references to dead connection objects. When code tries to use them, the underlying transport is already closed.

**Common Triggers**:
- MySQL `wait_timeout` (default 8h) kills idle connections
- Network firewall/NAT timeout closes long-lived TCP connections
- Connection pool doesn't validate connections before use

## Fix Strategy

### 1. Connection Health Check Before Yield
Add `_check_alive()` method that verifies connection state before yielding from pool context:

```python
async def _check_alive(self, sor):
    """Lightweight check: can we still talk to the DB?"""
    try:
        conn = sor.conn
        if conn is None:
            return False
        
        # aiomysql: check transport state
        writer = getattr(conn, '_writer', None)
        if writer is not None:
            transport = getattr(writer, 'transport', None)
            if transport is not None and transport.is_closing():
                return False
        
        # Try lightweight ping
        await sor.enter()
        await sor.execute(sor.test_sqlstr, {})
        await sor.exit()
        return True
    except Exception:
        try:
            await sor.exit()
        except:
            pass
        return False
```

### 2. Discard Dead Connections
Add helper to safely remove broken connections from pool:

```python
def _discard_sqlor(self, entry):
    """Remove broken connection from pool and clean up."""
    if entry in self.sqlors:
        self.sqlors.remove(entry)
    asyncio.ensure_future(self._del_sqlor(entry.sqlor))
```

### 3. Check in Context Manager
In `pool.context()`, verify connections before yielding:

```python
@asynccontextmanager
async def context(self):
    self._cleanup_idle()
    async with self.sema:
        yielded_sqlor = None
        # Try to find a healthy idle connection
        sqlors = [s for s in self.sqlors if not s.used]
        for s in sqlors:
            ok = await self._check_alive(s.sqlor)
            if ok:
                yielded_sqlor = s
                break
            else:
                debug(f'SqlorPool.context: discarding dead connection')
                self._discard_sqlor(s)
        
        if not yielded_sqlor:
            yielded_sqlor = await self._new_sqlor()
        yielded_sqlor.used = True
        yielded_sqlor.use_at = time.time()
        try:
            yield yielded_sqlor.sqlor
        except (RuntimeError, OSError) as e:
            err_msg = str(e)
            if 'closed' in err_msg or 'handler is closed' in err_msg:
                # Connection died during use — discard it
                self._discard_sqlor(yielded_sqlor)
                yielded_sqlor = None
            raise
        finally:
            if yielded_sqlor is not None:
                yielded_sqlor.used = False
                yielded_sqlor.use_at = time.time()
```

## Pitfalls

### _check_alive Snowball Under High Concurrency (CRITICAL)

**Symptom**: Under 200 concurrent requests, 57% of all log output is `discarding dead connection`. Each request discards ~12 dead connections before finding a live one. P50 latency 4.6s for a task that should take <200ms.

**Real-world data** (token platform, 200 concurrent, 3 minutes):
```
51,813  "discarding dead connection" log lines
90,777  total log lines in same period
4,314   successful requests forwarded to backend
12,948  params_kw entries (requests received)
→ 51813 / 4314 = ~12 dead connections discarded per request
```

**Why it snowballs**: `_check_alive()` does a full DB round-trip (SELECT 1) on every idle connection. When connections are stale (e.g., after a period of inactivity or a backend incident), ALL idle connections are dead. Under 200 concurrency:
- 8 app processes × 200 concurrent requests = 1600 concurrent health checks
- Each check holds the semaphore, blocking other requests
- Dead connection → discard → create new → but new connections also get checked
- The pool becomes a serial bottleneck: requests queue behind health checks

**Fix options** (in order of preference):
1. **Idle-time threshold**: Only run `_check_alive` on connections idle >30s
   ```python
   for s in sqlors:
       if not s.used:
           if time.time() - s.use_at < 30:
               yielded_sqlor = s  # Fresh enough, skip check
               break
           elif await self._check_alive(s.sqlor):
               yielded_sqlor = s
               break
           else:
               self._discard_sqlor(s)
   ```
2. **TCP-level probe**: Check `transport.is_closing()` without DB round-trip
3. **Larger pool + no validation**: Remove `_check_alive` entirely, catch `TCPTransport closed` in the except block and discard+retry
4. **Proactive reaper**: Background task closes idle connections after timeout, so pool always has live connections

**Key insight**: The `_check_alive` pattern is correct for low-concurrency use (1-10 concurrent). At 100+ concurrent with stale connections, it becomes the bottleneck. The fix must be at the connection pool layer, not the application layer.

### MVCC Snapshot Pinning: Uncommitted Read-Only Transaction on a Pooled Connection (CRITICAL)

**Symptom**: A long-lived background loop (accounting poller, queue worker) logs "got 0 records" forever even though matching rows exist. A fresh standalone script running the identical SQL returns rows. The loop may have been blind for weeks — check the newest row in its side-effect tables for the real failure date.

**Decisive diagnosis**:
```sql
SELECT trx_state, trx_started,
       TIMESTAMPDIFF(SECOND, trx_started, NOW()) age_s,
       trx_mysql_thread_id, trx_query
FROM information_schema.INNODB_TRX ORDER BY trx_started;
```
`RUNNING` + age ≈ process uptime + `trx_query = NULL` (connection idle!) = an open transaction parked on a pooled connection. Under REPEATABLE READ its snapshot is pinned at the first read — every later SELECT on that connection sees the same stale snapshot.

**Root cause chain**: aiomysql defaults `autocommit=False` → the loop's first SELECT implicitly opens a transaction → a context manager that only commits on writes (`if sqlor.dataChanged: commit()`) never ends the transaction on read-only exit → the connection returns to the pool still open → the next borrower inherits the pinned snapshot. Rows inserted after process start stay invisible forever. No exception is raised anywhere — pure silent staleness. "Works standalone, returns 0 in-process" is the signature; check transaction state before chasing the SQL.

**Fix**: end any leftover transaction when a connection is checked out — `mysqlor.enter()` commits before creating the cursor:
```python
async def enter(self):
    await self.conn.commit()   # ends leftover trx, refreshes MVCC snapshot
    self.cur = await self.conn.cursor()
```
(sqlor commit fab420c). Alternatives: `autocommit=True` at connect time, or rollback on read-only context exit.

**Verification**: after deploy+restart — INNODB_TRX shows no long-running trx from that process; the loop logs non-zero counts; previously invisible rows get processed.

Full evidence transcript + reproduction recipe: `references/mvcc-snapshot-pinning.md`.

### Don't Retry Inside @asynccontextmanager Generators
**Wrong**: Trying to retry inside the generator body:
```python
@asynccontextmanager
async def sqlorContext(self, name):
    for attempt in range(max_retries):
        async with pool.context() as sqlor:
            yield sqlor  # ❌ Can only yield once!
            if error and attempt < max_retries - 1:
                continue  # RuntimeError: generator already executing
```

**Right**: Do health checks BEFORE yielding, not retry after:
```python
@asynccontextmanager
async def context(self):
    # Check health here, before yield
    for s in sqlors:
        if await self._check_alive(s.sqlor):
            yielded_sqlor = s
            break
        else:
            self._discard_sqlor(s)
    
    # Now yield the verified connection
    yield yielded_sqlor.sqlor
```

### Transport Detection Varies by Driver
- **aiomysql**: `conn._writer.transport.is_closing()`
- **asyncpg**: `conn.is_closed()` or check `_protocol.is_connected()`
- **aiosqlite**: Connection object has `_closed` attribute

### Idle Timeout Cleanup
Also implement `_cleanup_idle()` to proactively close connections unused for >N minutes:
```python
def _cleanup_idle(self):
    now = time.time()
    to_remove = []
    for s in self.sqlors:
        if not s.used and (now - s.use_at) > self.IDLE_TIMEOUT:
            to_remove.append(s)
    for s in to_remove:
        self.sqlors = [x for x in self.sqlors if x != s]
        asyncio.ensure_future(self._del_sqlor(s.sqlor))
```

## Verification

After fix, monitor logs for:
- Reduced `TCPTransport closed` errors
- Connection pool creating new connections after idle periods
- Successful query execution after long gaps

## Related Skills
- `systematic-debugging` — for structured root cause analysis
- `mlops/serving-llms-vllm` — if dealing with high-throughput async DB patterns