pipeline_core/skills_library/all/async-db-connection-pool-reliability/SKILL.md

name	description	tags	version
async-db-connection-pool-reliability	Diagnose and fix async database connection pool issues — TCP transport closure, connection health checks, pool maintenance patterns.	database async connection-pool aiomysql uvloop reliability	1.0.0

Async Database Connection Pool Reliability

When to Use

• RuntimeError about TCPTransport closed or handler is closed when using async database connections
• Database queries fail intermittently after idle periods
• Connection pool holds stale/dead connections
• aiomysql/asyncpg/aiosqlite connection timeout issues

Core Problem Pattern

Symptom: RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False>; the handler is closed

Root Cause: Database server closes idle connections (TCP timeout), but client connection pool still holds references to dead connection objects. When code tries to use them, the underlying transport is already closed.

Common Triggers:

• MySQL wait_timeout (default 8h) kills idle connections
• Network firewall/NAT timeout closes long-lived TCP connections
• Connection pool doesn't validate connections before use

Fix Strategy

1. Connection Health Check Before Yield

Add _check_alive() method that verifies connection state before yielding from pool context:

async def _check_alive(self, sor):
    """Lightweight check: can we still talk to the DB?"""
    try:
        conn = sor.conn
        if conn is None:
            return False
        
        # aiomysql: check transport state
        writer = getattr(conn, '_writer', None)
        if writer is not None:
            transport = getattr(writer, 'transport', None)
            if transport is not None and transport.is_closing():
                return False
        
        # Try lightweight ping
        await sor.enter()
        await sor.execute(sor.test_sqlstr, {})
        await sor.exit()
        return True
    except Exception:
        try:
            await sor.exit()
        except:
            pass
        return False

2. Discard Dead Connections

Add helper to safely remove broken connections from pool:

def _discard_sqlor(self, entry):
    """Remove broken connection from pool and clean up."""
    if entry in self.sqlors:
        self.sqlors.remove(entry)
    asyncio.ensure_future(self._del_sqlor(entry.sqlor))

3. Check in Context Manager

In pool.context(), verify connections before yielding:

@asynccontextmanager
async def context(self):
    self._cleanup_idle()
    async with self.sema:
        yielded_sqlor = None
        # Try to find a healthy idle connection
        sqlors = [s for s in self.sqlors if not s.used]
        for s in sqlors:
            ok = await self._check_alive(s.sqlor)
            if ok:
                yielded_sqlor = s
                break
            else:
                debug(f'SqlorPool.context: discarding dead connection')
                self._discard_sqlor(s)
        
        if not yielded_sqlor:
            yielded_sqlor = await self._new_sqlor()
        yielded_sqlor.used = True
        yielded_sqlor.use_at = time.time()
        try:
            yield yielded_sqlor.sqlor
        except (RuntimeError, OSError) as e:
            err_msg = str(e)
            if 'closed' in err_msg or 'handler is closed' in err_msg:
                # Connection died during use — discard it
                self._discard_sqlor(yielded_sqlor)
                yielded_sqlor = None
            raise
        finally:
            if yielded_sqlor is not None:
                yielded_sqlor.used = False
                yielded_sqlor.use_at = time.time()

Pitfalls

_check_alive Snowball Under High Concurrency (CRITICAL)

Symptom: Under 200 concurrent requests, 57% of all log output is discarding dead connection. Each request discards ~12 dead connections before finding a live one. P50 latency 4.6s for a task that should take <200ms.

Real-world data (token platform, 200 concurrent, 3 minutes):

51,813  "discarding dead connection" log lines
90,777  total log lines in same period
4,314   successful requests forwarded to backend
12,948  params_kw entries (requests received)
→ 51813 / 4314 = ~12 dead connections discarded per request

Why it snowballs: _check_alive() does a full DB round-trip (SELECT 1) on every idle connection. When connections are stale (e.g., after a period of inactivity or a backend incident), ALL idle connections are dead. Under 200 concurrency:

• 8 app processes × 200 concurrent requests = 1600 concurrent health checks
• Each check holds the semaphore, blocking other requests
• Dead connection → discard → create new → but new connections also get checked
• The pool becomes a serial bottleneck: requests queue behind health checks

Fix options (in order of preference):

1. Idle-time threshold: Only run _check_alive on connections idle >30s

   for s in sqlors:
       if not s.used:
           if time.time() - s.use_at < 30:
               yielded_sqlor = s  # Fresh enough, skip check
               break
           elif await self._check_alive(s.sqlor):
               yielded_sqlor = s
               break
           else:
               self._discard_sqlor(s)
   

2. TCP-level probe: Check transport.is_closing() without DB round-trip
3. Larger pool + no validation: Remove _check_alive entirely, catch TCPTransport closed in the except block and discard+retry
4. Proactive reaper: Background task closes idle connections after timeout, so pool always has live connections

Key insight: The _check_alive pattern is correct for low-concurrency use (1-10 concurrent). At 100+ concurrent with stale connections, it becomes the bottleneck. The fix must be at the connection pool layer, not the application layer.

MVCC Snapshot Pinning: Uncommitted Read-Only Transaction on a Pooled Connection (CRITICAL)

Symptom: A long-lived background loop (accounting poller, queue worker) logs "got 0 records" forever even though matching rows exist. A fresh standalone script running the identical SQL returns rows. The loop may have been blind for weeks — check the newest row in its side-effect tables for the real failure date.

Decisive diagnosis:

SELECT trx_state, trx_started,
       TIMESTAMPDIFF(SECOND, trx_started, NOW()) age_s,
       trx_mysql_thread_id, trx_query
FROM information_schema.INNODB_TRX ORDER BY trx_started;

RUNNING + age ≈ process uptime + trx_query = NULL (connection idle!) = an open transaction parked on a pooled connection. Under REPEATABLE READ its snapshot is pinned at the first read — every later SELECT on that connection sees the same stale snapshot.

Root cause chain: aiomysql defaults autocommit=False → the loop's first SELECT implicitly opens a transaction → a context manager that only commits on writes (if sqlor.dataChanged: commit()) never ends the transaction on read-only exit → the connection returns to the pool still open → the next borrower inherits the pinned snapshot. Rows inserted after process start stay invisible forever. No exception is raised anywhere — pure silent staleness. "Works standalone, returns 0 in-process" is the signature; check transaction state before chasing the SQL.

Fix: end any leftover transaction when a connection is checked out — mysqlor.enter() commits before creating the cursor:

async def enter(self):
    await self.conn.commit()   # ends leftover trx, refreshes MVCC snapshot
    self.cur = await self.conn.cursor()

(sqlor commit fab420c). Alternatives: autocommit=True at connect time, or rollback on read-only context exit.

Verification: after deploy+restart — INNODB_TRX shows no long-running trx from that process; the loop logs non-zero counts; previously invisible rows get processed.

Full evidence transcript + reproduction recipe: references/mvcc-snapshot-pinning.md.

Don't Retry Inside @asynccontextmanager Generators

Wrong: Trying to retry inside the generator body:

@asynccontextmanager
async def sqlorContext(self, name):
    for attempt in range(max_retries):
        async with pool.context() as sqlor:
            yield sqlor  # ❌ Can only yield once!
            if error and attempt < max_retries - 1:
                continue  # RuntimeError: generator already executing

Right: Do health checks BEFORE yielding, not retry after:

@asynccontextmanager
async def context(self):
    # Check health here, before yield
    for s in sqlors:
        if await self._check_alive(s.sqlor):
            yielded_sqlor = s
            break
        else:
            self._discard_sqlor(s)
    
    # Now yield the verified connection
    yield yielded_sqlor.sqlor

Transport Detection Varies by Driver

• aiomysql: conn._writer.transport.is_closing()
• asyncpg: conn.is_closed() or check _protocol.is_connected()
• aiosqlite: Connection object has _closed attribute

Idle Timeout Cleanup

Also implement _cleanup_idle() to proactively close connections unused for >N minutes:

def _cleanup_idle(self):
    now = time.time()
    to_remove = []
    for s in self.sqlors:
        if not s.used and (now - s.use_at) > self.IDLE_TIMEOUT:
            to_remove.append(s)
    for s in to_remove:
        self.sqlors = [x for x in self.sqlors if x != s]
        asyncio.ensure_future(self._del_sqlor(s.sqlor))

Verification

After fix, monitor logs for:

• Reduced TCPTransport closed errors
• Connection pool creating new connections after idle periods
• Successful query execution after long gaps

Related Skills

• systematic-debugging — for structured root cause analysis
• mlops/serving-llms-vllm — if dealing with high-throughput async DB patterns