A Go worker pool's db.QueryContext calls return context deadline exceeded while the database logs no slow queries — how do you prove the connection pool is the bottleneck?

seniorshould knowfreq 46%

answer

the timeout never reached the server
the acquire and the query share one deadline
two cumulative counters, so diff them
mean wait equals duration over count
compare worker count with the open limit

basics

~20 s

Sample db.Stats over time. Rising WaitCount with WaitDuration divided by WaitCount approaching the deadline, InUse pinned at the open limit and Idle at zero, proves callers are timing out while queueing for a connection rather than while running a query.

solid answer

~50 s

The signature is that the deadline is consumed before the statement is sent: when the pool is at SetMaxOpenConns the acquire blocks, and if the context expires there the call returns context.DeadlineExceeded having sent nothing, which is exactly why the database has no record of a slow query. To prove it, sample db.Stats() on a timer and diff successive samples: WaitCount climbing means callers are queueing, and the delta in WaitDuration divided by the delta in WaitCount gives the mean wait per acquire — when that approaches your per-call deadline, the pool is the bottleneck. InUse sitting at MaxOpenConnections with Idle at zero confirms it. Then fix the mismatch: worker count above the open limit is the usual cause, so bound the workers, raise the limit if your connection budget allows, or shorten how long each job holds a connection.

The signature is that the deadline is consumed *before* the statement is sent: when the pool is at `SetMaxOpenConns` the acquire blocks, and if the context expires there the call returns `context.DeadlineExceeded` having sent nothing, which is exactly why the database has no record of a slow query. To prove it, sample `db.Stats()` on a timer and diff successive samples: `WaitCount` climbing means callers are queueing, and the delta in `WaitDuration` divided by the delta in `WaitCount` gives the mean wait per acquire — when that approaches your per-call deadline, the pool is the bottleneck. `InUse` sitting at `MaxOpenConnections` with `Idle` at zero confirms it. Then fix the mismatch: worker count above the open limit is the usual cause, so bound the workers, raise the limit if your connection budget allows, or shorten how long each job holds a connection.

code

go · 16 lines

jobs := make(chan int64)
var wg sync.WaitGroup
for w := 0; w < 200; w++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		for id := range jobs {
			ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
			_, err := db.ExecContext(ctx, updateStmt, id)
			cancel()
			if err != nil {
				log.Printf("job %d: %v", id, err)
			}
		}
	}()
}

go deeper

for a junior

Understand that one context deadline covers both getting a connection and running the statement, so a query can time out without the database ever receiving it.

for a middle

Be able to name the db.Stats fields that expose queueing and explain that WaitCount and WaitDuration are cumulative totals you must sample and difference to get a rate and a mean.

for a senior

Walk the whole diagnosis: symptom, the client-side queue hypothesis, the stats that confirm it, the worker-to-connection arithmetic, and a fix that bounds producers rather than simply enlarging the pool.

for a principal

Insist these signals exist before the incident: pool saturation and mean acquire wait should be standard dashboards and alerts, so an exhaustion argument is settled with data instead of by raising limits hopefully.

The symptom and why it is confusing

A batch worker pool — some number of goroutines reading jobs off a channel, each doing one database call per job — starts returning context deadline exceeded under a burst. The obvious hypothesis is a slow database, but the database reports nothing: no slow-query log entries, low CPU, and a query count well below what the workers should be generating. The two observations look contradictory until you notice where the time went.

Where the deadline actually goes

Every db.QueryContext, db.ExecContext or db.BeginTx call does two things under one context: it acquires a pooled connection, then it executes. When the pool already has SetMaxOpenConns connections open and all of them are in use, the acquire blocks in the pool's wait queue. If the context's deadline fires during that wait, the call returns the context's error and nothing was ever sent to the server. The database cannot log a slow query it never received, and your query count is low precisely because the work never got there.

This is the single most useful fact about pool exhaustion in Go: the timeout you see is charged to the query call, but it was spent queueing.

Proving it with db.Stats

db.Stats() returns a sql.DBStats snapshot. The fields that matter here:

MaxOpenConnections — the cap you configured.
OpenConnections, InUse, Idle — instantaneous gauges.
WaitCount — cumulative number of acquires that had to wait.
WaitDuration — cumulative time spent waiting across all acquires.

Because WaitCount and WaitDuration are cumulative totals since the pool was created, a single reading tells you almost nothing; you must sample on a timer and diff. Two derived numbers do the work:

The rate of WaitCount — how many acquires per second are queueing at all. Zero means the pool is not the constraint.
delta(WaitDuration) / delta(WaitCount) — the mean wait per queued acquire. Compare it against your per-call deadline. When the mean wait is within the same order of magnitude as the deadline, a meaningful share of calls will be timing out purely in the queue.

Alongside those, InUse == MaxOpenConnections with Idle == 0 sustained across samples says the pool is fully saturated rather than momentarily busy. Export all of these; they cost a mutex-protected snapshot and are the only view you get of the queue.

Confirming from the other direction

Two cheap cross-checks:

Arithmetic. Count your workers and compare with the open limit. Two hundred goroutines sharing twenty-five connections means at least 175 of them are waiting at any instant during a burst; the mean wait is roughly the job's database time multiplied by the ratio of workers to connections. If that product exceeds your deadline, the timeouts are arithmetic, not a mystery.
Goroutine states. A goroutine profile or a stack dump during the incident shows the waiters parked inside the pool's connection acquisition, not inside the driver's network read. That distinguishes "waiting for a connection" from "waiting for the database".

What to change

Once the pool is confirmed as the bottleneck, the fixes are, roughly in order of preference:

Bound the producers to match the pool. If 200 workers share 25 connections, the extra 175 goroutines add queueing and timeouts but no throughput. Sizing the worker count to the connection budget makes the system behave predictably, and the excess work waits in the jobs channel — where it is visible and cheap — instead of in the pool's queue.
Reduce how long each job holds a connection. A connection is held for the whole operation, so anything expensive done while holding it lengthens the queue for everyone.
Raise the open limit — but only within your connection budget. This is the tempting first move and the one with a cost outside your service: your cap multiplies by your replica count against a database shared with others.
Give the acquire its own budget. A per-call deadline is what makes exhaustion fail fast instead of piling up; without one, callers queue indefinitely and the backlog grows unbounded.

What not to conclude

A timeout that is spent in the queue is not evidence that queries are slow, and raising the per-call deadline does not fix it — it converts timeouts into a longer queue and a bigger backlog. Equally, if WaitCount is flat while calls still time out, the pool is not the bottleneck and the time is going to the database or the network; the same db.Stats sampling is what rules that out.

## The symptom and why it is confusing A batch worker pool — some number of goroutines reading jobs off a channel, each doing one database call per job — starts returning `context deadline exceeded` under a burst. The obvious hypothesis is a slow database, but the database reports nothing: no slow-query log entries, low CPU, and a query count well below what the workers should be generating. The two observations look contradictory until you notice where the time went. ## Where the deadline actually goes Every `db.QueryContext`, `db.ExecContext` or `db.BeginTx` call does two things under one context: it **acquires** a pooled connection, then it **executes**. When the pool already has `SetMaxOpenConns` connections open and all of them are in use, the acquire blocks in the pool's wait queue. If the context's deadline fires during that wait, the call returns the context's error and **nothing was ever sent to the server**. The database cannot log a slow query it never received, and your query count is low precisely because the work never got there. This is the single most useful fact about pool exhaustion in Go: the timeout you see is charged to the query call, but it was spent queueing. ## Proving it with db.Stats `db.Stats()` returns a `sql.DBStats` snapshot. The fields that matter here: - `MaxOpenConnections` — the cap you configured. - `OpenConnections`, `InUse`, `Idle` — instantaneous gauges. - `WaitCount` — cumulative number of acquires that had to wait. - `WaitDuration` — cumulative time spent waiting across all acquires. Because `WaitCount` and `WaitDuration` are **cumulative totals since the pool was created**, a single reading tells you almost nothing; you must sample on a timer and diff. Two derived numbers do the work: 1. The rate of `WaitCount` — how many acquires per second are queueing at all. Zero means the pool is not the constraint. 2. `delta(WaitDuration) / delta(WaitCount)` — the mean wait per queued acquire. Compare it against your per-call deadline. When the mean wait is within the same order of magnitude as the deadline, a meaningful share of calls will be timing out purely in the queue. Alongside those, `InUse == MaxOpenConnections` with `Idle == 0` sustained across samples says the pool is fully saturated rather than momentarily busy. Export all of these; they cost a mutex-protected snapshot and are the only view you get of the queue. ## Confirming from the other direction Two cheap cross-checks: - **Arithmetic.** Count your workers and compare with the open limit. Two hundred goroutines sharing twenty-five connections means at least 175 of them are waiting at any instant during a burst; the mean wait is roughly the job's database time multiplied by the ratio of workers to connections. If that product exceeds your deadline, the timeouts are arithmetic, not a mystery. - **Goroutine states.** A goroutine profile or a stack dump during the incident shows the waiters parked inside the pool's connection acquisition, not inside the driver's network read. That distinguishes "waiting for a connection" from "waiting for the database". ## What to change Once the pool is confirmed as the bottleneck, the fixes are, roughly in order of preference: 1. **Bound the producers to match the pool.** If 200 workers share 25 connections, the extra 175 goroutines add queueing and timeouts but no throughput. Sizing the worker count to the connection budget makes the system behave predictably, and the excess work waits in the jobs channel — where it is visible and cheap — instead of in the pool's queue. 2. **Reduce how long each job holds a connection.** A connection is held for the whole operation, so anything expensive done while holding it lengthens the queue for everyone. 3. **Raise the open limit — but only within your connection budget.** This is the tempting first move and the one with a cost outside your service: your cap multiplies by your replica count against a database shared with others. 4. **Give the acquire its own budget.** A per-call deadline is what makes exhaustion fail fast instead of piling up; without one, callers queue indefinitely and the backlog grows unbounded. ## What not to conclude A timeout that is spent in the queue is not evidence that queries are slow, and raising the per-call deadline does not fix it — it converts timeouts into a longer queue and a bigger backlog. Equally, if `WaitCount` is flat while calls still time out, the pool is *not* the bottleneck and the time is going to the database or the network; the same `db.Stats` sampling is what rules that out.

Why does raising the per-call deadline not fix this?
It changes where the pain shows up, not the throughput. Throughput is bounded by the open connection limit divided by the average time each job holds a connection; a longer deadline just lets more goroutines queue for longer, growing the backlog and memory while the same number of jobs complete per second. The timeouts return as soon as the burst is slightly larger.
How would you distinguish pool starvation from a genuinely slow database using the same signals?
Look at whether `WaitCount` is moving. If it is flat while calls still time out, callers are getting connections immediately and the time is going to the database or the network. If it is climbing and the mean wait approaches the deadline, the queue is the problem. Both cases look identical from the caller's error alone.
Is WaitCount a gauge of how many callers are waiting right now?
No. `WaitCount` is a cumulative total of acquires that have ever had to wait, and `WaitDuration` is the cumulative time they spent waiting. Neither resets when you read them, so a single value is meaningless in isolation; you sample periodically and use the differences between samples.
Why does the database's own query log show nothing for the failed calls?
Because the deadline elapsed during the acquire, before any statement was written to a connection. The server never saw those calls at all. That absence is itself diagnostic: errors on the client with no matching activity on the server points at the client-side queue rather than at execution.

saying these in an interview costs you the question

Concludes the database is slow because queries time out
Raises the per-call deadline and calls it fixed
Reads WaitCount once and treats it as a current gauge
Adds more worker goroutines to increase throughput
Never exports db.Stats and diagnoses only from error rates

A Go worker pool's db.QueryContext calls return context deadline exceeded while the database logs no slow queries — how do you prove the connection pool is the bottleneck?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The symptom and why it is confusing

Where the deadline actually goes

Proving it with db.Stats

Confirming from the other direction

What to change

What not to conclude

▾common follow-ups4 questions

▾red flags5 flags

The symptom and why it is confusing

Where the deadline actually goes

Proving it with db.Stats

Confirming from the other direction

What to change

What not to conclude

answer

go deeper