How do you set SetMaxOpenConns across a service's replicas when the database's connection budget is shared with the rest of the fleet?

## What is actually being decided `db.SetMaxOpenConns(n)` looks like a local performance knob and is not. The database is a shared, finite resource with a hard limit on concurrent sessions, and every service pointed at it is spending from the same pot. The quantity the database experiences is: ``` cap x max replicas x environments + batch jobs + migration runners + canaries + operator sessions ``` So the decision is a *budget allocation*, and it is one the database's owner — a DBA or a platform team — is entitled to overrule. That is what makes it a leadership call rather than a tuning exercise: you commit to a number, you can be asked to justify it, and the cost of being wrong lands on people who do not read your code. Note what is *not* being decided here. The internal question of how many concurrent database operations your workload needs to hit its throughput target is capacity planning, and the question of how many sessions a given database server can usefully serve belongs to whoever operates it. What you own is: turning a granted allocation into per-process settings, and choosing the behaviour when it is exhausted. ## Deriving the per-process cap Work backwards from the allocation, and use the *maximum* replica count, not the current one. An autoscaler that goes from four to forty pods multiplies your database footprint by ten without anybody changing a line of configuration; a cap chosen at four replicas is a latent incident that fires during your busiest hour. If the platform cannot bound the replica count, that bound has to come from somewhere else, and saying so is part of the job. Then subtract the processes that are easy to forget. Migration runners, nightly batch jobs, a staging environment sharing the server, and human operators all consume sessions. Leaving headroom for an administrator to connect during an incident is not a nicety: a database whose session slots are entirely consumed by application pools cannot be inspected while it is failing. The result is usually a smaller per-process cap than engineers expect, and that is fine — a smaller cap makes contention visible in your own metrics rather than as connection refusals scattered across every other service. ## Choosing the exhaustion posture The second half of the decision is what callers experience when the cap is reached, and there are three defensible postures: - **Bounded wait, then fail.** Every call carries a context deadline, so a caller queues briefly and then gives up. This is the default posture and it should be deliberate: it converts saturation into a fast, countable error rather than an unbounded backlog. - **Shed earlier, at the edge.** Under sustained overload, queueing at the pool means every request waits and then fails, which is the worst of both. Rejecting or throttling at admission keeps latency sane for the requests you do accept. - **Prioritise.** If one workload must not starve another — interactive requests versus a batch backfill — the pool gives you no priorities, so separate pools with separate caps, or a bounded admission gate in front of the batch path, are how you enforce it. What you should refuse is the unbounded posture: no deadline on database calls means goroutines and their request state pile up until memory, not the database, ends the incident. ## Making the argument with data A request for more connections that comes with numbers is a different conversation from one that does not. Export `db.Stats()` continuously: `MaxOpenConnections`, `InUse`, `Idle`, the rate of `WaitCount`, and the mean acquire wait derived from `WaitDuration` over `WaitCount`. Saturation that is real looks like `InUse` pinned at the cap with a rising wait rate during traffic peaks. Saturation that is imaginary looks like a flat `WaitCount`, and in that case the connections are not your problem and asking for more of them wastes the budget. The same instrumentation protects you in the other direction: if you can show your service holds well under its allocation, you have an argument against being cut when the pot is redistributed. ## When the allocation genuinely is not enough In rough order of what to try: 1. **Hold connections for less time.** Throughput per connection is the inverse of holding time; halving the time a job occupies a connection is worth exactly as much as doubling the cap and costs the database nothing. 2. **Reduce the number of pools.** Consolidating chatty replicas or moving batch work into fewer, longer-lived processes frees sessions immediately, because the multiplier is replicas, not requests. 3. **Put a connection pooler in front of the database.** A server-side pooler multiplexes many client connections onto few server sessions and is the standard answer when a fleet's process count, not its query rate, is the constraint. It also changes the semantics of session-scoped state, which is a tradeoff to raise explicitly rather than discover. 4. **Move read traffic off the primary, or split the workload.** This is an escalation into someone else's design space, so it belongs in the conversation, not in a unilateral change. ## The posture to hold Commit to a number, write down what it multiplies out to, instrument it, and revisit it when the replica ceiling changes. The failure mode this discipline prevents is not a slow service — it is one team's autoscaling event exhausting a shared server's sessions and taking down services that never changed anything.