In a system using an Object Pool, what is a "pool leak", how does it typically manifest in production, and what design and operational measures prevent or detect it?

## Definition A **pool leak** is a checked-out instance that is never returned. Unlike a memory leak, it does not necessarily consume growing memory — it consumes **capacity**. Each leak permanently reduces the pool's effective size by one until, at N leaks in a pool of N, the pool is dead. ## Why it happens ### 1. Return not on every path ``` res = pool.acquire() doWork(res) // throws pool.release(res) // never reached ``` Any exception, early `return`, `break`, `continue`, or `goto` past the release leaks. So does a release that is itself inside a conditional (`if (success) pool.release(res)`). ### 2. Acquire inside a loop, release outside ``` for (item in items) { res = pool.acquire(); use(res, item) } pool.release(res) // releases only the last one ``` Leaks `n − 1` instances per call. This one exhausts a pool in seconds rather than days. ### 3. Ownership handed to asynchronous code An instance passed into a callback, future, or another thread is returned only if that continuation actually runs. Dropped futures, cancelled tasks, an executor rejecting the completion task, or an exception in a callback that has no handler all leak. Ownership transfer across an async boundary is the hardest case, because "who releases it" is no longer visible in one function. ### 4. Infinite hold The object is technically going to be returned — after a socket read that never completes. Without a read timeout, statement timeout, or query cancellation, this is indistinguishable from a leak and is far more common than most teams expect. It also affects otherwise-correct code. ### 5. Nested / recursive acquire A function that acquires an instance and calls a helper that acquires another consumes two per invocation and can deadlock a bounded pool even without leaking (each of N callers holds one, all wait for a second that never comes). ### 6. Double release The mirror image. Releasing twice can put one instance in the idle set twice, so two callers hold the *same* object concurrently — silent data corruption on a database connection, and much harder to diagnose than a leak. Guard by invalidating the handle on release and ignoring/erroring on a second release. ## How it looks in production - **Monotonic in-use count**: the honest signal. At low traffic the in-use count should return to (or near) zero. If its **floor rises over time**, you are leaking. Peak values are noisy; the floor is not. - **Delayed onset**: with a pool of 20 and one leak per thousand requests, a service can run fine for days and then hit a wall — typically at an off-peak hour, disconnected from any deploy. - **Non-local symptoms**: the requests that fail are the innocent ones arriving after the pool dries up. The culprit path may have run hours earlier and succeeded from the user's perspective. - **Restart "fixes" it**: capacity resets on redeploy, which masks the trend and trains teams to restart rather than investigate. If "restart every night" is folklore in the team, suspect a leak. ## Prevention: make the failure impossible, not merely unlikely 1. **Scoped/lexical API as the only public entry point.** Force `pool.withResource { res -> ... }` and don't expose raw `acquire`. The pool then owns the `finally`. Most leaks die at this design decision. 2. **Deterministic cleanup constructs** where the language has them: try-with-resources / `use`, `using`, `with`, `defer`, RAII destructors, context managers. Make the leased handle implement the language's closeable interface so linters flag an unclosed one. 3. **Static analysis**: "resource not closed on all paths" checks catch a large share, cheaply, at review time. 4. **Handle invalidation on return** — makes both use-after-return and double-release loud instead of silent. 5. **Hard timeouts everywhere below the pool**: socket connect/read timeouts, statement/query timeouts, an overall lease timeout. These convert a permanent leak into a bounded outage. 6. **Don't transfer ownership across async boundaries** unless one component provably owns cleanup — prefer acquiring inside the async task and releasing before it completes, rather than passing a live lease into it. ## Detection: what to instrument - **Leak-detection threshold** (HikariCP's `leakDetectionThreshold` and equivalents): if a lease exceeds N seconds, log a warning **with the stack trace captured at acquire time**. That stack trace is the single most valuable artifact — it names the leaking call site directly, which no aggregate metric can. - **Metrics**: in-use, idle, total, pending acquires, acquire wait p99, timeout rate. Alert on *rising in-use floor*, not just on exhaustion — the floor gives you days of warning; exhaustion gives you an outage. - **Max lease age / forcible reclaim**: some pools can abandon and destroy an instance held beyond a limit. Use with care: the previous holder may still be using it, so reclaim must invalidate the handle so the old holder fails loudly rather than corrupting shared state. - **Load-test with a deliberately tiny pool** (size 1 or 2). Leaks that take days at size 20 surface in seconds at size 1, and this is a cheap CI check. - **Assert-clean in tests**: at the end of each integration test, assert `inUse == 0`; a leaked lease then fails the test that caused it — the right place to learn about it. ## The reclaim dilemma Forcibly reclaiming a long-held instance trades one hazard for another: if the original holder is merely slow (a legitimately long query), you have yanked a live connection out from under it. Safer sequencing is: warn with stack trace first, cancel the in-flight operation, invalidate the handle, then **destroy** rather than recycle the instance — never quietly hand a possibly-still-in-use object to a new borrower.