A Java service on Tomcat behind a reverse proxy starts returning gateway timeouts to users, yet Tomcat's own logs look quiet, CPU is near idle and the host has plenty of memory. How do you confirm the connector's thread pool is exhausted, and what do you do once you have?

## Read the symptom before touching config Three facts in the report already narrow the diagnosis. Users see gateway timeouts, which means the *proxy* gave up waiting — the failure is being reported by the hop in front, not generated by Tomcat. CPU is idle, so nothing is computing; threads are waiting. And Tomcat's log is quiet, which is not evidence of health: Tomcat writes an access-log entry when the response is complete, so requests still in flight — exactly the ones timing out — have not been logged yet. Every one of those points to the same shape: request threads alive but blocked. ## Confirming it with a thread dump A thread dump is the definitive evidence, and it costs almost nothing to take. ```bash jcmd <pid> Thread.print > /tmp/dump1.txt grep -c 'http-nio-8080-exec-' /tmp/dump1.txt ``` What you are looking for is two things at once: the **count** of `http-nio-8080-exec-` threads equalling the connector's `maxThreads`, and a **shared stack frame** near the top of most of them. If two hundred threads are all sitting in a socket read inside the same HTTP client call to the same downstream service, you have both the diagnosis and the culprit in one artefact. Take two or three dumps a few seconds apart: threads that are progressing move between dumps, threads that are stuck do not. The stack tells you which category of stall this is. Blocked in a socket read on a downstream call means a dependency is slow or has stopped answering. Blocked waiting for a JDBC connection means the *database pool*, not the connector pool, is the real ceiling and the connector pool is merely the queue behind it. Blocked on a `synchronized` monitor held by one thread means a lock convoy, and the holder's stack names the offending code. ## Confirming it continuously with JMX A dump is a snapshot; you also want this as a standing signal. The connector registers a `Catalina:type=ThreadPool` MBean whose `currentThreadsBusy` is the number of exec threads currently processing requests, alongside `currentThreadCount` (threads in the pool) and `connectionCount`. `currentThreadsBusy / maxThreads` is the single most valuable Tomcat saturation metric there is, and the sequence during an incident is characteristic: busy climbs to `maxThreads` and pins there, connections keep climbing behind it, throughput flattens, latency grows without bound, error rate at Tomcat stays zero. Alerting on sustained busy ratio catches this long before customers do. ## Why the errors appear at the proxy first Once all exec threads are occupied, Tomcat still accepts new connections — up to `maxConnections`, with the OS accept queue (`acceptCount`) behind that — and simply does not process them yet. So the connections succeed, the requests are never answered, and the proxy's read timeout fires. From the proxy's logs it looks like a backend that stopped responding; from the host it looks like an idle machine. This is precisely why a thread-exhausted Tomcat is repeatedly misfiled as a network problem, and why knowing the layering wins the interview. ## Fixing it in the right order 1. **Bound the blocking call.** The root cause is nearly always an unbounded wait. Connect and read timeouts on the HTTP client, a query timeout and a pool-acquisition timeout on JDBC. A request that cannot exceed two seconds cannot hold a thread for five minutes. This is the fix that actually works. 2. **Isolate.** If one endpoint depends on a flaky service, do not let it consume the pool the healthy endpoints share. A circuit breaker sheds load once the dependency is failing; a bulkhead — a separate bounded pool for that call, or a separate connector with its own `<Executor>` — caps the damage at a fraction of capacity. 3. **Shed load deliberately.** A shorter accept backlog and a client-side timeout shorter than the work's worst case turn a slow death into fast, retryable failures. Serving a response nobody is waiting for is pure waste. 4. **Only then, resize.** If threads are genuinely productive and merely numerous — lots of short waits, not one long one — a larger `maxThreads` is legitimate. Confirm first that the downstream can absorb the extra concurrency, because otherwise you have moved the queue, not removed it. ## What to change so it is caught next time Export the busy-thread ratio and the connection count as metrics, and log request duration so slow-but-successful requests are visible before they become timeouts. Have an operator-triggerable thread dump documented in the runbook. And write down the timeout for each downstream dependency as a deliberate number — the most common root cause here is not a missing metric but a client library that defaulted to waiting forever.