Your load balancer's access logs show a burst of HTTP 502 and 504 responses, while the application servers behind it log almost no errors. How do you work out where the failure actually is?

## Why the origin's clean logs are the clue, not a contradiction 502 and 504 are emitted **by the intermediary**, not by the application. So a quiet origin error log is exactly what you expect: in the 502 case the application often never ran, and in the 504 case it ran and believed it succeeded. Any diagnosis that starts by grepping application exceptions will find nothing and stall. Start instead at the hop that generated the status. ## Step 1: split the signal Chart 502 and 504 separately over time, per upstream instance and per route. The shapes tell you a lot: - **502 spikes aligned with deploys or restarts** — lifecycle problem. - **502 at a steady low rate, evenly spread** — connection reuse or header-size problem. - **504 rising with latency percentiles** — a saturation or dependency problem. - **Errors concentrated on one or two instances** — a sick host, not a systemic fault. ## Step 2: the 502 checklist 1. **Process health.** Are origin processes crashing, being OOM-killed, or restarting? Correlate 502 timestamps with restart counts and kernel OOM messages. 2. **Deploy and drain.** During a rolling update, instances must be removed from rotation and allowed to finish in-flight requests before the process is signalled. If termination beats deregistration, the proxy forwards to a socket that is closing — refused or reset connections, and 502s. 3. **Keep-alive races.** If the origin's idle keep-alive timeout is shorter than the proxy's, the proxy will eventually pick a pooled connection the origin has just closed and get a reset mid-request. The fix is to make the upstream's idle timeout comfortably longer than the proxy's, and to allow retry on idempotent requests that fail before any bytes of the response arrive. 4. **Header and buffer limits.** A response whose header block exceeds the proxy's buffer is rejected as invalid. Large cookies, long redirect targets and verbose auth headers are the usual culprits. 5. **Protocol mismatch.** Speaking plaintext to a TLS listener, or an HTTP/2 expectation against an HTTP/1.1-only upstream, yields bytes the proxy cannot parse. ## Step 3: the 504 checklist 1. **Compare latency to the timeout.** Pull the origin's response-time distribution for the affected route. If p99 is brushing the proxy's read timeout, the 504 rate is simply the tail crossing the line. 2. **Audit the timeout budget end to end.** Each hop should time out sooner than the hop in front of it, so failure surfaces at the layer that can act. When an inner timeout is longer than the outer one, the outer hop always gives up first and you get 504s while the inner call keeps working — and keeps holding a connection and a thread. 3. **Find the slow dependency.** Traces beat logs here: which span dominates? Lock waits, a saturated connection pool, an N+1 query, a slow third-party call. 4. **Check for queueing.** If requests are waiting in an accept queue or a thread pool before work begins, origin-side timers that start at handler entry will look healthy while wall-clock latency at the edge is terrible. Measure time from connection accept, not from handler start. ## Step 4: correlate Inject a request id at the edge, propagate it downstream, and log it everywhere. Then for a sample of failures ask one question: **did the origin ever see this request?** - No matching origin line → the request died before or during forwarding: pure 502 territory. - Origin line exists, status 200, completing after the edge already responded → 504 confirmed; the work continued after the client gave up. - Origin line exists with a 5xx of its own → the proxy may be translating an origin failure, and you are back to reading application logs after all. ## Step 5: consider correctness, not just availability 504 has a nasty property: the origin may still complete the work. If the route is non-idempotent — creating a payment, sending a message — a client that retries on 504 can duplicate the effect. Part of the fix is operational (raise capacity, cut latency) and part is contractual (make the operation safely repeatable). Similarly, retry-on-502 is usually safe but should be restricted to failures that occurred before any response bytes were received. ## What good answers include Interviewers listen for three things: that you know the codes come from the proxy and therefore start at the proxy; that you have concrete, named causes rather than "check the network"; and that you mention correlation and timeout budgets rather than only raising limits. Raising the proxy timeout to make 504s disappear is the answer that fails the question — it converts a fast failure into a slow one and moves the queue upstream.