You are taking a backend out of a load balancer's pool for maintenance. Why is "stop sending it new requests" not the same thing as draining it, and what does the proxy have to do to remove it without failing work that is already in flight?

## Two different questions "Is this backend eligible for the next routing decision?" and "is this backend free of work?" are separate questions, and only the first one is answered instantly when you take a member out of the pool. Draining is the process of getting from the first answer to the second without turning in-flight requests into errors. ## What is still attached when you mark it down At the instant the backend leaves the rotation, the proxy is still holding several kinds of attachment to it: - **Requests already dispatched.** A request forwarded three seconds ago is still executing. The response has to come back over the connection it left on. - **Idle pooled connections.** If the proxy pools upstream connections, it holds open sockets to that backend. Worse, there is a small window where a worker selected the backend just before the state change and is about to write a request onto one of those sockets. - **Sticky clients.** With cookie or hash affinity, some clients are pinned to that member. Drain has to decide what happens to them: usually new sessions go elsewhere while existing ones ride out the drain window. - **Streams.** WebSocket upgrades, gRPC server streams, server-sent events and long-polling connections stay open by design. They will not finish because you stopped routing. None of those are addressed by a routing change alone. ## What a real drain does A usable drain has four parts. **A distinct state.** Not "healthy" and not "failed" — a third state meaning *no new work, finish what you have*. This matters because the blunt substitute people reach for, deliberately failing the health check, is a different instruction: many proxies treat a failed member as a candidate for immediate ejection and may reset connections to it rather than let them complete. **An in-flight counter.** You need to see the number of active requests and connections on the draining member fall toward zero; otherwise you are guessing when it is safe to stop the process. Most proxies expose this in their stats or admin interface. **A deadline.** Pick it from the distribution of request durations, not from a round number: roughly p99 request duration plus margin covers ordinary traffic. Without a deadline, one hung request holds the maintenance open indefinitely. HAProxy's `hard-stop-after` is one expression of this idea at the process level. **A forced close at the deadline.** When the timer expires the remaining connections are closed. For streaming traffic this is the normal path, not the exception, and it means clients must be able to reconnect — which is a client-side property you have to have designed for in advance. ## The proxy's own shutdown is the same problem The symmetric case is replacing the proxy process itself during a config reload or a deploy. Graceful shutdown means the process stops accepting new connections, keeps serving established ones, and exits when they finish or the deadline hits. nginx does exactly this on SIGQUIT (old workers linger while new ones take over); HAProxy soft-stops on SIGUSR1. In both cases the old and new processes serve simultaneously for the drain window, which is a capacity fact your sizing has to accommodate. ## Why the load balancer cannot just re-send the work A tempting shortcut: when the backend goes away, replay its in-flight requests on another member. The proxy generally must not. Once the request was forwarded, the backend may have already committed a side effect — charged a card, written a row — and the proxy has no way to know. Replaying is only safe when the request provably never reached the application, which is why proxy-level retry conditions are restricted to connection-establishment failures rather than "anything that did not come back". ## Failure signatures When draining is missing or misconfigured, the symptoms are recognisable: - A burst of 502s at the exact moment of a scale-in or deploy, with a duration equal to the tail of your request-latency distribution. - Errors that stop as soon as you add a manual pause between removing the member and stopping the process — the classic sign that the pause *is* the missing drain. - Streaming clients dropping en masse at deploy time, because nothing ever asked them to reconnect gracefully. ## The one-line version Removal is about the future; drain is about the present. Give the draining member a state of its own, watch its in-flight count, bound the wait with a deadline you derived from real latency data, and accept that anything long-lived will be cut rather than finished.