During rolling restarts, a Spring Boot service on embedded Tomcat returns errors for requests that were already in flight. What does `server.shutdown=graceful` change about how the embedded container stops, which property bounds it, and why is that setting alone often not enough?

## What the default does `server.shutdown` defaults to `immediate`. When the JVM receives a termination signal, Spring's shutdown hook closes the application context and the web server is stopped without regard for work in progress. A request halfway through a database transaction has its connection torn out; the client sees a connection reset, or, if a proxy sits in front, a 502. Under a rolling deploy of a busy service this is not a rare edge case — every instance replacement drops whatever it was holding. ## What graceful adds ```properties server.shutdown=graceful spring.lifecycle.timeout-per-shutdown-phase=30s ``` With `graceful`, shutdown becomes two phases. First the web server stops accepting: new connections and new requests are refused, so nothing else enters the application. Then the container waits for active requests to complete. Once they drain — or once the grace period expires — the context closes and the process exits. `spring.lifecycle.timeout-per-shutdown-phase` bounds that wait, defaulting to 30 seconds. Choosing it is a real trade. Too short and long requests are still killed, just later. Too long and a deploy crawls, or a wedged request holds the instance until the platform loses patience and sends a kill signal — at which point you are back to an abrupt stop with extra delay added. A sensible value sits just above the service's realistic worst-case request duration, which is itself bounded by the timeouts on its own downstream calls. If a request can legitimately run for two minutes, no shutdown setting will save it; shorten the request. ## Why the process must be asked nicely Graceful shutdown runs from the JVM shutdown hook, so it only happens if the process receives a signal it can handle. A forced kill bypasses everything. That has two practical implications: the process must be the signal recipient (a shell wrapper that does not forward signals will swallow it, leaving the JVM oblivious until it is killed), and the platform's termination grace period must be longer than the Boot shutdown timeout, or the kill lands mid-drain. ## The gap graceful shutdown cannot close This is the part candidates usually miss. Graceful shutdown protects requests that have *already arrived*. It does nothing about the requests still being sent. A load balancer or registry learns an instance is going away only when its health check fails or its deregistration propagates, and that takes seconds. During those seconds the instance is refusing — so the caller gets connection errors from an instance that is behaving exactly as configured. The fix is sequencing, and it lives outside the application: 1. Mark the instance out of rotation, or let its health signal go unhealthy. 2. **Wait** long enough for the routing tier to observe that and stop sending — typically a fixed delay in the deployment platform before the termination signal is sent. 3. Only then send the signal, letting graceful shutdown drain what remains. Step 2 is the one that is forgotten. Without it, graceful shutdown converts "errors on in-flight requests" into "errors on newly routed requests" and the error rate barely moves, which is exactly why the property alone often does not fix the reported symptom. ## What still gets cut Graceful shutdown drains requests, not connections. Long-lived streams — server-sent events, WebSocket sessions, a long-polling endpoint — will simply be running when the grace period expires and will be terminated. Those need their own application-level handling: a close frame or a signal that tells clients to reconnect elsewhere before the process goes. Background work started by a request but running on a separate executor is likewise not tracked as an in-flight request; its lifecycle is separate. ## Verifying it Do not take the property on trust. Put a slow endpoint under steady load, send the same termination signal the platform sends, and check three things: the in-flight requests completed with a normal status, the process exited within the grace period, and the error count during the window is zero rather than merely lower. If errors moved from in-flight requests to connection refusals, the missing piece is the delay in step 2, not the Boot configuration.