When a container sends SIGTERM to an ASP.NET Core 10 API mid-traffic, what is the shutdown sequence, and what do in-flight requests experience?
answer
- the signal becomes StopApplication
- server stops first, then hosted services
- stop accepting, then drain
- Connection: close and HTTP/2 GOAWAY
- 30 s ShutdownTimeout, then abort
basics
~10 sSIGTERM triggers StopApplication: ApplicationStopping fires, Kestrel stops accepting connections and lets in-flight requests finish, for up to HostOptions.ShutdownTimeout (30 seconds). Requests still running then have their connections aborted; hosted services stop and ApplicationStopped fires.
solid answer
~40 sThe host's console lifetime catches SIGTERM (and Ctrl+C), suppresses the default termination and calls `StopApplication()`. `ApplicationStopping` callbacks run, then `app.Run()` calls the host's `StopAsync` with a token that cancels after `HostOptions.ShutdownTimeout`, 30 seconds by default. Hosted services stop in reverse order, and `WebApplication` registers the web server last, so Kestrel stops first: it unbinds its listeners, disables keep-alive on HTTP/1.1 connections (responses carry `Connection: close`), sends HTTP/2 `GOAWAY`, and waits for in-flight requests. A seat purchase that completes within the window succeeds normally. If the token fires first, Kestrel aborts the remaining connections, which cancels their `RequestAborted` and leaves the client with a dropped connection. Then the remaining hosted services stop, `ApplicationStopped` fires and `Run` returns.
code
http · 6 linesHTTP/1.1 201 Created
Content-Type: application/json
Connection: close
Location: /orders/88412
{"orderId":88412,"seat":"B-14"}go deeper
Recall that SIGTERM and Ctrl+C start a graceful shutdown and that requests already running get time to finish.
Walk through the order: ApplicationStopping, server stop and drain, the 30-second ShutdownTimeout, abort, hosted services, ApplicationStopped.
Explain what each request state experiences, why the server stops before background services, and how to tell from logs that the drain timed out.
Align application drain time, platform grace period and traffic shifting into one shutdown contract that teams can rely on during deploys.
## From signal to StopApplication A container platform stops an instance by sending **SIGTERM** to the process, and later SIGKILL if it is still alive. In an ASP.NET Core app the Generic Host's **console lifetime** registers handlers for SIGINT (Ctrl+C), SIGQUIT and SIGTERM. On Linux, as in a container, each handler marks the signal as handled, so the runtime does not terminate the process, and calls `IHostApplicationLifetime.StopApplication()`. Take a concert-ticket API in the middle of an on-sale: hundreds of seat purchases are in flight when the orchestrator replaces the instance. ## The sequence in order 1. **`ApplicationStopping` fires.** Its callbacks run synchronously, and `Microsoft.Hosting.Lifetime` logs `Application is shutting down...`. 2. **The host's `StopAsync` starts.** `app.Run()` was waiting on `ApplicationStopping`; it now calls `StopAsync` and creates a cancellation token that fires after **`HostOptions.ShutdownTimeout`**, **30 seconds** by default. Everything below shares that one budget. 3. **The web server stops first.** Hosted services stop in reverse registration order, and `WebApplicationBuilder.Build()` adds the server's hosted service after yours, so Kestrel is stopped before your background services. 4. **Kestrel stops accepting.** It unbinds its listening sockets, so new connections to this instance are refused. 5. **Kestrel asks every connection to close gracefully:** - an HTTP/1.1 connection with a request in progress finishes it, and the response carries `Connection: close`; an idle keep-alive connection is closed; - an HTTP/2 connection is sent `GOAWAY`, telling the client not to open new streams on it while existing streams finish. 6. **Kestrel waits** for all connections to finish, or for the shutdown token. 7. **If the token fires first**, Kestrel aborts what is left with a `ConnectionAbortedException` whose message says request processing did not complete within `HostOptions.ShutdownTimeout`. Each affected request's `HttpContext.RequestAborted` is cancelled, and the client sees the connection drop mid-response. 8. **Your hosted services stop** (`StopAsync`, with the same, possibly already cancelled, token). 9. **`ApplicationStopped` fires**, `app.Run()` returns, and the process exits when `Main` ends. ## What the purchases experience | Request state when SIGTERM arrives | Outcome | |---|---| | Already running, finishes within the budget | completes normally; response may carry `Connection: close` | | Still running when the budget runs out | connection aborted, client gets a reset, server code sees `RequestAborted` cancelled | | New connection after the listeners closed | refused by this instance; it must be routed elsewhere | | New request on an existing HTTP/1.1 keep-alive connection | not served: the connection is closed after its current request | Note what does **not** happen: shutdown does not cancel `RequestAborted` at the start, and it does not throw into running handlers. A purchase that commits its database transaction in two seconds simply completes. ## Where it goes wrong - **The platform kills the process before the drain finishes.** If its grace period is shorter than the time requests and hosted services need, SIGKILL ends everything with no cleanup. - **Traffic is still routed to the stopping instance.** Kestrel refuses new connections immediately, so the load balancer must stop sending traffic; readiness signalling is a separate concern. - **Long-lived requests** (streams, long polls) keep the drain waiting until the timeout and are then aborted, unless they react to `ApplicationStopping`. - **Slow `ApplicationStopping` callbacks** delay the drain itself. ## Observing it - `Application is shutting down...` marks step 1. - With `Microsoft.AspNetCore.Server.Kestrel.Connections` at `Debug`, the line `Some connections failed to close gracefully during server shutdown.` means the 30-second budget ran out and connections were aborted.
- Why does the server stop before your BackgroundService in a WebApplication?The host stops hosted services in reverse registration order, and `WebApplicationBuilder.Build()` adds the web server's hosted service after all of yours. So it starts last and stops first: requests drain while background services are still running, and they stop afterwards with what is left of the same `ShutdownTimeout` budget.
- A purchase request was still running when the 30 seconds expired. What did the client and the server code see?Kestrel aborted the connection, so the client got a dropped connection or reset rather than a response. On the server, `HttpContext.RequestAborted` was cancelled, so awaited calls that take that token throw `OperationCanceledException`; code not observing it keeps running, but nothing it writes can reach the client any more. Whether the purchase committed depends on where it was, which is why idempotency matters.
- Does SIGTERM cancel RequestAborted for every in-flight request immediately?No. Shutdown starts with `ApplicationStopping`; in-flight requests keep running with their `RequestAborted` untouched. That token is cancelled only if the client disconnects or Kestrel aborts the connection when `HostOptions.ShutdownTimeout` expires.
A box office at closing time: the doors are locked to new customers, everyone already at a window is served, and after a fixed time the manager pulls the shutters on anyone still being served.
- SIGTERM kills an ASP.NET Core process immediately unless you handle it yourself.
- All in-flight requests are cancelled as soon as shutdown starts.
- Kestrel keeps accepting new connections until ShutdownTimeout expires.
- HostOptions.ShutdownTimeout defaults to 5 seconds.
- Background services stop before Kestrel drains requests.
Assumes ASP.NET Core 10 on .NET 10, where HostOptions.ShutdownTimeout defaults to 30 seconds (5 seconds in .NET 5 and earlier) and WebApplication registers the server's hosted service last.