Health & Readiness Checks

## What a health check actually is A **health check** is a question the platform asks a running instance on a fixed schedule — over the network, by running a command inside the container, or by opening a connection to a port. The answer is only ever success or failure. The platform counts **consecutive failures**, and when that count reaches a configured **failure threshold** it takes an action. Everything interesting is in the action, not the question. Two checks can call an identical endpoint with identical settings and still be completely different instruments, because the platform does completely different things when each one fails. ## The liveness verdict: end this process A check wired to the restart action — a **liveness check** — asserts exactly one thing: *this process is no longer worth keeping*. Past its threshold, the platform ends the container and starts a fresh one. That is a destructive verdict, and its costs all follow from one fact: a new process starts empty. - **In-memory state is gone.** Caches, loaded indexes, open pools and whatever warm-up the process had accumulated since it started are discarded. - **In-flight requests are lost.** Work the instance had accepted and not finished dies with it. - **Capacity is gone for a whole start-up.** If the service takes minutes to become useful, the fleet is short one instance for minutes. - **Nothing is diagnosed.** A restart clears the symptom and destroys most of the evidence in the same move. Liveness exists for one narrow case: a process that is running but permanently stuck — deadlocked, spinning, or wedged on something it will never get — where nothing short of a restart recovers it. If the condition would clear by itself, restarting is a strictly worse way to wait. ## The readiness verdict: do not send me work A check wired to the routing action — a **readiness check** — asserts something much weaker: *right now, do not send me requests*. Past its threshold, the platform removes the instance from the **routing set**: the set of instances the service-level address is allowed to hand traffic to. Nothing else happens. - The process keeps running, with all its memory intact. - It keeps the slot it was placed into, so its reservation on the node is still held. - It can keep doing background work — loading, reconnecting, compacting, draining. - The moment the check passes again, the platform returns it to the routing set. Readiness is therefore reversible and self-correcting, and it is the right answer to every condition that is temporary: still warming up, a local queue too deep to accept more, a connection this instance alone needs to re-establish. ## The cost of failing each | | liveness failure | readiness failure | |---|---|---| | what the platform does | ends the process, starts a fresh one | removes it from the routing set | | the process | killed | keeps running | | in-memory state | lost | kept | | in-flight requests | lost | allowed to finish; no new ones arrive | | can the instance undo it? | no — that instance is gone | yes — pass once and it rejoins | | time to full recovery | a complete start-up | roughly one check interval | ## Why answering both from the same signal is the trap If one signal drives both verdicts, every condition gets answered with the most expensive action available. A twenty-second stall that should have parked the instance restarts it instead, throwing away state that took minutes to build. Worse, conditions like these are often identical on every instance at once, so the expensive verdict fires fleet-wide rather than on one bad instance. The useful mental test before wiring anything to a restart: *if I restarted this instance right now, would the condition be gone?* If the honest answer is no — because the cause is outside this process — the condition belongs to the routing verdict, or to how individual requests are answered, not to liveness. ## The third job, in one line Neither verdict knows that an instance has never served yet, which is why platforms also offer a **startup check**: a bounded allowance that holds the other two off while a slow start completes. It buys time; it does not restart and it does not route. ## What an interviewer is listening for Not the definitions — the consequences, stated in the right direction: restart versus unroute, destructive versus reversible, and the judgment that a cheap reversible verdict should be preferred wherever it is honest.