A replica stops reporting ready but keeps running — what happens to its membership of the set behind the service's virtual address?

## What readiness controls here A readiness report is a per-replica statement of one thing: *this replica can serve requests right now*. The platform uses that report, in this part of the model, for exactly one purpose — **membership of the set behind the service's name and virtual address**. Address in the set while the replica reports ready; address out of the set while it does not. (The check that produces the report, how often it runs and how many failures it takes, belongs to the lifecycle material; what matters here is the consequence for addressing.) So the member set is not a static list of the replicas you declared. It is a continuously edited list of the ones that say they can work. ## What leaving the set does and does not do This is the pairing candidates most often state backwards, so be explicit about both halves: - **New connections stop being dispatched to it.** Dispatch rules on every host are updated to choose from the remaining members. - **The process keeps running.** It is not stopped, not restarted, not replaced, and nothing has failed from the scheduler's point of view. - **It keeps its address and its identity.** Logs and metrics keep flowing; you can still reach it directly if you know where it is. - **Connections it already accepted generally continue.** Removing a member edits which addresses new connections may be sent to; it does not reach into established flows and cut them. - **Capacity behind the name silently drops** by one replica's worth, while the count of running replicas is unchanged. Contrast that with the other consequence a failing check can have in this model: a failing liveness check **restarts** the container, whereas a failing readiness report **removes it from routing and leaves it running**. Naming which one you mean, and matching it to the right consequence, is most of the answer. ## The rejoin nobody sees Recovery is symmetric and silent. The moment the replica reports ready again, its address goes back into the set and the dispatch rules on every host start including it. No caller does anything. No caller is told. Two things follow that bite in production: - A **flapping** replica — one that passes, fails, passes — rotates back into service between failures and takes a share of traffic each time it is in. From the caller's side this looks like a small, wandering fraction of failing requests with no single culprit. - A replica whose readiness report is **shallower than its real health** (it answers a trivial check but its downstream dependency is broken) is a full member of the set the whole time. The operational lesson is to alarm on the number of *ready* members behind a name, not on the number of running replicas — those two counts diverge exactly when this material matters. ## When the set empties 1. The last member stops reporting ready and its address leaves the set. 2. The service declaration is untouched, so **the name still resolves and the virtual address still exists**. 3. New connections to that address fail — refused or timing out — because there is no member to rewrite them to. 4. The caller therefore sees a connection failure against a name that resolves perfectly well. That asymmetry is a genuinely useful diagnostic. A name that fails to resolve points at the declaration or at the caller's permission to see it; a name that resolves cleanly while every connection fails points at an empty or nearly empty member set — that is, at readiness, not at addressing. ## Propagation is not instantaneous A membership change has to reach the dispatch rules on every host that might send traffic. Between the moment a replica stops reporting ready and the moment the last host has been updated, some connections still arrive at it. This is why a replica that is shutting down should keep serving what it has accepted for a grace period rather than closing its listener the instant it decides to leave, and why platforms sequence removal from the set ahead of termination. Treat the removal as *eventually* effective, not as a switch. | what happened to one replica | in the member set | process running | what acts on it | |---|---|---|---| | reports ready | yes | yes | nothing | | stops reporting ready | no | yes | traffic stops arriving; nothing restarts it | | removed, then reports ready again | yes, silently | yes | traffic resumes with no caller action | Read that table in both directions before an interview: the middle row is the one people invert.