Two instances of a service are registered in Consul and one instance's HTTP check has gone critical. Explain what a DNS lookup and the `/v1/health` and `/v1/catalog` HTTP endpoints each return now, and how that state differs from the instance having been deregistered.

## Registration and health are two separate facts A Consul registration says "this node offers this service on this address and port". A check says "and right now it is passing, warning or critical". Nothing about a failing check removes the registration. Understanding that separation explains almost every surprising result in this area. ## The three check states Checks report **passing**, **warning** or **critical**. Warning is a real, distinct state, not a synonym for failing — a script check exiting `1` yields warning, exiting `2` or higher yields critical. It matters here because *by default, warning instances are still returned by DNS*. Only critical instances are excluded. If you want warning excluded too, you set `only_passing` in the agent's `dns_config`. Teams that assume warning is filtered are often routing traffic to instances that reported degradation. There is also a check nobody registers explicitly: **`serfHealth`**, the node-level check maintained from gossip. When a node stops gossiping, `serfHealth` goes critical and *every* service on that node drops out of discovery, regardless of how those services' own checks last reported. ## What each query surface does **DNS** (`web.service.consul`) returns addresses for instances that pass the health filter. No detail, no reason, no way to see the excluded ones. **`GET /v1/health/service/web`** returns every instance *with* its full check list, letting you decide yourself. Adding `?passing` makes Consul apply the filter server-side, which is what most client libraries do. **`GET /v1/catalog/service/web`** returns the registrations, unfiltered. This endpoint is about *what is registered*, not *what is usable*. Using it to build an upstream pool is a genuine production bug: the pool includes instances Consul already knows are down. ``` # usable instances only curl -s 'http://127.0.0.1:8500/v1/health/service/web?passing' # every registration, healthy or not curl -s 'http://127.0.0.1:8500/v1/catalog/service/web' ``` ## Critical versus deregistered | | Check critical | Deregistered | |---|---|---| | In the catalog | yes | no | | Returned by DNS | no | no | | Visible in the UI | yes, in red | no | | Returns on recovery | automatically, at the next passing check | only if something registers it again | | Who caused it | the check failing | an explicit deregistration, or an automatic one | The critical state is a *soft* removal from traffic. It is reversible, self-healing, and it preserves the information that this instance exists and is unwell — which is exactly what you want during a deploy, a slow restart or a dependency blip. Deregistration is a *hard* removal: the registration is gone and the instance vanishes from the catalog. ## How each one gets triggered Deregistration happens when: - The application or its supervisor calls `PUT /v1/agent/service/deregister/<service_id>` on the local agent — the correct thing to do in a shutdown hook. - The agent is stopped gracefully with `consul leave`, which deregisters the node's services on the way out. - A check has been critical long enough to trip `deregister_critical_service_after` on the check definition, if it was set. A critical state that *persists* usually means none of those happened: the process was killed, or the container was replaced while the agent kept running, and the check simply keeps failing against nothing. ## The consequence people meet in production A node's agent is killed abruptly. Gossip notices, `serfHealth` goes critical, and the node's services stop resolving — good, traffic moves away in seconds. But the node and its services stay in the catalog, red in the UI, for a long time. Nothing is broken; the catalog is telling you a registration exists whose owner is unreachable. Operators who "clean up" by deleting catalog entries while the agent is still alive discover the entries reappear, because the agent's anti-entropy sync re-asserts its own node's state. The removal has to happen at the agent that owns the registration.