An HAProxy backend keeps dispatching to a server whose /healthz probe still returns 200 while real requests to it fail or return 5xx. Which HAProxy server keywords let live traffic itself mark that server down, and what has to be in place for them to take effect?

## Why the probe can lie An active check tests a synthetic request against a path you chose. Real traffic is different: different URLs, different payload sizes, different code paths, and often a different pool of database connections. A server can therefore serve `/healthz` perfectly while failing everything else — a wedged worker pool that still answers the cheap endpoint, a bad deploy that broke one route, a connection pool exhausted only by the expensive queries. HAProxy's answer is to let the traffic that is already flowing act as the signal. ## The three keywords ```haproxy backend api option httpchk http-check send meth GET uri /healthz retries 3 option redispatch server app1 10.0.0.1:8080 check observe layer7 error-limit 5 on-error mark-down server app2 10.0.0.2:8080 check observe layer7 error-limit 5 on-error mark-down ``` **`observe <layer>`** turns the counting on: - `observe layer4` counts connection-level problems — connection refused, reset, connect timeout, premature close. - `observe layer7` counts those *and* application-level errors on the HTTP response: 5xx status codes and responses HAProxy cannot parse. It requires the proxy to be in HTTP mode, since there is no response status to inspect in TCP mode. **`error-limit <count>`** is how many consecutive errors are tolerated before the `on-error` action fires. The default is 10. A successful request resets the counter, so this measures a run of failures, not a rate over a window. **`on-error <action>`** decides what happens when the limit is reached: - `fastinter` — the default. Switch the check schedule to the `fastinter` interval so the active checker investigates sooner. The gentlest option: traffic errors only *accelerate* verification, they never evict on their own. - `fail-check` — count one failed health check (and switch to `fastinter`). Several rounds of this reach `fall` and the server goes DOWN. - `sudden-death` — set the health counter to one short of fatal, so a single subsequent failed check marks the server DOWN. - `mark-down` — mark the server DOWN immediately, without waiting for a check. ## The requirement people miss Every `on-error` action is expressed in terms of the health-check state machine — accelerating it, adding a failure to it, or forcing its outcome. A server line with `observe layer7` but no `check` therefore has nothing to act on. Enable `check` alongside it, always. ## Choosing the aggressiveness `mark-down` with a low `error-limit` is tempting and dangerous. Consider what happens when the fault is *not* in the servers: a shared database goes read-only, so every server starts returning 500 on live traffic while `/healthz` — which does not touch the database — keeps passing. With `observe layer7 error-limit 5 on-error mark-down`, HAProxy dutifully ejects the entire backend within a few seconds and starts answering 503 itself. HAProxy has no notion of a minimum healthy fraction that must remain in rotation; when the last server goes down the backend fails, unless you have declared `backup` servers to fall back to. That argues for a specific posture: use `observe layer7` with a generous `error-limit` and `on-error fastinter` or `fail-check`, so live errors *bring the active checker's attention forward* rather than deciding the outcome by themselves — and let a well-designed check make the eviction call. Reserve `mark-down` for cases where a per-server fault is clearly distinguishable, and where the health endpoint genuinely cannot detect it. Also remember what `observe layer7` counts: **any** 5xx, including ones the application returns legitimately. An API that answers 503 as a documented back-pressure response, or 500s produced by a client sending malformed input to one particular server, all feed the same counter. There is no way to tell `observe` which statuses to ignore, which is another reason not to wire it straight to eviction. ## Not losing the request that found the fault Detection is only half the outcome. The request that hit the broken server still failed. `retries <n>` tells HAProxy how many times to re-attempt a failed connection attempt, and `option redispatch` allows that retry to go to a *different* server rather than the same one — including overriding a persistence decision that would otherwise pin the client. Without `redispatch`, retries against a dead server just fail three times more slowly. For errors that appear after the connection was established, `retry-on` (HAProxy 2.0 and later) extends retries to layer-7 conditions such as `empty-response`, `response-timeout` or specific statuses — with the usual caveat that re-sending a request the server may already have processed is only safe for operations that tolerate it. ## What to say in an interview Name `observe layer4|layer7`, `error-limit` and `on-error`; state the `check` prerequisite; and show the failure mode you are guarding against — a correlated fault ejecting an entire healthy pool because live traffic errors were wired directly to eviction.