HAProxy's default HTTP log line ends each request with a four-character termination state such as `sH--` or `SC--`. What do the first two characters encode, and what do `sH`, `sQ`, `cD` and `SC` each tell you about a failed request?
answer
- two letters: who, and when
- case carries meaning, not just the letter
- lowercase is a timeout expiring
- uppercase is somebody closing or resetting
- phase letters follow the request's journey
basics
~20 sIn HAProxy logs the first character says who or what ended the session and the second says which phase it was in. sH is a server-side timeout waiting for response headers, sQ a timeout in the queue, cD a client timeout during data transfer, and SC a server refusing or resetting the connection.
solid answer
~60 sThe first character is the cause of termination and the second is the session state when it happened. Lowercase `c` and `s` mean a client-side or server-side *timeout*; uppercase `C` and `S` mean the client or server actively aborted; `P` is the proxy itself (a rule denied it), `D` means the server was down, `R` a resource shortage, `-` a normal close. The second character marks the phase: `R` reading the request, `Q` waiting in the queue, `C` establishing the connection, `H` waiting for the response headers, `D` in the data phase, `L` in the final transmission. So `sH` is `timeout server` firing while HAProxy waited for the response headers — the app is too slow, or the timeout is too short. `sQ` means the request aged out of the queue under `timeout queue` because no server had a free slot. `cD` is `timeout client` during body transfer, typically a client that went away. `SC` is the server or something in front of it refusing or resetting the TCP connection at connect time.
go deeper
Know that HAProxy logs a short termination-state code on every request and that it is the first thing to read when a request failed, rather than guessing from the status code alone.
Explain the structure — first character the cause, second the phase — and decode the common pairs, in particular that lowercase means a timeout expired while uppercase means an end actively closed or reset.
Use them to drive a real diagnosis: sH sending you to application latency rather than the timeout knob, sQ to server maxconn and queue depth, and a mismatch between sH in the logs and L7OK in the stats as evidence the probe is not testing what traffic exercises.
Make the signal usable at scale: ensure the log format is standardised and parsed across every proxy, that termination flags are a first-class dimension in dashboards and alerts, and that on-call runbooks name the flags rather than describing symptoms.
## Where the field lives HAProxy's default HTTP log line carries a `termination_state` field of four characters, right after the timing and connection-count fields. TCP mode logs the first two only. Nothing else in the line tells you *why* a request ended the way it did, which makes these two letters the fastest diagnostic HAProxy gives you. ``` ... 0/0/0/-1/30001 504 194 - - sH-- 12/12/0/0/0 0/0 "GET /orders HTTP/1.1" ``` ## Character one: who ended it - `-` — normal termination, nothing went wrong. - `C` — the **client** aborted: closed or reset the connection itself. - `c` — a **client-side timeout** expired (`timeout client`). - `S` — the **server** aborted, or something between HAProxy and the server did: a reset or an abrupt close. - `s` — a **server-side timeout** expired (`timeout server`, or `timeout queue`/`timeout connect` depending on phase). - `P` — the **proxy** terminated it, typically because a rule denied the request or a limit was enforced. - `R` — a **resource** was exhausted (memory, sockets, file descriptors). - `I` — an **internal** error in HAProxy. - `D` — the **server was down** at dispatch time. - `L` — the response was **served locally** by HAProxy (a redirect or a stats page, not from a server). - `K` — the session was **killed** by an administrator. The lowercase/uppercase distinction is the one to internalise: **lowercase means a timeout fired, uppercase means somebody closed or reset**. That single rule separates "we gave up waiting" from "the other end hung up", which are very different investigations. ## Character two: which phase - `R` — reading or waiting for the complete request. - `Q` — waiting in the backend or server queue. - `C` — establishing the TCP connection to the server. - `H` — waiting for, or processing, the server's response headers. - `D` — the data phase, body bytes moving in either direction. - `L` — the last transmission of data to the client. - `T` — the request was tarpitted. - `-` — normal, no special phase. ## Reading the common combinations **`sH`** — `timeout server` expired while HAProxy waited for the response headers. The server accepted the connection and then said nothing in time. It is the single most common flag behind a 504, and it means the application is slow or stuck, not that the network broke. The reflex to raise `timeout server` is usually wrong; find out why the response took that long first, and only then decide whether the timeout is genuinely mis-sized for this route. **`sQ`** — the request sat in the queue until `timeout queue` expired without a server slot opening. This is a *capacity* signal: `maxconn` on the servers is limiting concurrency and demand exceeded it for longer than you were willing to wait. Look at the queue counters in the stats before touching timeouts. **`cD`** — `timeout client` fired during the data phase. Usually a client that went away mid-transfer: a mobile connection dropped, a user navigated away from a download, an upload stalled. Common and often benign in bulk, alarming only if it correlates with a particular route or a rise in volume. **`SC`** — the server, or a firewall or NAT device in front of it, refused or reset the connection during the connect phase. Connection refused, a security group dropping the packet, or a listener that died. Note the contrast with `sC`, which is `timeout connect` expiring: refused-immediately versus never-answered are different faults with different causes. **`SH`** — the server reset the connection while sending headers, or sent something HAProxy could not parse as a valid response. Often a crashed worker or a protocol mismatch. **`PR`** — the proxy rejected the request during the request phase: a denying rule, or a malformed or oversized request. When HAProxy could not select a server at all, the server field of the log reads `<NOSRV>` rather than a server name — pair that with the flags and you can distinguish "the backend is empty" from "the chosen server failed". ## Pairing flags with check status The log tells you what happened to a request; the stats tell you what the checker thinks. The stats page's last-check column, and the `check_status` field of `show stat`, carry codes with the same layered vocabulary: `L4OK`, `L4CON` (connection refused or reset), `L4TOUT` (connect timed out), `L7OK`, `L7STS` with the returned code (an HTTP status that failed the expectation), `L7TOUT`, `L7RSP` (an invalid or unparseable response). A burst of `SC` in the logs alongside `L4CON` in the stats is a coherent story: the server is refusing connections and the checker agrees. A burst of `sH` alongside a cheerful `L7OK` is the more interesting case — real traffic is timing out while the probe passes, which is the argument for probing something meatier or for watching live traffic as well. ## Why it is worth memorising four of them You will not remember the whole table, and you do not have to. Remember that lowercase is a timeout and uppercase is an abort, remember `H` is "waiting for the response" and `C` is "connecting", and you can decode most incidents on sight — then look the rest up.
- Your logs are full of `sH` and 504s. Is raising `timeout server` the right fix?Almost never as the first move. `sH` says the server accepted the connection and then produced no response headers within `timeout server`, so the question is why that route became slow — a lock, a slow dependency, an unindexed query. Raising the timeout only converts fast failures into slow ones and lets requests pile up. Fix the latency, then size the timeout deliberately for that route.
- How do you tell from HAProxy's logs whether a 503 was caused by an empty backend or by a failed connection to a chosen server?Look at the server field alongside the flags. When no server could be selected the field logs `<NOSRV>`, which points at every server being DOWN, in MAINT, or the backend having none. If a server name is logged, HAProxy picked one and the flags — `SC` for refused or reset, `sC` for a connect timeout — tell you how the attempt to reach it failed.
- Where do you look to see why a server's last health check failed, as opposed to why a request failed?The stats page's last-check column, or the `check_status` field in `show stat`. The codes use the same layered vocabulary as the flags: `L4CON` for a refused or reset connection, `L4TOUT` for a connect timeout, `L7STS` with the returned code when an HTTP status failed the expectation, `L7TOUT` for no response in time, `L7RSP` for a response it could not parse.
The termination flags are HAProxy's two-letter cause-of-death note on every request: the first letter says who hung up, the second says at what point in the conversation it happened.
saying these in an interview costs you the question
- Ignores letter case and reads c and C as the same thing
- Reads sH as a network problem rather than a slow server
- Raises timeout server as the first response to 504s
- Confuses SC (refused or reset) with sC (connect timed out)
- Treats sQ as a latency issue instead of a concurrency limit