In a performance run, why is a reply that arrives with a success status not necessarily a success?

## What a success status actually promises A reply that arrives carrying a success status tells you one thing: something in the path accepted the request and produced an answer. It does not tell you that the answer came from the code you were trying to exercise, that it contains the data the request asked for, or that it is complete. Status and content are separate claims, and a performance run that reads only the first is counting deliveries rather than results. This matters more here than in a functional test, because a performance run counts things. Every reply it accepts becomes a completed unit of work in the throughput figure and a sample in the response-time figures. A reply that is wrong but well formed inflates both. ## Four shapes of false success | Shape | What actually arrives | Cheapest check that catches it | |---|---|---| | Rendered failure page | A human-readable apology page produced by an intermediary or a shared error handler | A field or marker only the real path emits | | Truncated body | A partial answer cut short when the connection closed or a size ceiling was hit | A minimum length, plus a check that the terminating structure is present | | Empty or default payload | A well-formed answer containing zero rows where the seeded input guarantees some | A minimum result count for a request whose input you control | | Stale or degraded fallback | A cached or default answer served because the live path was too slow | A freshness marker or an identifier that changes per request | The fourth is the one that catches experienced engineers out. It is not a bug — it is the graceful degradation the team deliberately built — but during a run it converts an overloaded dependency into a fast, cheerful, entirely fictional success. ## Why load makes these more common, not less The pressure of the run is exactly the condition that triggers them: - **Fallbacks are designed to fire under stress.** The busier the system, the larger the share of answers that come from the cheap path instead of the real one. - **Caches serve stale copies for longer** when the refresh path is queueing behind everything else. - **Intermediaries substitute their own documents** when an upstream is slow, and they do it with a success status if they are configured to present a friendly page. - **Partial results become a deliberate strategy** — return what is ready, drop the rest — which is sound engineering and indistinguishable from a full answer if you only measure arrival. So the share of form-correct, content-wrong replies is at its highest precisely in the part of the run you most want to reason about. ## What it does to the numbers Three figures move, all in the reassuring direction: 1. **Success rate is overstated**, because false successes sit in the numerator. 2. **Throughput is overstated**, because a rejection produced in two milliseconds counts as one completed unit of work exactly like a real answer that took two hundred. 3. **Response times are understated**, because cheap wrong answers are fast and they pull the reported figures down. A run in this state does not look broken. It looks unusually good, which is why the finding is so often celebrated instead of investigated. ## The cheapest check worth making You do not need full functional assertions inside a performance run. You need one constant-cost check per reply that a false success cannot pass. In order of value: 1. **A required marker.** One field, header value or token that only the genuine path produces. A shared error page and a cached default both fail it. 2. **A length floor.** A number of bytes below which no real answer can be valid. Truncation and empty payloads both fall through it. 3. **A minimum count.** When the request is built from data you seeded, you know how many rows a correct answer contains; assert the floor, not the exact set. Whatever the check, its failures must land in the error tally rather than being logged and forgotten, and the report must say which check was applied. A run that asserts nothing beyond arrival is still useful — it measures whether the system accepted and answered demand at a rate — but it cannot claim that the work was actually done, and it should not be quoted as if it did.