In a performance run, how do you count requests that expired, were reset, or were never sent?

seniorshould knowfreq 47%

answer

Failures are outcomes, not missing data
The denominator is requests issued
The slowest requests expire first
Censored at the deadline, never deleted

basics

~20 s

Count them as outcomes, not as missing data. Each stays inside the issued total, lands in a named failure category, and the report states what time was recorded for it. Deleting them describes only the requests the system managed to serve.

solid answer

~50 s

Every request the profile called for ends in exactly one named outcome, and those outcomes sum back to the issued total. Deadline expiries and transport failures are errors in their own categories; requests never sent are unmet demand rather than errors. Rates are computed over requests issued, never over replies received — deleting failures swaps the denominator and reports the success rate among the requests that succeeded. It is worse than it sounds, because expiry is not random: a request expires because it was slow, so removing them removes the worst observations and leaves a fast, unrepresentative remainder. An expiry is a censored observation — at least the deadline, exact value unknown — so record it as a bound and say so. And a harness that silently retries is quietly converting failures into successes while adding demand nobody planned.

Every request the profile called for ends in exactly one named outcome, and those outcomes sum back to the issued total. Deadline expiries and transport failures are errors in their own categories; requests never sent are unmet demand rather than errors. Rates are computed over **requests issued**, never over replies received — deleting failures swaps the denominator and reports the success rate among the requests that succeeded. It is worse than it sounds, because expiry is not random: a request expires because it was slow, so removing them removes the worst observations and leaves a fast, unrepresentative remainder. An expiry is a **censored** observation — at least the deadline, exact value unknown — so record it as a bound and say so. And a harness that silently retries is quietly converting failures into successes while adding demand nobody planned.

code

pseudocode · 17 lines

pseudocode

DEADLINE = 2000 ms

for each request in profile:
    if not sent:
        record(outcome = "never_issued", time = none)
        continue
    result = await_reply(request, DEADLINE)
    if result.expired:
        record(outcome = "expired", time = DEADLINE, censored = true)
    else if result.connection_failed:
        record(outcome = "transport", time = result.elapsed, partial = true)
    else:
        record(outcome = accept(result), time = result.elapsed)

# rates use issued as the denominator, always
success_rate = count("accepted") / issued
assert sum(count(o) for o in all_outcomes) == issued

go deeper

for a junior

Be ready to say that a request that timed out or failed still counts, and that the success rate is worked out over the requests that were sent rather than over the replies that came back.

for a middle

Explain the two distortions deletion causes: the denominator quietly changes to replies received, and because slow requests are the ones that expire, the surviving sample is systematically fast.

for a senior

Show the reporting discipline — one named outcome per issued request, counts that sum to the issued total, expiries recorded as bounds, retries counted as demand — and the instinct to distrust a near-perfect success rate.

for a principal

Own the convention itself: what a run is required to publish about censored observations and retries, so that results from different teams mean the same thing without anyone having to ask.

Three outcomes that are not missing data

A request that never produced an accepted reply still happened. Someone applied it, something either refused it or ignored it, and if it had been a real user that user waited and got nothing. The temptation is to treat it as a hole in the dataset — a sample that failed to record — and to leave it out. That is the single most common way a performance result is made to look better than the system it measured.

Outcome	What happened	How it is counted	Time recorded
Deadline expiry	The run stopped waiting; no reply arrived	An error, in its own category	At least the deadline, marked as a bound
Transport failure	Connection refused, or closed part-way through a reply	An error, in its own category	Time until the failure, flagged as partial
Abandoned before issue	The profile called for it and it was never sent	Unmet demand, not an error	None — excluded and declared

Keeping the three apart matters because they mean different things to a user. An expiry is someone who waited the full deadline. A reset part-way through a reply is someone who saw half an answer. An unissued request is demand the run never actually applied, and pretending otherwise overstates what was tested.

Why deleting them flatters everything at once

Two separate distortions arrive together.

The first is a denominator swap. Success rate should be computed over requests issued. Delete the failures and the denominator silently becomes replies received, at which point the run is reporting the success rate among the requests that succeeded — a number that tends towards a hundred percent by construction.

The second is survivorship. Expiry is not random. A request expires because it was slow, so the requests removed are precisely the worst observations in the run. What remains is the fast subset, and every figure computed from it describes a system that was never under the strain you applied.

Suppose 100,000 requests are issued with a two-second deadline. 92,000 reply within 400 milliseconds and 8,000 expire. Delete the expiries and the run reports a hundred percent success and a comfortable summary drawn from 92,000 fast samples. Count them and the run reports 92 percent success, with eight percent of users having waited two full seconds for nothing. The system behaved identically in both reports.

How to count them instead

Five rules, and they are all cheap:

The denominator is requests issued, never replies received. Every rate the run publishes uses it.
Every issued request ends in exactly one named outcome, and the outcome counts sum back to the issued total. If the sum does not close, the report is hiding something.
A failure keeps a recorded time where one exists, flagged for what it is. An expiry is a censored observation: you know the request took at least the deadline and you do not know what it would have taken. Recording it at exactly the deadline is a lower bound, not a measurement, and the report must say so.
Retries are extra demand, not second chances. A harness that silently retries turns a failure into a success and adds load the profile never planned. If the profile includes retries because real clients retry, count every attempt as issued demand and report the outcome of each.
State the convention. Whether expiries are recorded at the deadline or excluded from the timing figures changes the numbers materially, so the convention belongs beside the result rather than in someone's memory.

The honest report

The shape that survives scrutiny is a small table: requests issued, then a line per outcome, then the rates computed over the issued total, then one sentence naming the deadline and the convention used for censored observations. It is unglamorous, it is three lines longer than the version that quotes a single headline figure, and it is the difference between a result that can be defended and one that quietly assumes the failures away.

A useful instinct: whenever a run reports a success rate at or very near a hundred percent under demand you believed was aggressive, check the denominator before celebrating. It is far more often an accounting artefact than a system that never failed.

## Three outcomes that are not missing data A request that never produced an accepted reply still happened. Someone applied it, something either refused it or ignored it, and if it had been a real user that user waited and got nothing. The temptation is to treat it as a hole in the dataset — a sample that failed to record — and to leave it out. That is the single most common way a performance result is made to look better than the system it measured. | Outcome | What happened | How it is counted | Time recorded | |---|---|---|---| | Deadline expiry | The run stopped waiting; no reply arrived | An error, in its own category | At least the deadline, marked as a bound | | Transport failure | Connection refused, or closed part-way through a reply | An error, in its own category | Time until the failure, flagged as partial | | Abandoned before issue | The profile called for it and it was never sent | Unmet demand, not an error | None — excluded and declared | Keeping the three apart matters because they mean different things to a user. An expiry is someone who waited the full deadline. A reset part-way through a reply is someone who saw half an answer. An unissued request is demand the run never actually applied, and pretending otherwise overstates what was tested. ## Why deleting them flatters everything at once Two separate distortions arrive together. The first is a **denominator swap**. Success rate should be computed over requests issued. Delete the failures and the denominator silently becomes replies received, at which point the run is reporting the success rate *among the requests that succeeded* — a number that tends towards a hundred percent by construction. The second is **survivorship**. Expiry is not random. A request expires because it was slow, so the requests removed are precisely the worst observations in the run. What remains is the fast subset, and every figure computed from it describes a system that was never under the strain you applied. Suppose 100,000 requests are issued with a two-second deadline. 92,000 reply within 400 milliseconds and 8,000 expire. Delete the expiries and the run reports a hundred percent success and a comfortable summary drawn from 92,000 fast samples. Count them and the run reports 92 percent success, with eight percent of users having waited two full seconds for nothing. The system behaved identically in both reports. ## How to count them instead Five rules, and they are all cheap: 1. **The denominator is requests issued**, never replies received. Every rate the run publishes uses it. 2. **Every issued request ends in exactly one named outcome**, and the outcome counts sum back to the issued total. If the sum does not close, the report is hiding something. 3. **A failure keeps a recorded time where one exists**, flagged for what it is. An expiry is a *censored* observation: you know the request took at least the deadline and you do not know what it would have taken. Recording it at exactly the deadline is a lower bound, not a measurement, and the report must say so. 4. **Retries are extra demand, not second chances.** A harness that silently retries turns a failure into a success and adds load the profile never planned. If the profile includes retries because real clients retry, count every attempt as issued demand and report the outcome of each. 5. **State the convention.** Whether expiries are recorded at the deadline or excluded from the timing figures changes the numbers materially, so the convention belongs beside the result rather than in someone's memory. ## The honest report The shape that survives scrutiny is a small table: requests issued, then a line per outcome, then the rates computed over the issued total, then one sentence naming the deadline and the convention used for censored observations. It is unglamorous, it is three lines longer than the version that quotes a single headline figure, and it is the difference between a result that can be defended and one that quietly assumes the failures away. A useful instinct: whenever a run reports a success rate at or very near a hundred percent under demand you believed was aggressive, check the denominator before celebrating. It is far more often an accounting artefact than a system that never failed.

What does calling an expiry a censored observation actually change in how you report it?
It stops you from treating the deadline as the measured time. You know the request would have taken at least two seconds and you have no idea what it would have taken in the end, so any figure computed with the deadline substituted in is a lower bound on the true one. The report gives the count of censored observations and the bound, so a reader knows which direction the numbers are wrong in.
The profile deliberately includes retries because real clients retry. How do you keep the counting honest?
Count every attempt as issued demand, and record the outcome of each attempt separately from the outcome of the logical operation. Then the report can state both: what share of attempts failed, and what share of user-level operations eventually succeeded and after how many tries. Reporting only the second hides the load the retries added.
A run reports 99.99 percent success under demand you expected to hurt. What do you check?
The denominator first. A near-perfect rate under aggressive demand is far more often an accounting artefact than a system that never failed. Check that outcome counts sum to the issued total, that expiries and transport failures have their own categories, and that nothing is being retried into a success behind the scenes.

Dropping expired requests from the results is like surveying a restaurant's diners about the wait and skipping everyone who gave up and left before being seated.

saying these in an interview costs you the question

Drops expired requests as failed samples rather than counting them
Computes success rate over replies received instead of requests issued
Treats the deadline value as the measured time for an expiry
Folds expiries, resets and unsent requests into one failure bucket
Lets silent retries turn failed attempts into reported successes

In a performance run, how do you count requests that expired, were reset, or were never sent?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Three outcomes that are not missing data

Why deleting them flatters everything at once

How to count them instead

The honest report

▾common follow-ups3 questions

▾analogy~10 s

▾red flags5 flags

Three outcomes that are not missing data

Why deleting them flatters everything at once

How to count them instead

The honest report

answer

go deeper