Non-Functional Testing

Run the same fixed workload against several dataset sizes — today's, four times, sixteen times — and compare the shape of the curve rather than one pass or fail. Anything whose cost rises faster than the data is the finding.

Drive the run from an injected clock, not the machine clock, and enumerate boundary dates deliberately: the repeated local hour, the missing one, month ends, fractional offsets. Assert one charge per account per period.

A client's clock and the system's own clock cover different intervals. A gap that does not change with load is fixed transport cost outside the handler. A gap that widens with load is waiting in a segment nobody times.

Per-step averages hide a step that is slow occasionally or runs many times per request. Compare each step's share of total time in the slowest requests against median ones, then confirm by neutralising the suspect.

Count them as outcomes, not as missing data. Each stays inside the issued total, lands in a named failure category, and the report states what time was recorded for it. Deleting them describes only the requests the system managed to serve.

Because a generator that waits for each response stops sending while the system is stalled, so the requests that would have been slowest are never issued and never measured. This is coordinated omission: the missing samples are exactly the bad ones.

Compare achieved load against offered load and instrument the generator hosts: processor use, memory pressure, port and descriptor exhaustion, network saturation. If the achieved rate falls short while the system under test sits idle, the rig is the limit.

Three results fail regardless of timing: work applied in part and left inconsistent, a request accepted and lost with nobody told, and a queue or in-flight set that grows with no ceiling. These are damage, not slowness.

Shedding holds completed work near the sustainable level, answers rejections in milliseconds, keeps survivors near normal speed, and holds backlog and memory flat. Collapse shows completed work falling, everyone waiting, and a backlog that keeps growing.

Description only: what was applied, where timings were taken, and where the numbers sat. Not a pass or a fail - a bound chosen once results are visible is fitted to them. Its real output is the next run's rule.

Each shape asks a different question. A long hold must still meet the bound in its final intervals; an abrupt increase in arrival rate is judged on recovery time; a run past the intended rate is judged on refusal, not latency.

It implies one population with one centre. With two clusters the figure lands in whichever cluster holds the rank, describes only that one, and moves with the proportion between them rather than with either cluster's own speed.

Summing per-hop 99th percentiles prices a request in which all four hops were simultaneously in their own slowest one percent, which is rare — so the sum usually overstates. Correlated hops can make it understate. Measure whole-path time per request instead.

Derive it from the measured run-to-run spread: set the alarm level a multiple above the spread of repeats, per metric, then check that size against what change actually matters. A round percentage chosen by habit is either noise or nothing.

Check whether the climb advances with work done or with elapsed time. Replot it against cumulative completed work, run a low-rate control, probe a path the run never writes to, and restart the process keeping its data.

Start at the size the system would reach under baseline demand, never pre-raised to peak, and timestamp two series: offered demand and capacity actually serving. The interval between them is the exposure window, and what happened inside it is the result.

Tag every submitted unit with a unique identifier before demand rises, then reconcile after the observation window: submitted equals completed plus refused plus pending. Separately compare distinct completed identifiers against total completions, because totals alone hide loss and repetition together.

Keep saved data written by every supported release, upgrade each one and assert the content afterwards, then reverse it: install the previous release over the upgraded data and check the older build reads it without discarding what it cannot interpret.

Kill deliberately at the boundaries between irreversible steps, restart, then judge by invariants over the stored data: acknowledged operations minus applied ones must be empty, nothing applied twice, no orphaned rows, and cross-store totals reconcile.

Derive the transaction mix, arrival shape and data profile from observed traffic rather than opinion, state the reference period, and keep the model a versioned, owned artefact that is re-derived on a schedule instead of ageing into fiction.

It is legitimate when the shift in level has an identified, intentional cause, was reproduced across repeats, and still meets the obligation the team owes. It is a moved goalpost when the reason is that meeting the old figure became inconvenient.

Weigh your own usage telemetry and the revenue behind it against the cell's cost, honour stated commitments and the vendor's own support dates, then drop with announced lead time, a pinned last-supported release, and a check that the tail migrated.

Treat the three as separate cases. Expiry needs a controllable clock rather than a sleep, revocation needs a stated bound on how long access may survive withdrawal, and tampering means altering one field and asserting refusal rather than silent acceptance.

Build a golden ordered fixture per locale containing accents, case variation, a locale-specific digraph and punctuation, and assert the full order. Then assert every layer uses the same comparator, and that composed and decomposed spellings compare equal.

Rehearse it against a volume-representative copy, measure the rate chunk by chunk instead of only the total, and extrapolate with the non-linear effects stated. Then set a go/no-go checkpoint during the real run and practise the abort.

Vary overlap alone. Run the same arrival rate with few busy clients and then with many idle ones: if duration differs, overlap is the cause. Repeat with requests spread across distinct records to locate what is shared.

Only checks cheap enough to fit inside the applying side's own budget: a length floor, a required field, a marker. Deep parsing per reply steals the capacity that generates demand, so sample it instead and report what share was verified.

The read-out locates the bucket holding the ranked request, not its time, so the figure carries that bucket's width as uncertainty. Interpolating inside assumes a spread the slow end does not have, and an unbounded top bucket reports nothing at all.

Decide from what the hold must prove. Arrange the run to hit periodic work - scheduled jobs, credential rotation, cache expiry, index maintenance - when that interaction is the risk; otherwise disable it and record that the trend excludes it.

Compare what the load generator offered against what the service received: the offered rate is known and unchanged, so any excess was manufactured inside the system. Repeats of identifiers already submitted, arriving synchronised at a fixed delay, confirm it.