Non-Functional Testing

Volume testing keeps the request rate fixed and grows the stored dataset instead: more rows, longer collections, larger files. It exposes defects that appear only at size, such as unbounded queries, memory that scales with result size, and jobs that outgrow their window.

An internationalization defect is in the product code: a string that was never extracted, a hardcoded date format, a container that cannot hold a longer word. A translation defect is in the text itself, wrong or missing wording.

Ramp-up is the stretch while offered load climbs to the target level; steady state is where that load is held constant; warm-up is the early portion where caches, pools and compiled code settle. Report the steady state only.

A support matrix is the published grid of platform combinations — operating system, version, engine, device class — where a product promises to work. Each listed cell is a commitment: a defect that reproduces there is one you owe a fix.

Write the expected degraded behaviour first: for each affected feature, what the caller gets, what the interface shows, which side effects stay allowed, and what must never happen. Without that written oracle, anything short of a crash looks like a pass.

Rows are actor classes, columns are operations, and each cell says allow or deny. A parameterized suite iterates the whole table, and every deny cell asserts a refusal, an unchanged state, no information leak and the required record of the attempt.

Service time is what one request costs with nothing competing for the resource; queueing delay is what waiting adds. Measure at a concurrency of one, then watch duration grow as concurrency rises — the growth is wait, not work.

Offered throughput is the demand the run applied; achieved throughput is the work that actually completed and was counted. A gap means requests were refused, expired or never finished, so the completed-work figure describes only part of the intended load.

In an open model the generator sends requests at a chosen arrival rate regardless of how the system responds. In a closed model a fixed population of simulated users each waits for a response, then thinks, then sends again, so slowness throttles the load.

"Degrades gracefully" must become outcome rules written before the run: which requests are still completed and inside what response-time bound, which may be refused, how fast a refusal must arrive, and which results fail outright.

Fix five things before the run starts: which measurement, at which point in its distribution, over which window, under which workload, and read from which vantage point. Anything left open is chosen afterwards to suit the numbers.

Percentiles are ranks over a set of samples, not quantities that add or average. The mean of per-interval figures weights a quiet minute like a busy one and throws away each interval's shape. Merge raw samples or bucket counts instead.

Comparable performance runs hold the same build, the same dataset size and shape, the same environment and its neighbours, the same warm state and the same applied workload. When one of those differs, a change in the numbers cannot be attributed.

Derive it from the build-up you expect to surface: the hold must be long enough for that accumulation to clear measurement noise and cover a readable fraction of the headroom. Calendar convenience decides nothing.

Recovery needs more than response times falling. The backlog must drain back to its pre-surge depth, resources must be released rather than settling higher, accepted work must still complete, and that state must hold across a defined observation window.

Fix three numbers before the run: the settled baseline arrival rate, the multiple applied to it, and the transition time over which the rate rises. Vary one per run, and hold the peak long enough to read a result.

A success status only says something answered the request. The body may be a rendered error page, a truncated document, or an empty result where a data record was expected, so a run reading only the status overstates how much work really succeeded.

A peak says how high a figure got; accumulation is a direction. Measure retained memory, open handles, cached entries and disk use as a slope across the hold, then compare that slope with a period expected to be level.

Generated rows tend to be uniform: equal group sizes, same-width text, no gaps, one format. Real data is skewed, with hot keys, long tails and rare variants. Uniform seeds hide the worst case, so the run measures an average production never has.

Pseudo-localization replaces every extracted string with a mechanically accented, padded, bracketed version of itself and runs the product against that fake locale. Anything still in plain source text was never extracted, and padded text exposes truncation.

A fixed pool caps throughput near its slot count divided by mean service time, so throughput flattens while duration rises with offered concurrency. Change the slot count: if the ceiling moves in proportion, the pool was the cap.

By asserting three things about every rejection: it comes back quickly rather than after a full wait, it reaches the caller as an explicit refusal it can act on, and it never counts as a success.

Attach the bound to a stated point in the distribution rather than a central measure, pair it with an outer cap on the worst reported interval, and say how many intervals may breach before the run fails.

An aggregate percentile ranks every request together, so weight follows request share, not importance. An operation carrying under 5% of requests can be entirely above the 95% cut and never move it. Report a distribution per operation type.

Measure the run-to-run spread first: repeat one unchanged configuration several times and record how much the compared figure varies on its own. A difference smaller than that spread is not evidence of anything.

Emulated environments borrow the host's processor, memory, storage speed and network, and they run a clean reference build of the platform. So they miss slow or thermally throttled hardware, vendor-modified system builds, real sensors and cameras, memory-pressure eviction, and real network transitions.

State the exact platform coordinates where it fails and the cells where the same steps passed, isolate one axis at a time so the report names the differing variable, give a reproduction rate, and name the failing cell's tier.

Failback catches everything the switch itself created: writes accepted on the standby that must reach the primary, a replication path that has to reverse, stale reads right after the return, clients still pinned to the old node, and configuration edited during the incident.

Seed two fully populated accounts with distinguishable data, then replay every recorded happy-path request with a session from the other account and with lower-privileged roles. Each must be refused, change no state, and leak nothing about the other account.