When is a pairwise-only configuration suite unsafe, and what do you add to close the gap?

## The guarantee, stated precisely A strength-2 covering array guarantees exactly one thing: every combination of values from any two parameters occurs in some row. Everything a candidate wants to believe beyond that — that three-value interactions get "mostly" covered, that the set is representative, that unusual configurations are included — is outside the guarantee. Senior judgement here is mostly the discipline of not extending the promise. ## Failure mode one: interactions above strength 2 The classic worked example. In the quote engine, quotes for the umbrella product line, on the monthly cadence, in rating region R3 hit a **partial-failure rollback**: the pricing write succeeds, the downstream document write fails, the compensating rollback misses the pricing row, and an orphaned quote survives with no document. Any *two* of those three values are perfectly healthy. The 43-row covering array contained umbrella+monthly, umbrella+R3 and monthly+R3 — each in a different row — and passed cleanly for four consecutive 3-week release trains before a customer reported a quote with no paperwork. This is not a defect in the array; it is the guarantee working as documented. The mitigation is not to distrust pairwise but to be explicit that strength-2 buys strength-2. Where three-way exposure is plausible — anything touching partial writes, compensations, caching or feature interaction — raise strength on a narrow sub-model of the parameters actually implicated, and leave the rest at 2. Going to strength 3 across all six parameters would have taken the set from 43 rows to several hundred; strength 3 across the three parameters that touch the write path cost a couple of dozen. ## Failure mode two: order and history A covering array is a set of static configurations. It has no notion of "the second attempt after a timeout" or "renew, then endorse, then cancel". Faults that live in sequence — stale state carried between operations, a retry that double-applies a discount — are structurally outside it, and no increase in strength reaches them. That material belongs to a different design technique and should be planned as a separate body of cases rather than hopefully bolted onto the configuration run. ## Failure mode three: obligations attached to named configurations Sometimes the requirement is not statistical coverage but evidence for a specific configuration: an audit that names the combinations in scope, a large broker whose exact setup must be shown working before a release, a production configuration you have already burned once. A generated array might happen to include those rows, and might drop them on the next generation. The fix is **seeding**: supply those configurations as mandatory rows before generation. A good generator includes them verbatim, counts the pairs they already cover, and generates only what is still missing — so seeding usually *reduces* total rows while making the guarantee explicit. Seed the last few production-escaped configurations too; they cost almost nothing and they answer "would we catch that again?" with a row rather than an opinion. ## Failure mode four: the oracle, not the array The most common real-world failure is not the array at all. A configuration suite that asserts only that the run completed will happily pass 43 rows while a whole product line prices incorrectly. The array chooses inputs; something else has to decide whether each outcome was right. That may be a reference calculation, an invariant that must hold for any configuration, a comparison against the previous release's output for the same row, or explicitly stated expected values for the seeded rows. Any row whose only assertion is the absence of a crash is a row that will not repay its run time. ## Two secondary traps **Instability across runs.** If you regenerate the array on every pipeline run, a failing row may not exist next run, and week-to-week comparison collapses. Generate deliberately, commit the array, and regenerate when the model changes — treating the generated set as a versioned artefact rather than as pipeline output. **Values chosen badly.** The array reasons only over the values you supplied for each parameter. If those values are not representative of the parameter's real behaviour, a perfectly covered array tests the wrong thing thoroughly. The array's quality is capped by the model's, and the model is a human judgement call made before generation ever starts. ## How to answer the interview question Name the guarantee, name the categories it excludes, and attach a concrete remedy to each rather than concluding "so pairwise is unreliable". The strong answer is proportionate: keep strength 2 as the default because it is cheap, spend the extra strength where a mechanism makes higher-order interaction plausible, seed what must be evidenced, and put the real effort into what each row asserts.