What is the difference between a covering array and an orthogonal array in pairwise test design?
answer
- both cover every two-way combination
- one requires at least once
- the other requires equally often
- balance needs a rigid shape
- equal counts serve effect estimation
basics
~20 sA covering array requires every pair of values to appear at least once. An orthogonal array additionally requires every such pair to appear the same number of times. Balance costs rows, and testing rarely needs it.
solid answer
~50 sBoth are row sets over the same parameters, and both cover every 2-way combination; the difference is **balance**. In an orthogonal array each pair of values from any two parameters appears an identical number of times, which is what makes the design useful for statistical experiments where you want to estimate each factor's effect without confounding. A covering array demands only *at least once*. That relaxation matters practically: orthogonal arrays exist only for particular combinations of parameter counts and value counts, so an awkward model has to be padded with dummy values or forced to a uniform level count, which inflates the row count. Covering arrays tolerate mixed value counts and constraints, and are typically smaller. For finding interaction faults, appearing once is sufficient, so covering arrays are the normal choice and orthogonal arrays are the specialised one.
go deeper
It is enough to remember the one-line distinction: a covering array needs every pair at least once, an orthogonal array needs every pair the same number of times. Nobody will fault you for not going further.
Be able to say why balance costs rows — the rigid shape requirement, mixed value counts needing padding — and why the extra repetitions buy nothing when the goal is detecting an interaction fault.
Show you know when balance is the right property: separating factor effects in a measurement study rather than detecting a defect. Also note that constraints and seeded rows destroy balance outright.
Own the framing that this is a detection-versus-estimation choice, and resist tool claims of orthogonality on models with mixed levels or exclusions, where the property cannot hold whatever the marketing says.
## Two structures, one criterion apart Both artefacts are tables: one column per parameter, one row per test configuration, each cell holding one value of that column's parameter. Both satisfy 2-way coverage. They diverge on a single extra requirement. A **covering array** of strength 2 requires that for every pair of columns, every combination of one value from each appears in **at least one** row. Nothing constrains how often. An **orthogonal array** of strength 2 requires that for every pair of columns, every such combination appears in **exactly the same number** of rows. That constant is the array's *index*; an index of 1 means every pair appears exactly once. This property is what "orthogonal" names: the columns are statistically independent of one another, so the effect of one factor is not confounded with the effect of another. ## Why balance costs rows Balance is a strong structural demand. For a 2-way orthogonal array of index 1 over parameters with v values each, the row count must be a multiple of v squared, and such arrays exist only for particular parameter counts and value counts — they are constructed from algebraic objects, not searched for greedily. Real models are lopsided. The quote-engine model has value counts 4, 3, 7, 2, 3 and 5. There is no orthogonal array shaped like that. To use one you must inflate every parameter to a common level count, filling the extra slots with repeated or dummy values, then discard or re-map the surplus. The result covers all the pairs you wanted, plus a lot of redundancy: a mixed-level orthogonal design for this model runs to dozens more rows than the 43 a covering-array generator produced, and some of those rows test the same pair for the third time. Covering arrays have no such shape requirement. They accept mixed value counts natively, accept constraints that forbid combinations, and accept **seeded** rows you insist on including — none of which a balanced design tolerates, because every one of them breaks balance. ## When balance is actually worth paying for Balance is not a superstition; it is the right property for a different job. If you intend to *measure* rather than merely *detect* — comparing mean response time across product lines while several other factors also vary, and attributing the difference to the product line rather than to a correlated channel — then equal appearance counts are what let you separate the effects. That is the design-of-experiments use, and it is why the structures were studied long before anyone applied them to software configuration. Functional interaction testing does not need it. A defect that fires when umbrella cover meets the monthly cadence fires the first time that pair is exercised; exercising it three more times, evenly, adds run time and no information. Since a covering array is smaller and more flexible, it wins for this purpose, and modern generation tooling in the testing literature produces covering arrays. ## The vocabulary trap The terms get used loosely, and "orthogonal array testing" is sometimes written as a synonym for pairwise testing generally. In an interview, state the distinguishing property rather than arguing about the label: *covering means at least once, orthogonal means equally often*. If someone claims their pairwise tool produces orthogonal arrays, the checkable question is whether their model has mixed value counts or constraints — if it does, the output is a covering array, whatever it is called, because balance cannot survive either. A last nuance worth knowing: every orthogonal array of strength 2 is also a covering array of strength 2, since appearing exactly k times implies appearing at least once. The reverse is almost never true. So an orthogonal array is a valid but usually oversized answer to the pairwise problem.
- Is every orthogonal array also a covering array?Yes, at the same strength. If every pair appears exactly k times with k of at least one, then every pair appears at least once, which is precisely the covering criterion. The converse fails almost always: a covering array makes no promise about how often a pair recurs, and generated ones are deliberately uneven because unevenness is what lets them be small.
- Your model has mixed value counts and two exclusion rules. Which structure can you use?A covering array. Balance requires a rigid relationship between value counts and row counts, and both mixed levels and exclusions break it — an excluded combination cannot appear at all, let alone the same number of times as its peers. You could force an orthogonal design by padding parameters to a common level count and dropping infeasible rows, but the padding inflates the set and the dropping destroys the balance you paid for.
saying these in an interview costs you the question
- Uses orthogonal and covering array as interchangeable terms
- Thinks balance makes an array better at finding defects
- Claims an orthogonal array exists for any set of parameters
- Believes a covering array must cover each pair exactly once
- Pads parameters with dummy values without noticing the cost