skip to content

In NUnit 5, how do [Values], [Range], [Combinatorial], [Sequential] and [Pairwise] generate test cases, and how many does each produce?

level: middleimportance: nice to knowfreq 28%

answer

  1. attributes on parameters, not rows
  2. empty [Values] on bool or enum
  3. Range includes both ends
  4. the default strategy is the full grid
  5. all pairs, not all combinations

basics

~20 s

[Values] and [Range] put a value list on each parameter, and a combining attribute joins the lists: [Combinatorial] (the default) produces every combination, [Sequential] zips the lists, and [Pairwise] covers every pair of values with far fewer cases.

solid answer

~40 s

Instead of rows, these attributes describe each **parameter**: `[Values(1, 2, 5)]` lists values, an empty `[Values]` on a `bool` yields `true` and `false` and on an enum every member, and `[Range(1, 5)]` yields 1 to 5 inclusive (with an optional step). A method-level strategy then joins the lists. `[Combinatorial]` is the default when none is given and produces the Cartesian product, so 3 zones x 2 express flags x 5 weights is 30 cases. `[Sequential]` takes the first value of each list, then the second, and so on, padding shorter lists with `null`. `[Pairwise]` produces a smaller set in which every pair of values from any two parameters appears at least once, which keeps large grids affordable.

code

csharp · 23 lines
csharp
public class ShippingInvariantTests
{
    private readonly ShippingCostCalculator _calculator = new();

    [Test]
    public void Cost_is_always_positive(
        [Range(1, 5)] int weightKg,
        [Values] Zone zone,
        [Values] bool express)
    {
        // 5 x 3 x 2 = 30 cases under the default [Combinatorial] strategy
        var cost = _calculator.Calculate(weightKg, zone, express);
        Assert.That(cost, Is.Positive);
    }

    [Test, Pairwise]
    public void Quote_has_a_carrier(
        [Values] Zone zone, [Values] bool express,
        [Values] PackageType package, [Values] Insurance insurance)
    {
        Assert.That(_calculator.Quote(1m, zone, express, package, insurance).Carrier, Is.Not.Null);
    }
}

go deeper

for a junior

Recall that [Values] and [Range] sit on parameters and that NUnit crosses them into every combination unless told otherwise.

for a middle

Explain the three strategies, count the cases each produces, and know what an empty [Values] supplies for bool and enum parameters.

for a senior

Use [Pairwise] to keep a growing parameter grid affordable, and say plainly which interaction defects that trade-off can let through.

for a principal

Set budgets for generated grids in the suite and decide where all-pairs coverage is acceptable and where the full product is required.

## Parameter-level data instead of rows `[TestCase]` and `[TestCaseSource]` describe **whole rows**. NUnit's other model describes **each parameter separately** and lets a strategy build the rows. That suits rule engines where the question is "does the rule hold for every zone, every speed and a spread of weights?" The parameter attributes: | Attribute | Values it supplies | |---|---| | `[Values(a, b, c)]` | exactly the listed values, converted to the parameter type | | `[Values]` on `bool` | `true`, `false` | | `[Values]` on `bool?` | `null`, `true`, `false` | | `[Values]` on an enum | every member of the enum (plus `null` for a nullable enum) | | `[Range(from, to)]` | from..to **inclusive**, step 1 (or -1 if `from > to`) | | `[Range(from, to, step)]` | from..to inclusive in steps; `double` ranges always need a step | | `[Random(...)]` | random values in a range | | `[ValueSource(nameof(Weights))]` | values from a static member | `[Range]` validates the step: a step that points away from `to` (for example `[Range(1, 5, -1)]`) is rejected. ## The combining strategies A method with data on its parameters needs a **combining strategy** to turn the per-parameter lists into test cases. 1. **`[Combinatorial]`** - the Cartesian product of all lists. It is the **default**: if no strategy attribute is present, NUnit adds `[Combinatorial]` itself. 2. **`[Sequential]`** - zips the lists: case 1 takes each list's first value, case 2 each list's second value, and so on until the longest list is used up; shorter lists are padded with `null`. 3. **`[Pairwise]`** - an all-pairs strategy based on the "jenny" algorithm. It guarantees that **every pair of values from any two parameters** appears in at least one case, using far fewer cases than the full grid. ## Counting the cases For `Calculate(int weightKg, Zone zone, bool express)` with `[Range(1, 5)]`, `[Values]` on a three-member `Zone` enum and `[Values]` on `bool`: | Strategy | Cases | Why | |---|---|---| | `[Combinatorial]` (default) | 30 | 5 x 3 x 2 | | `[Sequential]` | 5 | the longest list has 5 values; `zone` and `express` run out and get `null` | | `[Pairwise]` | at least 15 | the 5 x 3 weight-zone pairs alone need 15 cases; express values are folded in | Two practical notes on those numbers: - **Sequential padding** with `null` only works for nullable parameters. A non-nullable `Zone` or `bool` receiving `null` fails that case, so `[Sequential]` is for lists of equal length. - **Pairwise** does not promise a fixed count, only full pair coverage. The exact number depends on the generator, so do not assert on it. ## Choosing a strategy - Use the **default combinatorial** grid when the product is small and every combination matters, for example three zones and two speeds. - Use **`[Pairwise]`** when the grid explodes: six parameters with five values each is 15,625 combinations, while all-pairs coverage needs a small fraction of that. Defects caused by the interaction of two parameters are still caught; a defect that needs three specific values at once may not be. - Use **`[Sequential]`** when the lists are really columns of a table, though `[TestCase]` rows usually say that more clearly. - Do not mix in `[TestCase]` rows and expect them to be crossed with `[Values]`; each data attribute produces its own cases. ## Pitfalls in practice - **Grids grow silently.** Adding a fourth member to a three-member `Zone` enum adds a third more cases to every test that uses an empty `[Values]` on it. That is usually desired (new zones are tested automatically), but it also lengthens the run. - **Names get long.** Every generated case carries all its argument values in its name; for wide grids, failures are easier to read when the test asserts with a clear message. - **Floating-point ranges.** A `double` range such as `[Range(0.1, 0.5, 0.1)]` is built by repeated addition, so it can produce values like 0.30000000000000004; prefer integer ranges and convert inside the test when exact values matter. - **Data that is not uniform.** If some combinations are invalid (express is not offered internationally), a grid generates cases that must be skipped or asserted differently. Explicit `[TestCase]` rows for the exceptions, and a grid for the rule that holds everywhere, read better than conditionals inside one test. ## NUnit's [Theory] is something else NUnit also has a `[Theory]` attribute, but it is not a row-data attribute. It works with `[Datapoint]` and `[DatapointSource]` members and `Assume.That`, testing that a property holds for every applicable data point. For a grid of parameter values, use the attributes above.

  • What does an NUnit [Values] attribute with no arguments supply?
    It generates values from the parameter type: `true` and `false` for `bool`, `null`, `true` and `false` for `bool?`, every member for an enum, and those members plus `null` for a nullable enum. For other types it supplies nothing, so explicit values are needed.
  • Is NUnit's [Range(1, 5)] inclusive of 5?
    Yes. NUnit's ranges include both ends, so `[Range(1, 5)]` yields 1, 2, 3, 4 and 5. The default step is 1 (or -1 when `from` is greater than `to`), and a step pointing away from the end is rejected.
  • Why might a defect survive an NUnit [Pairwise] test?
    Pairwise only guarantees that every pair of values appears together at least once. A defect that needs a specific combination of three or more values may never be generated. Use the full combinatorial grid for the few parameters whose interaction is known to matter.

  • Without [Combinatorial] NUnit uses only the first value of each list
  • [Range(1, 5)] stops at 4 because the end is exclusive
  • [Pairwise] runs every combination, just in a different order
  • [Sequential] crosses every value of each list with every other
  • An empty [Values] attribute on an int supplies a default range
  • NUnit's [Theory] is the attribute for a grid of parameter values