skip to content

In BenchmarkDotNet, how do Baseline = true, [MemoryDiagnoser] and [Params] show that a rewritten CSV tokenizer is faster and allocates less?

level: middleimportance: should knowfreq 38%

answer

  1. one method is the yardstick
  2. Ratio is a mean of ratios
  3. allocation counts per operation
  4. collections per thousand operations
  5. a row for every parameter value

basics

~20 s

Mark the old tokenizer [Benchmark(Baseline = true)] so each row gets a Ratio against it, add [MemoryDiagnoser] for Allocated bytes and Gen 0/1/2 collections per 1,000 operations, and use [Params] to repeat both for several input sizes.

solid answer

~40 s

Put the old and the new tokenizer in one benchmark class and mark the old one `[Benchmark(Baseline = true)]`; the summary gains a `Ratio` column, the mean of the per-measurement ratios against the baseline, plus `RatioSD` for its spread. `[MemoryDiagnoser]` on the class adds Gen 0, Gen 1 and Gen 2 columns (collections per 1,000 operations) and `Allocated` (managed bytes allocated per operation), which is how you prove "allocates less"; it is not on by default. A `[Params(16, 256, 4096)]` property gives a row for each value, so you see whether the win holds for short and long lines. Build inputs in `[GlobalSetup]` so they are not timed, return a value from each method, and keep exactly one baseline in the class.

code

csharp · 40 lines
csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<CsvTokenizerBenchmarks>();

[MemoryDiagnoser]
public class CsvTokenizerBenchmarks
{
    private string _line = "";

    [Params(16, 256, 4096)]
    public int FieldCount { get; set; }

    [GlobalSetup]
    public void Setup() =>
        _line = string.Join(',', Enumerable.Range(0, FieldCount).Select(i => $"field{i}"));

    [Benchmark(Baseline = true)]
    public int SplitTokenizer()
    {
        int total = 0;
        foreach (string field in _line.Split(','))
            total += field.Length;
        return total;
    }

    [Benchmark]
    public int SpanTokenizer()
    {
        int total = 0;
        ReadOnlySpan<char> rest = _line;
        while (true)
        {
            int comma = rest.IndexOf(',');
            if (comma < 0) return total + rest.Length;
            total += comma;
            rest = rest[(comma + 1)..];
        }
    }
}

go deeper

for a junior

Recall that Baseline = true adds a Ratio column and that [MemoryDiagnoser] adds the Allocated column.

for a middle

Explain Ratio as a mean of per-measurement ratios, Allocated as managed bytes per operation, gen 0 collections per 1,000 operations, and why [Params] rows are read separately.

for a senior

Design the benchmark so the claim survives review: setup in [GlobalSetup], equal return values, several input sizes, and allocation numbers that explain the timing change.

for a principal

Use allocation per operation as the deterministic part of a team's performance evidence, and decide which input sizes represent production traffic.

## The question a review asks "The new tokenizer is faster and allocates less" is two claims, each relative to the old code and each dependent on input size. BenchmarkDotNet expresses all three parts with attributes from `BenchmarkDotNet.Attributes`: - **`[Benchmark(Baseline = true)]`** marks the yardstick method. - **`[MemoryDiagnoser]`** turns on allocation and collection columns. - **`[Params]`** or **`[ParamsSource]`** repeat every benchmark for several inputs. ## Baselines and the Ratio column `BenchmarkAttribute` has a `Baseline` property. Mark the current implementation with it and every other benchmark in the class is shown relative to it: - **`Ratio`** is the mean of the ratio distribution: BenchmarkDotNet divides each measurement of a method by the matching measurement of the baseline and averages those ratios. The docs stress that this is close to, but not the same as, dividing the two means. The column used to be called `Scaled`. - **`RatioSD`** is the standard deviation of that ratio distribution; BenchmarkDotNet adds it in complex situations, such as a baseline spoiled by an outlier. A ratio of 0.40 with a large `RatioSD` is a weaker claim than it looks. - The baseline itself reads `1.00`. Only one baseline per class is allowed: the mandatory `BaselineValidator` fails the run when more than one benchmark in a class has `Baseline = true` (benchmark categories are the documented way to have a baseline per group). ## Allocations with [MemoryDiagnoser] The memory diagnoser is built in and cross-platform, but the docs say it **is not enabled by default anymore**. Apply `[MemoryDiagnoser]` to the class (or pass `-m` / `--memory` on the command line) and the summary gains: | Column | Meaning | |---|---| | Gen 0, Gen 1, Gen 2 | collections of that generation per 1,000 operations | | `Allocated` | managed memory allocated per single operation, inclusive, with 1 KB = 1024 B | `Allocated` counts **managed** allocations only, so native memory is outside it. The attribute's constructor parameter `displayGenColumns` is true by default; `[MemoryDiagnoser(false)]` hides the generation columns when you only care about bytes. A drop to zero bytes per operation is the strongest form of "allocates less". ## Parameters `[Params]` goes on a public field or property and takes compile-time constants; BenchmarkDotNet produces results for every combination of parameter values. When values are computed or numerous, `[ParamsSource(nameof(Sizes))]` points at a public member that returns an `IEnumerable`. Method arguments use `[Arguments]` instead. Parameters matter because a rewrite often wins on long lines and loses on short ones, where a fixed setup cost dominates. Read the table per parameter value, not as one average. ## Setup that stays out of the measurement 1. Build the input in a `[GlobalSetup]` method; it runs once per benchmark method after parameters are set and before any invocations, so building the line is not timed. 2. Return a value (here, the total field length) from each benchmark so the JIT cannot discard the work. 3. Avoid `[IterationSetup]` for nanosecond methods: it forces one invocation per iteration and the docs warn it can spoil microbenchmark results. 4. Check that both implementations return the same value; a faster method that computes something else proves nothing. The optional `ReturnValueValidator` checks that non-void benchmarks return equal values. ## Jobs and runtimes A job describes how and where the benchmark runs. Attributes such as `[ShortRunJob]` trade accuracy for speed, and `[SimpleJob(RuntimeMoniker.Net80)]` adds a job for a specific runtime, so one class can compare the same code on two runtimes. In a pull request, keep the default job: it is the one whose statistics you trust. ## Reading the rows for a tokenizer For each `FieldCount`, read `Mean`, `Ratio` and `Allocated` together. A `string.Split`-based tokenizer allocates an array plus one string per field, so its `Allocated` grows with the field count; a tokenizer that walks the line with `IndexOf` and returns only lengths allocates nothing. Measured outside the harness on .NET 10 with `GC.GetAllocatedBytesForCurrentThread()`, the split version allocated 792, 14,584 and 227,448 bytes for 16, 256 and 4,096 fields and the other version 0; the exact figures depend on the runtime build, the shape does not. If the new code wins on long lines but its `Ratio` is above 1.00 for 16 fields, the review should hear that too. ## Reading the result honestly - A lower `Ratio` with overlapping error intervals is not yet a win; rerun or add a statistical test column. - `Allocated` falling from kilobytes to zero is deterministic for a given input, so it is usually the easiest part of the claim to prove. - Fewer gen 0 collections per 1,000 operations is the downstream effect of fewer allocations, not a separate improvement.

  • When should you use [ParamsSource] instead of [Params] in BenchmarkDotNet?
    `[Params]` values must be compile-time constants. When values are computed, numerous or not constants (a list of sample files, a set of generated inputs), `[ParamsSource]` names a public field, property or method returning an `IEnumerable`, which may be static or instance and may live in another type if you pass that type. Either way you get one row per value.
  • Why can Allocated be identical across runs while Mean moves by several percent?
    Managed allocation per operation is determined by the code path for a given input, so it repeats exactly; timing depends on CPU frequency, caches, other processes and the GC. That makes `Allocated` the stable half of a claim and the `Mean` and `Ratio` the half that needs error bars and repeated runs.

  • Ratio is simply the new method's Mean divided by the baseline's Mean.
  • The memory diagnoser runs by default, so allocation columns always appear.
  • Allocated includes native memory the code obtains through interop.
  • Building the input inside the benchmark method is fine because it is identical for both versions.
  • You can mark several methods as baseline in one class to compare each pair.