skip to content

In .NET, why does timing a method with a Stopwatch loop give misleading numbers, and what does BenchmarkDotNet do instead?

level: juniorimportance: should knowfreq 42%

answer

  1. the first call pays extra
  2. the code changes while you time it
  3. an unused result may vanish
  4. one sample is not a distribution
  5. separate Release process, pilot, warmup

basics

~20 s

A Stopwatch loop folds JIT compilation, tier changes, collections and noise into one number, and an unused result may be optimized away. BenchmarkDotNet runs a Release build in its own process, warms up, subtracts its overhead and reports statistics.

solid answer

~50 s

The first call to a method includes the JIT compiling it, and with tiered compilation the runtime later swaps the quick tier-0 code for optimized tier-1 code (the release runtime's default threshold is 30 calls), so a short loop times a mix of code versions. A garbage collection caused by earlier allocations can land inside the timed region, and if the loop discards the method's result the JIT may remove work whose result is never used. A Debug build or an attached debugger adds unoptimized code on top. BenchmarkDotNet generates and builds a Release project, runs each benchmark in a separate process, uses a pilot stage to choose how many invocations an iteration needs, measures and subtracts its own harness overhead, warms up, and then reports `Mean`, `Error` and `StdDev` over many iterations. You return the computed value from the `[Benchmark]` method so it is used rather than eliminated.

code

csharp · 24 lines
csharp
using System.Diagnostics;
using System.Globalization;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
string row = string.Join(',', Enumerable.Range(0, 200));

var sw = Stopwatch.StartNew();
int first = CountFields(row);                 // includes JIT-compiling CountFields
sw.Stop();
Console.WriteLine($"first call: {sw.Elapsed.TotalMicroseconds:F1} us");

sw.Restart();
for (int i = 0; i < 1_000; i++) CountFields(row);   // result discarded
sw.Stop();
Console.WriteLine($"next 1000 : {sw.Elapsed.TotalMicroseconds / 1_000:F2} us per call");
Console.WriteLine($"fields    : {first}");

static int CountFields(string line)
{
    int fields = 1;
    foreach (char c in line)
        if (c == ',') fields++;
    return fields;
}

go deeper

for a junior

Know that the first call includes JIT compilation, that benchmarks run in Release without a debugger, and that the benchmark method should return its result.

for a middle

Explain tier-0 versus tier-1 code during a measurement, dead-code elimination, and BenchmarkDotNet's pilot, overhead, warmup and workload stages.

for a senior

Spot a broken measurement in review: a Debug build, an [IterationSetup] on a nanosecond method, a deferred query returned, or a timing that ignores collections.

for a principal

Decide which claims need a harness at all; a Stopwatch log line around a seconds-long job is fine, while a nanosecond claim without a harness is not.

## What a Stopwatch loop really measures `System.Diagnostics.Stopwatch` is an accurate clock. The problem is what sits between `Start` and `Stop` when you wrap a loop around a method and divide by the iteration count. Several effects that have nothing to do with your algorithm end up inside that number: - **JIT compilation on the first call.** A .NET method starts as IL; the first call compiles it to machine code, so that call is far slower than the rest. In one run on .NET 10, the first call to a small parsing method took about 40 µs and each later call about 2 µs; the numbers depend on the machine, the gap does not. - **Tiered compilation.** The first code the JIT produces is quick, lightly optimized tier-0 code. After a method has been called enough times (30 calls by default in the release runtime, after a short startup delay) it is recompiled in the background as optimized tier-1 code. A loop of a few hundred iterations can time both versions. The mechanics belong to the material on the JIT and tiered compilation; for measurement, the point is that **the code under test changes during the test**. - **Garbage collections.** Allocations made before or during the loop can trigger a collection that pauses your thread inside the timed region. - **Dead-code elimination.** If the loop throws the result away, the optimizing JIT is allowed to remove computation whose result is never used, and you time an empty loop. - **Debug builds and debuggers.** A Debug build disables optimizations, and an attached debugger changes how code is generated. BenchmarkDotNet's guide says the debug version of a method can run 10 to 100 times slower. - **Noise.** Other processes, CPU frequency changes and timer resolution all move a single measurement, and one number has no error bar. ## What BenchmarkDotNet does instead BenchmarkDotNet is an open-source benchmarking library for .NET. Its "how it works" guide describes these steps: 1. It **generates an isolated project** for each runtime setting and builds it in Release mode. 2. It **launches a separate process** per benchmark (once by default), so your test host, debugger and earlier work do not leak in. 3. A **Pilot** stage doubles the invocation count until one iteration is long enough to measure reliably. 4. **OverheadWarmup** and **OverheadWorkload** stages time an empty method with the same signature, so the harness cost can be subtracted. 5. **ActualWarmup** runs your method until it settles, which covers JIT compilation and tiering. 6. **ActualWorkload** iterations are the real measurements; the result is the workload time minus the median overhead. 7. The summary reports `Mean`, `Error` (half of the 99.9% confidence interval) and `StdDev`, so you see the spread, not one number. ## The guard rails it adds - If the assembly that defines the benchmarks was compiled without optimizations, it reports: `Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.` and the benchmarks do not run until you rebuild in Release. - Its `EnvironmentAnalyser` warns in the summary when the application was built in Debug mode or ran with an attached debugger. - It warns when a benchmark returns a deferred `IEnumerable` or `IQueryable`, because you would time the creation of a query rather than its execution. - Returning the computed value from the `[Benchmark]` method is the documented way to avoid dead-code elimination. ## Side by side | Concern | Stopwatch loop | BenchmarkDotNet | |---|---|---| | JIT and tiering | timed together with your code | absorbed by warmup stages | | Harness cost | included | measured and subtracted | | Build configuration | whatever you ran | generated Release project | | Result | one number | mean, error, standard deviation | | Isolation | same process as everything else | separate process per benchmark | ## When a Stopwatch is still fine A `Stopwatch` around a long operation (a file import that takes seconds) is a reasonable log line, because JIT and harness costs are tiny compared with the work. For comparing two implementations of a hot method, or for any claim about nanoseconds or microseconds, use a benchmark harness. Production latency is a separate job for metrics and traces, not for either tool.

  • Why is returning the result from a [Benchmark] method enough to stop dead-code elimination?
    The JIT may remove a computation only when nothing observes its result. When the `[Benchmark]` method returns the value, BenchmarkDotNet's generated harness receives it, so the work is observable and stays. That is why its good-practices guide says to return `Math.Exp(1)` rather than call it and ignore it. A method that returns a lazy `IEnumerable` is different: return a materialized result or consume it, or you time only query creation.
  • When does [IterationSetup] spoil a BenchmarkDotNet microbenchmark?
    A method marked `[IterationSetup]` runs before each iteration, and BenchmarkDotNet then forces `UnrollFactor=1` and `InvocationCount=1`, so every measured iteration is a single invocation. For a method that takes nanoseconds, timer resolution and per-invocation overhead dominate. The docs advise against it for microbenchmarks; prepare inputs once in `[GlobalSetup]` instead.

Timing a Stopwatch loop is like timing a new commute on the first day: the first trip includes finding the way, later trips get faster as you learn shortcuts, and one trip says nothing about how much traffic varies. A benchmark harness drives the route many times after practice runs and reports the average and the spread.

  • Running the loop a million times and dividing removes every source of bias.
  • A Debug build is fine as long as both methods are measured the same way.
  • The first call to a method shows its steady-state speed.
  • Discarding a computed result cannot change what the JIT compiles.
  • By default BenchmarkDotNet times the method inside your own process and build.