skip to content

Benchmarks and Tracing

BenchmarkDotNet measures a change properly, and dotnet-counters, dotnet-trace and dotnet-dump show what a running process does. Interviewers ask how you proved it got faster.

on this pageshow

explore

questions

6

In .NET, why does timing a method with a Stopwatch loop give misleading numbers, and what does BenchmarkDotNet do instead?

level: juniorimportance: should knowfreq 42%

answer

  1. the first call pays extra
  2. the code changes while you time it
  3. an unused result may vanish
  4. one sample is not a distribution
  5. separate Release process, pilot, warmup

basics

~20 s

A Stopwatch loop folds JIT compilation, tier changes, collections and noise into one number, and an unused result may be optimized away. BenchmarkDotNet runs a Release build in its own process, warms up, subtracts its overhead and reports statistics.

solid answer

~50 s

The first call to a method includes the JIT compiling it, and with tiered compilation the runtime later swaps the quick tier-0 code for optimized tier-1 code (the release runtime's default threshold is 30 calls), so a short loop times a mix of code versions. A garbage collection caused by earlier allocations can land inside the timed region, and if the loop discards the method's result the JIT may remove work whose result is never used. A Debug build or an attached debugger adds unoptimized code on top. BenchmarkDotNet generates and builds a Release project, runs each benchmark in a separate process, uses a pilot stage to choose how many invocations an iteration needs, measures and subtracts its own harness overhead, warms up, and then reports `Mean`, `Error` and `StdDev` over many iterations. You return the computed value from the `[Benchmark]` method so it is used rather than eliminated.

code

csharp · 24 lines
csharp
using System.Diagnostics;
using System.Globalization;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
string row = string.Join(',', Enumerable.Range(0, 200));

var sw = Stopwatch.StartNew();
int first = CountFields(row);                 // includes JIT-compiling CountFields
sw.Stop();
Console.WriteLine($"first call: {sw.Elapsed.TotalMicroseconds:F1} us");

sw.Restart();
for (int i = 0; i < 1_000; i++) CountFields(row);   // result discarded
sw.Stop();
Console.WriteLine($"next 1000 : {sw.Elapsed.TotalMicroseconds / 1_000:F2} us per call");
Console.WriteLine($"fields    : {first}");

static int CountFields(string line)
{
    int fields = 1;
    foreach (char c in line)
        if (c == ',') fields++;
    return fields;
}

go deeper

for a junior

Know that the first call includes JIT compilation, that benchmarks run in Release without a debugger, and that the benchmark method should return its result.

for a middle

Explain tier-0 versus tier-1 code during a measurement, dead-code elimination, and BenchmarkDotNet's pilot, overhead, warmup and workload stages.

for a senior

Spot a broken measurement in review: a Debug build, an [IterationSetup] on a nanosecond method, a deferred query returned, or a timing that ignores collections.

for a principal

Decide which claims need a harness at all; a Stopwatch log line around a seconds-long job is fine, while a nanosecond claim without a harness is not.

## What a Stopwatch loop really measures `System.Diagnostics.Stopwatch` is an accurate clock. The problem is what sits between `Start` and `Stop` when you wrap a loop around a method and divide by the iteration count. Several effects that have nothing to do with your algorithm end up inside that number: - **JIT compilation on the first call.** A .NET method starts as IL; the first call compiles it to machine code, so that call is far slower than the rest. In one run on .NET 10, the first call to a small parsing method took about 40 µs and each later call about 2 µs; the numbers depend on the machine, the gap does not. - **Tiered compilation.** The first code the JIT produces is quick, lightly optimized tier-0 code. After a method has been called enough times (30 calls by default in the release runtime, after a short startup delay) it is recompiled in the background as optimized tier-1 code. A loop of a few hundred iterations can time both versions. The mechanics belong to the material on the JIT and tiered compilation; for measurement, the point is that **the code under test changes during the test**. - **Garbage collections.** Allocations made before or during the loop can trigger a collection that pauses your thread inside the timed region. - **Dead-code elimination.** If the loop throws the result away, the optimizing JIT is allowed to remove computation whose result is never used, and you time an empty loop. - **Debug builds and debuggers.** A Debug build disables optimizations, and an attached debugger changes how code is generated. BenchmarkDotNet's guide says the debug version of a method can run 10 to 100 times slower. - **Noise.** Other processes, CPU frequency changes and timer resolution all move a single measurement, and one number has no error bar. ## What BenchmarkDotNet does instead BenchmarkDotNet is an open-source benchmarking library for .NET. Its "how it works" guide describes these steps: 1. It **generates an isolated project** for each runtime setting and builds it in Release mode. 2. It **launches a separate process** per benchmark (once by default), so your test host, debugger and earlier work do not leak in. 3. A **Pilot** stage doubles the invocation count until one iteration is long enough to measure reliably. 4. **OverheadWarmup** and **OverheadWorkload** stages time an empty method with the same signature, so the harness cost can be subtracted. 5. **ActualWarmup** runs your method until it settles, which covers JIT compilation and tiering. 6. **ActualWorkload** iterations are the real measurements; the result is the workload time minus the median overhead. 7. The summary reports `Mean`, `Error` (half of the 99.9% confidence interval) and `StdDev`, so you see the spread, not one number. ## The guard rails it adds - If the assembly that defines the benchmarks was compiled without optimizations, it reports: `Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.` and the benchmarks do not run until you rebuild in Release. - Its `EnvironmentAnalyser` warns in the summary when the application was built in Debug mode or ran with an attached debugger. - It warns when a benchmark returns a deferred `IEnumerable` or `IQueryable`, because you would time the creation of a query rather than its execution. - Returning the computed value from the `[Benchmark]` method is the documented way to avoid dead-code elimination. ## Side by side | Concern | Stopwatch loop | BenchmarkDotNet | |---|---|---| | JIT and tiering | timed together with your code | absorbed by warmup stages | | Harness cost | included | measured and subtracted | | Build configuration | whatever you ran | generated Release project | | Result | one number | mean, error, standard deviation | | Isolation | same process as everything else | separate process per benchmark | ## When a Stopwatch is still fine A `Stopwatch` around a long operation (a file import that takes seconds) is a reasonable log line, because JIT and harness costs are tiny compared with the work. For comparing two implementations of a hot method, or for any claim about nanoseconds or microseconds, use a benchmark harness. Production latency is a separate job for metrics and traces, not for either tool.

  • Why is returning the result from a [Benchmark] method enough to stop dead-code elimination?
    The JIT may remove a computation only when nothing observes its result. When the `[Benchmark]` method returns the value, BenchmarkDotNet's generated harness receives it, so the work is observable and stays. That is why its good-practices guide says to return `Math.Exp(1)` rather than call it and ignore it. A method that returns a lazy `IEnumerable` is different: return a materialized result or consume it, or you time only query creation.
  • When does [IterationSetup] spoil a BenchmarkDotNet microbenchmark?
    A method marked `[IterationSetup]` runs before each iteration, and BenchmarkDotNet then forces `UnrollFactor=1` and `InvocationCount=1`, so every measured iteration is a single invocation. For a method that takes nanoseconds, timer resolution and per-invocation overhead dominate. The docs advise against it for microbenchmarks; prepare inputs once in `[GlobalSetup]` instead.

Timing a Stopwatch loop is like timing a new commute on the first day: the first trip includes finding the way, later trips get faster as you learn shortcuts, and one trip says nothing about how much traffic varies. A benchmark harness drives the route many times after practice runs and reports the average and the spread.

  • Running the loop a million times and dividing removes every source of bias.
  • A Debug build is fine as long as both methods are measured the same way.
  • The first call to a method shows its steady-state speed.
  • Discarding a computed result cannot change what the JIT compiles.
  • By default BenchmarkDotNet times the method inside your own process and build.
open as a page

In BenchmarkDotNet, how do Baseline = true, [MemoryDiagnoser] and [Params] show that a rewritten CSV tokenizer is faster and allocates less?

level: middleimportance: should knowfreq 38%

basics

~20 s

Mark the old tokenizer [Benchmark(Baseline = true)] so each row gets a Ratio against it, add [MemoryDiagnoser] for Allocated bytes and Gen 0/1/2 collections per 1,000 operations, and use [Params] to repeat both for several input sizes.

open as a page

Using dotnet-counters on a .NET 10 process at 90% CPU with climbing memory, which System.Runtime counters do you read first, and why?

level: middleimportance: should knowfreq 36%

basics

~20 s

Run dotnet-counters monitor -p <pid> --showDeltas; it shows the System.Runtime meter by default. Compare CPU time with GC pause time and collections, the allocation total's growth, heap size against working set, the thread pool queue and exceptions.

open as a page

A .NET 10 service sits at 90% CPU in production; how do you capture and read a CPU profile with dotnet-trace, and what are its limits?

level: seniorimportance: should knowfreq 32%

basics

~20 s

Run dotnet-trace collect -p <pid> --duration for a short window; it enables the dotnet-common and dotnet-sampled-thread-time profiles over EventPipe. Open the nettrace in PerfView or Visual Studio on Windows, or convert it to Speedscope, and read the hottest stacks.

open as a page

A .NET service's memory climbs for an hour; when do you take a dotnet-gcdump, when a dotnet-dump, and how do you find what keeps objects alive?

level: seniorimportance: should knowfreq 33%

basics

~20 s

dotnet-gcdump captures the managed object graph over EventPipe but forces a full gen 2 collection; dotnet-dump writes a process dump analyzed with SOS. Compare type counts between two snapshots, then run gcroot on an instance of the growing type.

open as a page

As the lead of a .NET team, what evidence would you require before a performance change merges, and how would you keep that rule cheap enough to follow?

level: principalimportance: should knowfreq 26%

basics

~20 s

Require a BenchmarkDotNet comparison with the old code as baseline and [MemoryDiagnoser], run in Release on a quiet machine, plus a before-and-after production signal from counters or a trace, and scale that bar to how risky the change is.

open as a page