As the lead of a .NET team, what evidence would you require before a performance change merges, and how would you keep that rule cheap enough to follow?
answer
- a claim needs a baseline
- a benchmark plus a production signal
- allocations are evidence too
- noise on shared machines
- scale the bar to the risk
basics
~20 sRequire a BenchmarkDotNet comparison with the old code as baseline and [MemoryDiagnoser], run in Release on a quiet machine, plus a before-and-after production signal from counters or a trace, and scale that bar to how risky the change is.
solid answer
~50 sI would make the claim carry its own proof. A change that claims speed must include a BenchmarkDotNet class with the old code as `Baseline = true`, `[MemoryDiagnoser]`, and `[Params]` sizes taken from real traffic, run in Release outside a debugger, with the summary and environment block pasted into the pull request. For small differences I would ask for a statistical test column rather than an eyeballed mean. Changes that touch a hot service also need a production signal: `dotnet-counters collect` output or the service's own metrics before and after, or a trace showing the hot path moved. I would keep it cheap by tiering it: no benchmark for readability changes, a benchmark for hot paths, and production confirmation only for risky ones. Automated regression gates go on dedicated machines with thresholds, because shared CI agents are too noisy for small percentages.
code
bash · 7 lines# the PR's benchmark run, with memory columns and a JSON export for tooling
dotnet run -c Release --project bench/Csv.Benchmarks -- \
--filter '*CsvTokenizer*' --memory --exporters JSON --statisticalTest 5%
# production signal: the same 10-minute window before and after the deploy
dotnet-counters collect -p 4127 --format csv -o after-deploy.csv --duration 00:00:10:00 \
--counters "System.Runtime[dotnet.process.cpu.time,dotnet.gc.heap.total_allocated,dotnet.gc.pause.time,dotnet.gc.collections]"go deeper
Expect to attach a benchmark with a baseline when you claim a change is faster, and to run it in Release.
Know how to make the benchmark admissible: same class and run, memory diagnoser, parameters from real traffic, results pasted with the environment.
Pair microbenchmarks with production counters or traces for hot services, and judge whether a measured win matters for the request as a whole.
Own the evidence ladder: what each kind of change must prove, which benchmarks run where, how noisy gates are avoided, and what the rule costs the team.
## Why a lead sets this rule Performance changes are easy to argue about and hard to verify. Without a rule, reviewers either accept "it's faster" on trust or block changes on gut feeling. Both are expensive: the first ships regressions and complexity that buys nothing; the second slows the team. The lead's job is to define **what counts as evidence**, **when it is required**, and **how much it costs to produce**. There is no single right answer: the right bar depends on how latency-sensitive the product is, how much of the code is hot, and what measurement infrastructure the team already has. ## What a claim must include A performance claim names a **baseline**, a **metric** and a **workload**. "The new tokenizer is faster" becomes "against the current `SplitTokenizer`, the new one takes 0.4 of the time and allocates nothing per operation for lines of 16, 256 and 4,096 fields". In .NET that maps directly onto BenchmarkDotNet: - **Baseline:** `[Benchmark(Baseline = true)]` on the existing code, so the `Ratio` column compares like with like in the same run. - **Metric:** time from `Mean`, `Error` and `Ratio`; allocation from `[MemoryDiagnoser]`'s `Allocated` and generation columns. - **Workload:** `[Params]` or `[ParamsSource]` values that look like production input, not just the size that makes the new code win. - **Environment:** the summary's header with runtime, OS and CPU, because results do not transfer between machines or runtimes. ## An evidence ladder | Change | Evidence required | |---|---| | refactor that claims no performance effect | none beyond normal review | | hot-path change claiming speed or fewer allocations | BenchmarkDotNet with baseline and memory diagnoser in the PR | | small claimed gain (a few percent) | the same, plus a statistical test column and repeated runs | | change to a hot production service | benchmark plus before-and-after production counters or traces | | runtime or configuration change (a new .NET version, a GC setting) | production comparison first; microbenchmarks rarely capture it | Allocation results deserve special weight because they are **deterministic** for a given input, while timings move with the machine. A change that removes allocations from a hot path also reduces gen 0 collections, which shows up in production counters. ## Rules that keep benchmarks admissible 1. Release build, no debugger, run from a terminal; BenchmarkDotNet refuses a non-optimized benchmark assembly and warns about an attached debugger. 2. Old and new code in the **same class and the same run**; ratios from one run are far more stable than absolute numbers compared across runs. 3. Inputs built in `[GlobalSetup]`, results returned from each method, no `[IterationSetup]` on nanosecond methods. 4. For small differences, `[StatisticalTestColumn]` (its default threshold is 10%, configurable) labels each result Faster, Same or Slower with p-values, instead of a reviewer judging overlapping error bars. ## Automated regression gates A CI job that runs benchmarks on every merge sounds attractive and often fails in practice: - Shared runners are noisy, so a gate tuned to catch 3% regressions fails at random and gets ignored. - Comparing to stored numbers from another day compares two machines' moods. What works better is a **dedicated machine**, a curated set of benchmarks for the hottest code, comparisons within one run or against a recent run on the same hardware, and thresholds that match the noise you measured. BenchmarkDotNet's JSON exporter (`--exporters JSON`) gives the job something to parse. ## Production confirmation Microbenchmarks run warm, with small inputs and a quiet heap. Production adds contention, cache misses and other code. For risky changes: - record `dotnet-counters collect --format csv` (CPU time, allocation total, GC pause time, collections) for a fixed window before and after the deploy; - or compare the service's existing metrics dashboards over equal traffic; - for CPU claims, a short `dotnet-trace` capture showing the old hot frame gone. ## Trade-offs a lead weighs - **Cost of proof vs cost of a regression:** demanding benchmarks for every change slows the team; demanding none ships regressions. - **Benchmark upkeep:** benchmark projects rot unless someone owns them; keep few and important ones. - **False confidence:** a 2x microbenchmark win in code that is 1% of a request is not a product improvement; ask for the share of time first.
- Why compare old and new code in one BenchmarkDotNet run instead of against numbers saved from last week?Absolute timings depend on CPU frequency, thermal state, other processes, the OS and the runtime build, all of which drift between days and machines. Measuring both implementations in one run on one machine cancels most of that, and the `Ratio` column then compares like with like. Stored numbers are only useful as a trend on a dedicated, stable machine.
- When is a microbenchmark win not worth merging?When the code it speeds up is a small share of real request time, when the win appears only at an input size production never sees, or when it adds complexity and unsafe code to save nanoseconds in a path dominated by I/O. A trace or profile showing the method's share of CPU decides whether the improvement matters.
- A benchmark run once on a laptop proves a 3% improvement.
- A CI gate on shared runners can reliably catch small regressions.
- If a microbenchmark is twice as fast, the service will be too.
- Allocation numbers are secondary; only the timing columns matter.
- Every change should require benchmarks, whatever it touches.