skip to content

With Minitest::Benchmark, what does assert_performance_linear 0.99 check, and why is it not a time limit for a slow test?

level: middleimportance: nice to knowfreq 14%

answer

  1. require "minitest/benchmark"
  2. bench_ methods, not test_
  3. bench_range 1, 10, 100, 1_000, 10_000
  4. curve fit, r squared >= 0.99
  5. prints tab-separated timings

basics

~20 s

It times the block at each size in bench_range and passes when those times fit a straight line with a coefficient of determination of at least 0.99. It checks how cost grows with input size, not how many seconds anything takes.

solid answer

~40 s

`Minitest::Benchmark` (loaded with `require "minitest/benchmark"`) is a `Minitest::Test` subclass whose runnable methods start with `bench_`. `assert_performance_linear 0.99 { |n| ... }` runs the block once for each `n` in `bench_range`, by default `[1, 10, 100, 1_000, 10_000]`, after a `GC.start`, prints the timings tab-separated, fits them to `y = a + bx` and asserts that the fit's r-squared is at least `0.99`. So it catches an algorithm that turned quadratic, not a test that became twice as slow. Siblings check other shapes: `assert_performance_constant` (slope near zero), `_logarithmic`, `_exponential`, `_power`. Timings on shared CI runners are noisy, which is why the README suggests loading benchmarks only when `ENV["BENCH"]` is set.

code

ruby · 11 lines
ruby
class BenchFineLookup < Minitest::Benchmark
  def self.bench_range
    bench_linear(1_000, 10_000, 1_000)
  end

  def bench_lookup_by_member
    assert_performance_constant 0.9999 do |n|
      FineIndex.build(n).lookup("member-42")
    end
  end
end

go deeper

for a junior

Recall that Minitest::Benchmark classes use bench_ methods and must be required separately with minitest/benchmark.

for a middle

Explain bench_range, the curve fits and r-squared threshold, and why a linear fit says nothing about absolute speed.

for a senior

Use scaling assertions to guard algorithms against accidental quadratic growth, and keep them out of noisy default CI runs.

for a principal

Decide which performance properties belong in the unit suite as growth checks and which need dedicated benchmark jobs with budgets.

## What Minitest::Benchmark is for Minitest ships a small benchmarking layer for one question: **does this code scale the way I think it does?** It is not loaded by `minitest/autorun`; you opt in with `require "minitest/benchmark"`. ```ruby require "minitest/autorun" require "minitest/benchmark" if ENV["BENCH"] class BenchFineReport < Minitest::Benchmark def bench_report_over_loans assert_performance_linear 0.99 do |n| FineReport.new(loans(n)).total end end end ``` - The class inherits from **`Minitest::Benchmark`**, itself a `Minitest::Test` subclass. - Its runnable methods match **`/^bench_/`**, not `/^test_/`. - It runs with the rest of the suite, so it shares seeds, filters and reporters. ## How the measurement works `assert_performance(validation) { |n| ... }` is the core: 1. It takes the class's **`bench_range`**, by default `bench_exp(1, 10_000)`, which is `[1, 10, 100, 1_000, 10_000]`. 2. For each `n` it calls `GC.start`, times one call of the block with a monotonic clock, and prints the time. 3. It passes the sizes and times to the validation, which fits a curve and asserts on it. The output is a tab-separated row per benchmark, meant to be pasted into a spreadsheet for graphing. ## The shape assertions | assertion | fits | passes when | |---|---|---| | `assert_performance_linear t` | `y = a + bx` | r-squared of the fit >= `t` | | `assert_performance_logarithmic t` | `y = a + b ln x` | r-squared >= `t` | | `assert_performance_exponential t` | `y = a e^(bx)` | r-squared >= `t` | | `assert_performance_power t` | `y = a x^b` | r-squared >= `t` | | `assert_performance_constant t` | `y = a + bx` | the slope `b` is within `1 - t` of zero | All thresholds default to `0.99`. r-squared (the coefficient of determination) measures how well the chosen curve explains the timings: 1.0 is a perfect fit. For the constant check the source notes that r-squared is a poor measure, so it tests the slope directly and suggests tightening the threshold. ## Why it is not a time limit - **No seconds anywhere.** A report that takes 3 seconds for 10,000 loans but grows linearly passes; one that takes 0.3 seconds but grows quadratically fails the linear check. - **It answers "how does it grow"**, which is what catches an accidental nested loop. - **Noise matters.** Five points on a shared CI runner can wobble enough to fail a 0.99 fit, especially at small `n` where times are microseconds. That is why benchmarks are usually opt-in (`BENCH=1`) or run on dedicated jobs. To tune the sizes, override `self.bench_range`, for example with `bench_linear(1_000, 10_000, 1_000)`. ## Common mistakes - Reading the threshold as seconds and loosening it to 0.5 to make a slow benchmark pass, which only accepts a worse fit. - Leaving the default range when the smallest sizes run in microseconds, so timer noise dominates the fit; start the range higher. - Doing setup inside the timed block, so the fit measures fixture building rather than the code under test. - Running benchmarks in parallel with other tests, where contention distorts every timing. ## Where it sits among tools - For finding which tests make a suite slow, Minitest's `-v` timings and `rake test:slow` are the tools, not `Minitest::Benchmark`. - For comparing two implementations' speed, Ruby's `Benchmark` library or benchmark-ips are the usual tools. - `Minitest::Benchmark` is for guarding an algorithm's growth rate inside the test suite, with a spec form (`bench_performance_linear` inside a `describe`) for spec users.

  • With Minitest::Benchmark, why can assert_performance_linear 0.99 pass for code that is far too slow?
    It only checks that the timings across `bench_range` fit a straight line with r-squared of at least 0.99. A consistently slow but linear implementation fits perfectly. An absolute budget needs a separate assertion on elapsed time, which is fragile on shared machines.
  • With Minitest::Benchmark, how do you keep benchmarks out of the normal test run?
    Load the library conditionally, `require "minitest/benchmark" if ENV["BENCH"]`, as the Minitest README suggests, and guard the benchmark class or its methods with the same condition so the class does not fail to load. CI can then run `BENCH=1` in a dedicated job.

saying these in an interview costs you the question

  • assert_performance_linear 0.99 fails when the block takes over 0.99 seconds.
  • Benchmark methods in Minitest::Benchmark are named test_ like other tests.
  • minitest/autorun loads Minitest::Benchmark automatically.
  • A 0.99 linear fit proves the code is fast enough for production.
  • Minitest::Benchmark is the tool for finding a suite's slowest tests.