skip to content

When should you return a value from a @Benchmark method versus using a Blackhole, and how do you sink multiple results correctly?

level: middleimportance: must knowfreq 68%

answer

  1. Return = one result; Blackhole = many
  2. One consume() per independent result
  3. Consume inside the loop, not after
  4. Never merge results to fit one return
  5. Blackhole = engineered sink, not a plain assign

basics

~20 s

Return the value when your benchmark produces a single result. Use a Blackhole when you produce more than one result, or a result inside a loop, calling bh.consume() on each so none of them gets deleted by the JIT.

solid answer

~40 s

A @Benchmark method can return exactly one value, and JMH consumes that return value so the JIT must produce it — so for a single-result benchmark, just return it. But many benchmarks produce several results, or results inside a loop, or operate on a void API. You can only return one thing, so for everything else you inject a Blackhole parameter and call blackhole.consume(result) for each value you want kept alive. Blackhole is JMH's carefully engineered sink: consuming a value is cheap, has near-constant cost, and the JIT cannot fold or eliminate it. A common mistake is to combine multiple results into one (e.g. summing them or only returning the last) — that changes what you measure and can let the JIT optimize the intermediate computations. Consume each result independently instead.

code

java · 22 lines
java
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;

@State(Scope.Thread)
public class HashBench {
    String a = "alpha";
    String b = "bravo";

    // Single result: just return it; JMH consumes the return value.
    @Benchmark
    public int oneHash() {
        return a.hashCode();
    }

    // Multiple results: consume each separately so neither is eliminated.
    @Benchmark
    public void twoHashes(Blackhole bh) {
        bh.consume(a.hashCode());
        bh.consume(b.hashCode());
        // WRONG: return a.hashCode() + b.hashCode();  // changes what you measure
    }
}

go deeper

for a junior

Knows to return the result for a single computation and that a Blackhole exists for more than one.

for a middle

Picks return vs Blackhole correctly, consumes each result separately, and avoids merging results into one return value.

for a senior

Explains why merging results distorts the measurement and re-enables folding, and consumes correctly inside loops; aware of Blackhole's near-constant cost.

for a principal

Understands Blackhole internals (volatile reads vs compiler blackholes), can reason about the small overhead it adds, and sets benchmarking conventions for a team to keep results trustworthy.

## Setup: the two JMH sinks JMH gives you two ways to make a computed result 'observable' so the **JIT** (the JVM's runtime compiler that turns hot code into optimized native code) cannot delete it as **dead code** (code whose result is never used). Both work by ensuring the value is consumed: 1. **Return it.** If your `@Benchmark` method returns a value, JMH internally feeds that return value into a Blackhole for you. Simple, zero boilerplate — but a method can return only **one** value. 2. **A `Blackhole`.** This is a JMH-provided object you declare as a method parameter; JMH injects it. You call `blackhole.consume(x)` to sink a value. You can call it as many times as you like. ## When each applies **Single result -> return it:** ```java @Benchmark public long hash() { return data.hashCode(); // one result, consumed by JMH } ``` **Multiple results -> Blackhole, one consume each:** ```java @Benchmark public void twoHashes(Blackhole bh) { bh.consume(a.hashCode()); bh.consume(b.hashCode()); } ``` If you only returned `a.hashCode()` and computed `b.hashCode()` without using it, the JIT would delete the `b` computation. Each independent result needs its own sink. **Result produced in a loop -> consume inside the loop:** ```java @Benchmark public void overList(Blackhole bh) { for (Item it : items) { bh.consume(transform(it)); // each iteration's result kept alive } } ``` ## Why Blackhole is special (not just an assignment) You might think 'I'll just assign to a field' or 'sum the results and return the sum'. These are subtly wrong: - **Assigning to a regular local** still leaves the value unobserved -> DCE applies. - **Summing/combining results** changes the computation: the JIT may be able to algebraically simplify or partially fold the combination, and you are now also measuring the addition. You wanted to measure the individual operations. - **Returning only the last result** lets the JIT delete all the earlier ones. `Blackhole.consume()` is engineered (with volatile reads, dead-branch tricks, and JVM-version-specific compiler blackholes) so that: (a) the JIT genuinely cannot prove the value is unused, and (b) the consumption itself costs a small, roughly constant amount that does not distort the measurement. Modern JVMs even support **compiler blackholes**, where the JIT recognizes the consume and models it as a true sink with near-zero overhead. ## Decision rule - Exactly one result, computed once -> **return** it. - More than one result, or a result inside a loop, or a void operation you must time -> **Blackhole**, consuming each result separately. ## Pitfall to avoid Do NOT merge results to fit the single return slot. Merging changes the measured work and can re-enable folding. The whole reason Blackhole accepts unlimited consumes is so each result stays independently alive.

  • Why is summing several results and returning the sum a worse defense than consuming each?
    It changes the measured work (you now also time the additions) and the JIT may algebraically simplify or partially fold the running sum, letting it optimize away some of the intermediate computations. Independent consumes keep each result genuinely live and measured.
  • What is a 'compiler blackhole' and why does it matter?
    Newer JVMs let the JIT recognize Blackhole.consume as an intrinsic true sink, so the consumption itself has essentially no overhead and cannot be optimized away — giving more accurate measurements than the older volatile-based blackhole tricks.

saying these in an interview costs you the question

  • Summing or combining results to squeeze them into one return value
  • Returning only the last result of a loop and assuming the rest are measured
  • Thinking assigning to a (non-volatile) field is an adequate substitute for Blackhole
  • Calling consume() once outside a loop when the loop produced many results

context

resolving links…