skip to content

In Jetpack Benchmark, how does a Microbenchmark with BenchmarkRule differ from a Macrobenchmark with MacrobenchmarkRule, and why run them on a physical device?

level: juniorimportance: should knowfreq 42%

answer

  1. inside the process vs outside it
  2. one function vs a user journey
  3. measureRepeated loop vs packageName and metrics
  4. emulator and debuggable are errors

basics

~20 s

A Microbenchmark loops one hot function inside the test process with BenchmarkRule; a Macrobenchmark drives the installed app from a separate test module to time startup or frames. Emulators measure the host, so both report an error.

solid answer

~40 s

A **Microbenchmark** wraps one piece of code in `BenchmarkRule.measureRepeated { }`: the library warms it up, runs it many times in the test process and reports time and allocations. It fits hot paths such as a parser. A **Macrobenchmark** lives in a separate `com.android.test` module: `MacrobenchmarkRule.measureRepeated(packageName, metrics, iterations)` launches and drives the installed app through UI Automator and reads `StartupTimingMetric` or `FrameTimingMetric` from a system trace per iteration. Both want a **physical device** and a **non-debuggable** build: an emulator measures the host machine and a debuggable build disables runtime optimizations, so the libraries report a configuration error rather than a misleading number.

code

kotlin · 33 lines
kotlin
@RunWith(AndroidJUnit4::class)
class FeedParserBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    private val json = loadFixture("feed.json")

    @Test
    fun parseFeed() {
        benchmarkRule.measureRepeated {
            val input = runWithMeasurementDisabled { json.copyOf() }
            FeedParser.parse(input)
        }
    }
}

@LargeTest
@RunWith(AndroidJUnit4::class)
class ColdStartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun coldStartup() = benchmarkRule.measureRepeated(
        packageName = "com.example.feed",
        metrics = listOf(StartupTimingMetric()),
        iterations = 10,
        startupMode = StartupMode.COLD,
        setupBlock = { pressHome() },
    ) {
        startActivityAndWait()
    }
}

go deeper

for a junior

Recall the split: BenchmarkRule loops one function inside the test process; MacrobenchmarkRule drives the whole installed app from a separate test module. Name the device rule too.

for a middle

Explain what each library controls: warmup, repetitions and allocations for micro; launch, compilation state and a trace per iteration for macro, and why setup lives outside the timed part.

for a senior

Show you trust only release-like numbers: a non-debuggable benchmark module, a physical device, enough battery, and results renamed when an error is suppressed.

for a principal

Frame it as which claims the team needs proven: a few microbenchmarks for routines tuned often, macrobenchmarks for journeys users feel, and what each costs to keep in CI.

## Two libraries for two different questions Jetpack Benchmark is really two libraries that answer two different questions about speed, and an interviewer wants to hear that you know which one fits which question. - **Microbenchmark** (`androidx.benchmark:benchmark-junit4`) answers *"how long does this one piece of code take?"*. You call your own function directly, many times in a loop, inside the test process. - **Macrobenchmark** (`androidx.benchmark:benchmark-macro-junit4`) answers *"how does the app feel to a user?"*. It launches and drives the installed app from the outside, the way a person would, and reads the timings from a system trace. Both are written as ordinary JUnit4 instrumented tests and both run on an Android device, but they are set up, run and read very differently. ## Microbenchmark: a measured loop around hot code A **microbenchmark** is a JUnit rule, `BenchmarkRule`, plus a block you pass to `measureRepeated { }`. The library runs the block over and over: it warms the code up first, then times many repetitions and reports the result, with allocation counts, in the Android Studio console and in a JSON file. Key pieces to know: - `BenchmarkRule.measureRepeated { }` is the loop. It must not be called from the main thread; UI code (a View or a composable being measured in-process) uses `measureRepeatedOnMainThread { }`, added in 1.3.0, which runs the block on the main thread but yields it periodically, under a hard deadline, so a long run does not trigger an ANR. - `runWithMeasurementDisabled { }` inside the loop runs per-iteration setup, such as copying an unsorted array before sorting it, without counting its time. It replaced `runWithTimingDisabled { }`, which was deprecated in 1.4.0. - The test runner is `androidx.benchmark.junit4.AndroidBenchmarkRunner`. - The code under test usually sits in its own Gradle module that the benchmark module depends on, so the benchmark module can be **non-debuggable** without affecting your normal tests. - With the `androidx.benchmark` Gradle plugin on AGP 8.4 or later, the benchmark APK is fully compiled ahead of time by default, so the JIT warming up does not add noise. It suits **hot code paths**: a JSON parser, a diffing routine, a data conversion that runs thousands of times. It does not suit code that runs once, such as app startup, because a loop measures the warmed-up case. ## Macrobenchmark: the app as a user sees it A **macrobenchmark** lives in a separate `com.android.test` module that targets your app module. You write `MacrobenchmarkRule().measureRepeated(...)` and must give it at least the target `packageName`, a list of `metrics` and the number of `iterations`. Inside the block you do what a user does: start the activity, scroll a list, open a screen. Interaction goes through UI Automator, never through calls into your app's code. - Metrics include `StartupTimingMetric`, `FrameTimingMetric` and `TraceSectionMetric`; each iteration captures its own system trace. - `startupMode` and `compilationMode` control how the app is launched and how much of it is compiled ahead of time before measuring. - The target app must be `<profileable>`, close to a release build (non-debuggable, ideally minified) and should include ProfileInstaller so the library can install and reset compilation profiles. | | Microbenchmark | Macrobenchmark | |---|---|---| | Rule | `BenchmarkRule` | `MacrobenchmarkRule` | | Runs | inside the test process, calling your code | a separate test app driving the installed app | | Measures | one function, many repetitions | startup, frames, trace sections per iteration | | Module | library module plus a benchmark module | `com.android.test` module targeting the app | | Typical use | parsers, algorithms, hot loops | cold start, scroll jank, a user journey | ## Why a physical device and a release-like build Both libraries refuse to hand you misleading numbers. They check the environment and report an error instead of a result when: 1. **The device is an emulator.** An emulator shares the host computer's CPU, memory and disk, so its timings describe the host, not a phone your users hold. 2. **The code is debuggable.** A debuggable build turns off runtime optimizations to support debugging, so it runs drastically slower and can even move in the opposite direction from release performance. 3. **The battery is low.** Below 25% many devices slow cores down to save power, even while charging. 4. **A rooted device has unlocked clocks** (microbenchmark only). CPU frequency moves with temperature and load; locking clocks keeps it steady. You can suppress a specific error with the `androidx.benchmark.suppressErrors` instrumentation argument, but the library then renames the result, for example with a `DEBUGGABLE_` prefix, so nobody mistakes it for a real number. ## Choosing between them Ask what you want to prove. If the claim is "this function got faster", write a microbenchmark around it. If the claim is "the app starts faster" or "the feed scrolls without jank", write a macrobenchmark, because only it sees process start, class loading, rendering and the compilation state a user's device really has. Many teams keep both: microbenchmarks for the few hot routines they tune often, and macrobenchmarks for the journeys users notice.

  • Why does a Microbenchmark call measureRepeatedOnMainThread instead of measureRepeated when it measures View or Compose code?
    `measureRepeated` throws if called from the main thread, and wrapping a long benchmark in `@UiThreadTest` risks an ANR because the run can last far longer than five seconds. `measureRepeatedOnMainThread`, added in 1.3.0, runs the measured block on the main thread while yielding it periodically and enforcing a hard deadline, so UI code runs on the right thread without freezing it for the whole benchmark.
  • What happens if you suppress the DEBUGGABLE error with androidx.benchmark.suppressErrors and run anyway?
    The benchmark runs, but the library prefixes the result's test name with `DEBUGGABLE_` so the number is visibly marked as untrustworthy. The timing is real for a debuggable build, which runs without many runtime optimizations, so it can mislead you about release performance and should never feed a regression dashboard.

  • A Macrobenchmark calls the app's classes directly, like a Microbenchmark does.
  • An emulator is fine for benchmarks if the host machine is fast.
  • A debuggable build is only slightly slower, so its numbers still compare.
  • A Microbenchmark is the right tool to measure cold app startup.
  • A Macrobenchmark can live in the app module's own androidTest folder.

Assumes Benchmark 1.5: runWithMeasurementDisabled replaced runWithTimingDisabled (deprecated in 1.4.0) and measureRepeatedOnMainThread exists since 1.3.0.