In Flutter, how do you install a custom goldenFileComparator that tolerates small pixel differences, and what risk does the tolerance carry?

seniornice to knowfreq 24%

answer

extend LocalFileComparator
GoldenFileComparator.compareLists
diffPercent is a pixel fraction
set it in flutter_test_config.dart
restore it with addTearDown

basics

~20 s

Subclass LocalFileComparator, override compare to call GoldenFileComparator.compareLists and accept results whose diffPercent is under a threshold, then assign it to goldenFileComparator. The risk: diffPercent is the fraction of differing pixels, so a loose threshold hides real regressions.

solid answer

~40 s

goldenFileComparator is a global that matchesGoldenFile delegates to. I extend LocalFileComparator so path resolution, update for --update-goldens and failure output still work, and override compare(Uint8List imageBytes, Uri golden): call GoldenFileComparator.compareLists(imageBytes, await getGoldenBytes(golden)), pass when result.passed || result.diffPercent <= tolerance, otherwise throw with generateFailureOutput, disposing the result either way. To apply it to one test, assign it in the test and restore the previous one with addTearDown; for a directory, replace it in testExecutable inside flutter_test_config.dart, building it from the existing comparator's basedir. The catch: diffPercent counts pixels whose RGBA differs at all, divided by total pixels, so 1% on a 1080 x 1920 capture is about 20,700 pixels - enough to hide a changed price.

`goldenFileComparator` is a global that `matchesGoldenFile` delegates to. I extend `LocalFileComparator` so path resolution, `update` for `--update-goldens` and failure output still work, and override `compare(Uint8List imageBytes, Uri golden)`: call `GoldenFileComparator.compareLists(imageBytes, await getGoldenBytes(golden))`, pass when `result.passed || result.diffPercent <= tolerance`, otherwise throw with `generateFailureOutput`, disposing the result either way. To apply it to one test, assign it in the test and restore the previous one with `addTearDown`; for a directory, replace it in `testExecutable` inside `flutter_test_config.dart`, building it from the existing comparator's `basedir`. The catch: `diffPercent` counts pixels whose RGBA differs at all, divided by total pixels, so 1% on a 1080 x 1920 capture is about 20,700 pixels - enough to hide a changed price.

code

dart · 27 lines

dart

import 'dart:typed_data';

import 'package:flutter/foundation.dart';
import 'package:flutter_test/flutter_test.dart';

class TolerantGoldenComparator extends LocalFileComparator {
  TolerantGoldenComparator(super.testFile, {required this.tolerance})
    : assert(tolerance >= 0 && tolerance <= 1);

  /// Maximum fraction of pixels allowed to differ (0.0005 = 0.05%).
  final double tolerance;

  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    final ComparisonResult result = await GoldenFileComparator.compareLists(
      imageBytes,
      await getGoldenBytes(golden),
    );
    if (result.passed || result.diffPercent <= tolerance) {
      result.dispose();
      return true;
    }
    final String error = await generateFailureOutput(result, golden, basedir);
    result.dispose();
    throw FlutterError(error);
  }
}

go deeper

for a junior

Recall that golden comparisons go through the global goldenFileComparator, and that it can be replaced with a subclass of LocalFileComparator.

for a middle

Explain compare versus update, what compareLists returns, and why diffPercent is a fraction of changed pixels with no colour threshold.

for a senior

Scope tolerances narrowly, size them in hundredths of a percent, capture small regions and still review diffs, because a pixel budget can swallow real regressions.

for a principal

Treat tolerance as a stopgap: fund a single reference platform for goldens and require justification for each tolerant golden.

How the comparator fits in

matchesGoldenFile does not compare images itself. It captures and encodes a PNG, then calls the global goldenFileComparator:

in normal mode it calls compare(Uint8List imageBytes, Uri golden), which returns true or throws a descriptive failure;
with flutter test --update-goldens (autoUpdateGoldenFiles is true) it calls update(Uri golden, Uint8List imageBytes) instead.

The flutter test bootstrap installs a LocalFileComparator for each test file before your code runs. It resolves keys relative to the test file's directory (its basedir), requires an exact match, and writes diff images to a failures folder.

What the comparison measures

GoldenFileComparator.compareLists(test, master) decodes both PNGs and returns a ComparisonResult:

Field	Meaning
`passed`	true only when the images are identical
`diffPercent`	differing pixels divided by total pixels (0.0 to 1.0); 1.0 when sizes differ
`error`	a message such as "Pixel test failed, 0.37%, 1790px diff detected."
`diffs`	master, test, masked-diff and isolated-diff images

A pixel counts as different when the summed absolute difference of its red, green, blue and alpha channels is non-zero. There is no per-pixel colour threshold in this count: a one-step change in one channel counts the same as black turning white. The result holds images, so it must be dispose()d.

Writing the comparator

Extend LocalFileComparator, not the abstract GoldenFileComparator, so update, getGoldenBytes, basedir and generateFailureOutput come for free.
Override compare: get the result from compareLists, pass if result.passed || result.diffPercent <= tolerance.
On failure, build the message with generateFailureOutput(result, golden, basedir) (which also writes the failure images) and throw a FlutterError.
Dispose the ComparisonResult on both paths.
Validate the tolerance range in the constructor (0 to 1).

Installing it

One test: save the current comparator, assign yours, and restore it with addTearDown(() => goldenFileComparator = previous). The flutter_test API docs show exactly this pattern.
A directory: in flutter_test_config.dart, inside testExecutable, read the LocalFileComparator the bootstrap already installed and construct yours from its basedir, so keys still resolve relative to the right folder; then await testMain().

The risk

The tolerance is a budget of pixels, not a measure of how visible a change is:

On a receipt captured at 1080 x 1920 physical pixels (2,073,600 pixels), a tolerance of 0.01 accepts about 20,700 changed pixels. A total that changes from 42.50 to 42.60 alters far fewer than that.
A large capture dilutes a small change: the same broken total is a smaller fraction of a full-screen golden than of a card-sized one.
Tolerance hides the cause of drift - usually goldens generated on a different platform or Flutter version - rather than removing it.

Mitigations:

Keep the threshold tiny (hundredths of a percent), not whole percents.
Apply it only to the goldens that need it, such as those with real fonts or blurs.
Capture small, keyed RepaintBoundary regions so meaningful changes are a larger fraction.
Still review isolatedDiff.png whenever a tolerant golden reports differences close to its budget.

When it is justified

A tolerant comparator is reasonable when goldens must run on more than one platform and the remaining differences are verified rendering noise - for example blurs or gradients that differ by a handful of edge pixels. Where a single reference platform is possible, prefer it and keep the default exact comparator.

## How the comparator fits in `matchesGoldenFile` does not compare images itself. It captures and encodes a PNG, then calls the global **`goldenFileComparator`**: - in normal mode it calls `compare(Uint8List imageBytes, Uri golden)`, which returns `true` or throws a descriptive failure; - with `flutter test --update-goldens` (`autoUpdateGoldenFiles` is true) it calls `update(Uri golden, Uint8List imageBytes)` instead. The `flutter test` bootstrap installs a **`LocalFileComparator`** for each test file before your code runs. It resolves keys relative to the test file's directory (its `basedir`), requires an exact match, and writes diff images to a `failures` folder. ## What the comparison measures `GoldenFileComparator.compareLists(test, master)` decodes both PNGs and returns a **`ComparisonResult`**: | Field | Meaning | |---|---| | `passed` | true only when the images are identical | | `diffPercent` | differing pixels divided by total pixels (0.0 to 1.0); 1.0 when sizes differ | | `error` | a message such as "Pixel test failed, 0.37%, 1790px diff detected." | | `diffs` | master, test, masked-diff and isolated-diff images | A pixel counts as different when the summed absolute difference of its red, green, blue and alpha channels is non-zero. There is **no per-pixel colour threshold** in this count: a one-step change in one channel counts the same as black turning white. The result holds images, so it must be `dispose()`d. ## Writing the comparator 1. Extend `LocalFileComparator`, not the abstract `GoldenFileComparator`, so `update`, `getGoldenBytes`, `basedir` and `generateFailureOutput` come for free. 2. Override `compare`: get the result from `compareLists`, pass if `result.passed || result.diffPercent <= tolerance`. 3. On failure, build the message with `generateFailureOutput(result, golden, basedir)` (which also writes the failure images) and throw a `FlutterError`. 4. Dispose the `ComparisonResult` on both paths. 5. Validate the tolerance range in the constructor (0 to 1). ## Installing it - **One test**: save the current comparator, assign yours, and restore it with `addTearDown(() => goldenFileComparator = previous)`. The flutter_test API docs show exactly this pattern. - **A directory**: in `flutter_test_config.dart`, inside `testExecutable`, read the `LocalFileComparator` the bootstrap already installed and construct yours from its `basedir`, so keys still resolve relative to the right folder; then `await testMain()`. ## The risk The tolerance is a **budget of pixels**, not a measure of how visible a change is: - On a receipt captured at 1080 x 1920 physical pixels (2,073,600 pixels), a tolerance of `0.01` accepts about 20,700 changed pixels. A total that changes from 42.50 to 42.60 alters far fewer than that. - A large capture dilutes a small change: the same broken total is a smaller fraction of a full-screen golden than of a card-sized one. - Tolerance hides the **cause** of drift - usually goldens generated on a different platform or Flutter version - rather than removing it. Mitigations: - Keep the threshold tiny (hundredths of a percent), not whole percents. - Apply it only to the goldens that need it, such as those with real fonts or blurs. - Capture small, keyed `RepaintBoundary` regions so meaningful changes are a larger fraction. - Still review `isolatedDiff.png` whenever a tolerant golden reports differences close to its budget. ## When it is justified A tolerant comparator is reasonable when goldens must run on more than one platform and the remaining differences are verified rendering noise - for example blurs or gradients that differ by a handful of edge pixels. Where a single reference platform is possible, prefer it and keep the default exact comparator.

Why extend LocalFileComparator instead of implementing GoldenFileComparator directly?
`LocalFileComparator` already resolves keys against `basedir`, reads goldens with `getGoldenBytes`, implements `update` for `--update-goldens`, and writes the four failure images via `generateFailureOutput`. Implementing the abstract class means rewriting all of that; overriding only `compare` changes just the pass rule.
Does diffPercent weigh how much a pixel's colour changed?
No. A pixel counts once if its summed RGBA difference is non-zero, whether it moved one colour step or flipped from black to white. `diffPercent` is that count divided by total pixels, so it measures how many pixels changed, not how visible the change is.
What happens with a size mismatch under a tolerant comparator?
`compareLists` reports `diffPercent` 1.0 with 'image sizes do not match', so any tolerance below 1.0 still fails. Size differences point at the view size, pixel ratio or layout, not rendering noise.

saying these in an interview costs you the question

diffPercent measures how different the colours look, not how many pixels changed
A 1% tolerance only lets through anti-aliasing noise
Implement GoldenFileComparator from scratch to add a tolerance
Assign the comparator in one test and never restore it
Tolerance also absorbs goldens whose image size changed

In Flutter, how do you install a custom goldenFileComparator that tolerates small pixel differences, and what risk does the tolerance carry?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

How the comparator fits in

What the comparison measures

Writing the comparator

Installing it

The risk

When it is justified

▾common follow-ups3 questions

▾red flags5 flags

How the comparator fits in

What the comparison measures

Writing the comparator

Installing it

The risk

When it is justified

answer

go deeper