skip to content

Should a CI performance budget be an absolute limit or a comparison against the base branch's measurement? Explain the tradeoff, and how a team keeps the number from drifting upward over time.

level: middleimportance: should knowfreq 38%

answer

  1. fixed number versus difference
  2. who gets blamed for the drift
  3. small additions, none of them refused
  4. the baseline has to be rebuilt
  5. lower the ceiling when you earn it

basics

~20 s

Use both. An absolute limit expresses the contract with users but blames whoever happens to cross it; a delta against the base branch attributes the cost to the pull request that added it but permits unlimited slow creep. Together with a ratchet, they close each other's gap.

solid answer

~50 s

They fail in opposite directions, so mature setups run both. An **absolute** budget — "scripts on /product stay under 170 kB" — is the number that actually matters to a user, and it never needs a baseline build. Its weakness is attribution: it is silent for months while the site creeps toward the limit, then fails whichever unlucky pull request crosses it, and that author is not the one who spent the budget. A **delta** budget — "no pull request may add more than 5 kB" — attributes the cost correctly and catches the creep, but it permits unbounded growth as long as each step is small, and it needs a trustworthy baseline: you must build the base branch in the same job, since a stale stored artifact silently drifts. Run the absolute limit as the ceiling, the delta as the per-change rule, and **ratchet** the absolute limit down whenever a change genuinely improves the number.

code

bash · 8 lines
bash
BASE_BYTES=$(cat .perf-baseline/main.js.bytes)
HEAD_BYTES=$(wc -c < dist/main.js)
DELTA=$(( HEAD_BYTES - BASE_BYTES ))
echo "main.js: ${HEAD_BYTES} bytes (delta ${DELTA})"
if [ "$DELTA" -gt 5120 ]; then
  echo "FAIL: main.js grew more than 5 KB against the base branch"
  exit 1
fi

go deeper

for a junior

Understand the difference between checking a measurement against a fixed limit and checking it against the branch you are merging into, and that the second one tells you what your change cost.

for a middle

Be ready to name each form's blind spot — the absolute limit blames whoever crosses it after a year of drift, the delta permits unlimited creep in small steps — and to argue for running both.

for a senior

Show that you build the baseline in the same job rather than trusting a stored artifact, size delta thresholds above the measured noise floor, and lower absolute budgets deliberately when a change earns the saving.

for a principal

Own the long horizon: define who may raise a ceiling and what they owe for it, keep trend data per route so slow decay is visible as a slope, and make the ratchet a reviewed decision rather than an automated overwrite.

## The two shapes Every budget assertion is one of two forms. An **absolute** assertion compares a measurement against a fixed number: this route's scripts must be under 170 kB, its largest contentful paint under 2.5 seconds. A **delta** assertion compares the pull request's measurement against the same measurement on the branch it targets: this change must not add more than 5 kB, must not slow LCP by more than 150 ms. They look interchangeable and are not. They catch different regressions and misattribute different costs. ## What the absolute form gets right and wrong An absolute budget is the only one of the two that expresses anything about the user. Users experience 214 kB of JavaScript; they do not experience "46 kB more than last week". It is also cheap to run: no baseline build, no history, one measurement compared to one constant. Its failure mode is **the unlucky author**. Suppose the budget is 170 kB and the route currently ships 168 kB after a year of two-kilobyte additions that all passed. The next pull request adds three kilobytes and fails. That author did nothing unusual, has no context on the previous year of drift, and now owns a problem that twenty other changes created. In practice one of two things happens: they spend a week on unrelated optimisation, or someone raises the budget to 220 kB and the cycle restarts. The absolute budget was silent exactly while the damage was being done. ## What the delta form gets right and wrong A delta budget puts the cost where it belongs. Every pull request is measured against its own base, so the pull request that adds a 40 kB date picker is the one that has to justify it, at the moment when the alternatives — a smaller library, a lazy import, native inputs — are still on the table. It also catches slow creep at the only point where creep is legible. Its failure mode is **death by a thousand cuts within the limit**. If the rule is "no more than 5 kB per change", fifty conforming pull requests add 250 kB and every one of them was green. The delta assertion has no memory and no notion of a ceiling. It also has an operational cost: it needs a base measurement, and where that comes from decides whether the check is honest. Building the base branch inside the same job on the same runner is the reliable option — same toolchain, same machine, so the comparison is apples to apples. Reusing a stored artifact from the last main-branch build is cheaper but goes stale, misses merges that landed since, and silently compares across toolchain upgrades. For measured metrics rather than byte counts, the delta must also clear the run-to-run noise floor, or it fails on nothing. ## Running both The absolute limit is the ceiling and the contract; the delta is the per-change rule that keeps anyone from quietly walking up to the ceiling. Failing either blocks the merge, and the two messages read differently: "this route is over its limit" versus "this change adds 12 kB, over the 5 kB per-change allowance". ```bash BASE_BYTES=$(cat .perf-baseline/main.js.bytes) HEAD_BYTES=$(wc -c < dist/main.js) DELTA=$(( HEAD_BYTES - BASE_BYTES )) echo "main.js: ${HEAD_BYTES} bytes (delta ${DELTA})" if [ "$DELTA" -gt 5120 ]; then echo "FAIL: main.js grew more than 5 KB against the base branch" exit 1 fi ``` ## The ratchet Both forms are ceilings, and ceilings only ever move up unless something moves them down. The ratchet is the discipline that fixes this: when a change genuinely lowers the measured number — a dependency removed, a route split, an image format swapped — the absolute budget is lowered in the same pull request to the new value plus headroom. The saving becomes permanent instead of becoming spare capacity for the next feature. Automating the ratchet is tempting and mostly a mistake: a build that rewrites its own budget file will happily lock in a number produced by a fluke or by a temporarily disabled feature. A one-line diff in review, alongside the change that earned it, is both cheap and auditable. ## Trend as the third leg Neither assertion sees a shape over time. Record every main-branch run's numbers against its commit and chart them. That is what shows a route climbing steadily for six weeks while every individual pull request passed, tells you which week the slope changed, and gives the argument for spending a sprint on it. The per-change gate stops cliffs, the absolute limit defines the contract, and the trend line is what makes slow decay a thing someone can point at.

  • Where should the baseline measurement for a delta check come from?
    From building the base branch in the same job, on the same runner and toolchain, so the two numbers are directly comparable. A cached artifact from the last main-branch build is cheaper but drifts: it misses merges that landed since, and it can silently compare across a dependency or compiler upgrade, producing a delta that belongs to neither branch.
  • Why not just automate the ratchet by rewriting the budget file on every green main-branch build?
    Because the build would lock in whatever number it happened to see — including one produced by a temporarily disabled feature, an unusual code path, or a measurement fluke. Then the next honest pull request fails against a floor nobody agreed to. Lowering the budget as a reviewed one-line diff alongside the change that earned it keeps it deliberate.
  • A delta check on a measured metric like LCP fires constantly. What is wrong?
    The delta threshold is almost certainly inside the measurement's noise floor. Two runs of identical code already differ by more than the allowance, so the check reports noise. Either widen the threshold beyond the observed spread, apply it to a median of several runs, or keep delta checks for deterministic quantities such as bytes and leave timings to absolute warnings.

saying these in an interview costs you the question

  • A per-change delta limit alone prevents growth
  • Absolute budgets need no baseline so they are strictly better
  • Reuse yesterday's stored build as the baseline
  • Raise the budget whenever a pull request fails it
  • Let the pipeline rewrite the budget file automatically

context

resolving links…