skip to content

Why can a suite pass on the build your pipeline makes for testing and fail on the build users install?

level: juniorimportance: should knowfreq 46%

answer

  1. Two different artefacts, not one build
  2. Optimization removes code reached indirectly
  3. Test hooks and logging compiled out
  4. Release signing changes what the app inherits
  5. Smoke-test the artefact you will ship

basics

~20 s

They are different artefacts. The shipped build is optimized, has unused code stripped, is signed with the release identity, drops test hooks and verbose logging, and points at production configuration — so code the test build kept alive can vanish from it.

solid answer

~50 s

A testable build and a shipped build are compiled and packaged differently, so passing on one is not evidence about the other. The usual differences: the shipped build is optimized with unused code and symbols removed, which can strip anything reached indirectly rather than by a direct call; assertions, verbose logging and every test-only entry point are compiled out; it carries the release signing identity, which changes what secure storage and platform capabilities it inherits; and its configuration points at production rather than a staging deployment. The rule is to run at least a smoke pass against a **release-candidate artefact built exactly as the shipped one is**, driven only through entry points a user has — then ship that same file rather than rebuilding it. The deep suite stays on the testable build, because it needs the hooks the candidate must not have.

code

pseudocode · 11 lines
pseudocode
# WRONG: the suite never touches what users receive
testable = build(config = TESTABLE)   # hooks on, logging on, staging target
run_suite(testable)
ship(build(config = RELEASE))         # a different file, never exercised

# RIGHT: exercise the artefact you will ship, then ship that file
candidate = build(config = RELEASE)   # optimized, stripped, release-signed
run_suite(candidate, subset = SMOKE, entry = PUBLIC_ONLY)
run_suite(testable, subset = FULL)    # breadth stays on the testable build
require(fingerprint(candidate) == fingerprint(delivered_artefact))
ship(candidate)

go deeper

for a junior

Recall that the build you test and the build users install are produced with different settings, and that logging, assertions and test hooks are normally absent from the shipped one.

for a middle

Explain which differences can change behaviour: unused-code and symbol removal breaking anything reached indirectly, compiled-out assertions, release signing changing what the app may access, and configuration pointing elsewhere.

for a senior

Show the pipeline discipline: one release candidate built once, smoke-tested through public entry points, shipped without a rebuild, with an automated check that no test-only entry point survived into it.

for a principal

Own the parity rule as policy — nothing ships that was not exercised as the exact artefact — and decide what the team gives up for it, since a stripped candidate cannot use the back doors the rest of the suite depends on.

## Two artefacts, not one program A suite that goes green has proved something about **one artefact**: the exact file that was installed and run. It has proved nothing about a different file compiled from the same source with different settings. Yet the normal pipeline does exactly that — it builds a testable version, runs everything against it, and then produces a separate release build to hand to users. Those two files differ in ways that change behaviour, which is why 'it passed' and 'it works for users' can both be true statements about different things. ## What actually differs | Difference | Testable build | Shipped build | Why it can change behaviour | |---|---|---|---| | Optimization and unused-code removal | Usually off | Usually on | Anything reached indirectly rather than from a call site can look unreferenced and be removed | | Symbol and name mangling | Off | Often on | Logic that matches a type or member by its name as text stops matching | | Assertions and verbose logging | Compiled in | Compiled out | Checks that were quietly catching bad input stop running | | Test-only entry points | Present | Must be absent | Anything the suite drove through a back door is untested in the shipped shape | | Signing identity | Development | Release | What the app may read from secure storage, and which platform capabilities it inherits, can change | | Configuration | A staging deployment, seeded data | Production | Endpoints, keys and limits all differ | Every row is a real failure mode, and the first is the classic one: a feature reached by name or by lookup rather than from a call site disappears from the optimized build, and only from the optimized build. ## What that means for the suite 1. **Build the release candidate once, exactly as it will ship.** Same optimization, same stripping, same signing identity, same configuration. 2. **Run a smoke pass against that artefact**, driven only through entry points a user has — launch, sign in, the two or three flows the product exists for, and one deliberate error path. 3. **Ship that same file.** Not a rebuild. If the pipeline rebuilds after testing, the artefact users receive has no evidence behind it at all; compare a fingerprint of the candidate against what is delivered, and fail the release if they differ. 4. **Keep the deep suite on the testable build.** Cases needing seeded data, stand-in dependencies or a back door cannot run against the candidate by definition — and that is the point of the candidate. The split is deliberate: breadth on the build that can be instrumented, parity on the build that ships. ## The smoke pass is about parity, not coverage It is tempting to grow the candidate pass until it duplicates the suite. Resist it. Its question is narrow: **did optimization, stripping, signing or configuration change anything?** A handful of end-to-end flows answers that. Anything more re-tests logic that was already proved, and it pushes the team toward leaving hooks in the shipped build so the bigger suite can run there — which reintroduces the exact difference the candidate exists to detect. ## Keeping the back doors out A test-only entry point that is present but disabled by a flag is one misconfiguration away from being live. Prefer structural absence: - Compile the hook only into the testable configuration, so the shipped artefact cannot contain it even if a flag is wrong. - Add a check on the candidate that the hook is genuinely gone — inspect the artefact, or assert the entry point does not respond. - Treat a hook that genuinely must exist at runtime as a product feature with its own authorization, not as test scaffolding that happened to survive. ## The rule to state in an interview Test the artefact you will ship, ship the artefact you tested, and know exactly which differences the release settings introduce. When something passes in testing and fails in the field, the first question is not what changed in the code — it is **which build was that**. A team that can answer instantly, because there is one candidate carrying a fingerprint and a smoke result, spends the investigation on the defect itself. A team that cannot spend it re-establishing what their users are even running, which is the expensive half of most release incidents.

  • Which part of the suite do you run against the release candidate, given it has no test hooks?
    A smoke subset driven entirely through entry points a user has: launch, sign in, the two or three flows the product exists for, and one deliberate error path. Anything needing seeded data, a stand-in dependency or a back door stays on the testable build. The candidate pass is about artefact parity — did optimization, stripping or signing change behaviour — not about breadth.
  • How do you keep a test-only entry point from reaching users?
    Make its absence structural rather than conditional: compile it only into the testable configuration, so the shipped artefact cannot contain it even if a flag is set wrong. Then check the candidate to confirm it is genuinely gone. A runtime flag guarding a hook that is still present in the shipped file is one misconfiguration away from being reachable.

saying these in an interview costs you the question

  • Treats the testable build and the shipped build as the same thing
  • Rebuilds the artefact after testing it and ships the rebuild
  • Assumes optimization and code stripping cannot change behaviour
  • Leaves test-only entry points in the shipped build behind a flag
  • Discovers the configuration pointed at staging only after release