Why must the command that runs a suite exit non-zero when a case fails, and what does an always-zero exit cost?
answer
- The pipeline reads one number
- Evidence is not the gate
- Crashes and empty runs must fail too
- A wrapper can swallow the status
basics
~10 sA pipeline reads only the invoked command's exit status to decide pass or fail. An always-zero exit makes every stage green regardless of results, so failures are recorded in reports nobody blocks on.
solid answer
~40 sThe pipeline that invokes a suite is a process supervisor: it starts one command and reads one number when it ends. Zero means the stage passed; anything else means it failed. That number is the whole gate - the machine-readable results document the run also writes is evidence for people and dashboards, not the decision. The entry point must therefore exit non-zero whenever a blocking case did not pass, **and** for its own failures: a crash before the first case, a bad configuration, zero cases collected. An always-zero exit usually arrives by accident - a wrapper that swallows the status, or a reporting step that runs last and succeeds. The stage then goes green forever, broken changes advance, and the suite rots while its dashboard keeps filling up.
code
pseudocode · 15 linesrun_suite(selection):
results = execute(selection)
write_results_document(results)
if results.collected == 0:
return EXIT_NO_CASES # an empty run is not a pass
if results.crashed:
return EXIT_SUITE_ERROR # could not run, not "nothing failed"
if results.blocking_failures > 0:
return EXIT_FAILURES
return EXIT_OK
# the pipeline invokes this one command and reads only its return value;
# artefact publishing runs afterwards and never overwrites the verdictgo deeper
Be ready to say what a delivery pipeline actually checks: the command it ran either ended successfully or it did not. Know that a failing case must make that command report failure, or the stage goes green anyway.
Explain the mapping in both directions - which run outcomes must produce a failing status, including a crash before the first case and a run that collected nothing, and why the results document is evidence rather than the gate.
Show how an always-zero exit gets in: a wrapper returning a later step's status, an open-ended non-blocking exemption, or a parser that only fails when it finds failures. Be ready to say how you would detect and prove it.
Own the argument that the gate itself needs verifying. Talk about asserting the entry point's failure status as part of the build, bounding advisory exemptions with an expiry, and what a decorative stage costs a team's trust once discovered.
A delivery pipeline does not understand your suite. It starts a process, waits for it to end, and reads one small integer: the **exit status**. Zero means the command succeeded; any other value means it failed, and the stage fails with it. Everything else a run produces - a machine-readable results document, logs, captured artefacts, timings - is *evidence*, read afterwards by people and dashboards. The exit status is the only thing the pipeline **gates** on. That asymmetry is the whole subject of the entry contract. ## What the entry point promises A suite's entry point is the one command the pipeline invokes. Its contract has two halves: - **Zero exactly when the run is a pass.** Every blocking case ran and passed, the suite started cleanly, and the results document was written. - **Non-zero for everything else**, including the failures that are not case failures at all. The second half is the one teams get wrong. A suite that only fails when an assertion fails is silent about all the ways a run can be *worthless* rather than *red*. | What happened | Exit status | Why | | --- | --- | --- | | All blocking cases passed | zero | Nothing should stop the stage | | A blocking case failed | non-zero | The change is not safe to advance | | The process crashed before the first case | non-zero | No failures is not the same as no problems | | A filter selected zero cases | non-zero | An empty run reports a perfect pass rate | | The run exceeded its own deadline | non-zero | A timeout is a result, not the absence of one | | Only checks classified as non-blocking failed | zero, failures still reported | The classification, not the assertion, decides | The row that surprises people is **zero cases collected**. A mistyped selector, a renamed directory, or a filter expression that matches nothing produces a run with no failures - and a naive entry point exits zero. The suite has stopped testing anything, and the pipeline is congratulating it. A minimum-collected-count check is a few lines and removes the whole class. ## How an always-zero exit gets in Nobody sets out to build a gate that cannot fail. It arrives by accident, usually one of these ways: 1. **A wrapper swallows the status.** The suite runs inside a script that continues to a reporting or upload step, and the script returns *that* step's status. The upload succeeds, so the stage is green. 2. **A step is flagged to continue past failures** while an incident is triaged, and the flag is never removed. Six months later nobody remembers the stage is advisory. 3. **The results document becomes the gate.** A later step parses the run's output and is supposed to fail the stage, but it only fails when it *finds* failures - so a run that produced no document at all sails through. 4. **Failures are converted to warnings** to keep a branch moving during a migration, and the migration never formally ends. The common thread: the failure signal is passed through one more layer than it needs, and every layer is a chance to drop it. ## What an always-zero exit costs The immediate cost is obvious - a broken change advances. The compounding costs are worse: - **Silent decay.** Cases start failing and nobody is told. By the time anyone looks, dozens are red and the repair is a project rather than a fix. - **False evidence.** Reports and trend charts keep filling with data, so the suite looks healthy from outside. That is worse than having no suite, because people make decisions on it. - **A gate that must be re-earned.** Once a team discovers the stage was decorative, they distrust it even after the wiring is fixed, and the argument to keep it blocking has to be won again. - **An undated regression window.** You cannot say when the suite stopped gating, so you cannot bound which changes went out unverified. ## Making the contract hard to break Treat the exit status as a tested property of the suite, not a convention people remember: - **Keep the invocation one command.** The fewer layers between the run and the pipeline, the fewer places the status can be dropped. If artefacts must be published afterwards, publish them in a step that runs *regardless* of the outcome and never owns the verdict. - **Assert the contract.** A small check that invokes the entry point against a deliberately failing case and asserts a non-zero status catches the wrapper bug permanently, and catches it in the change that introduced it. - **Fail on an empty run.** Require a minimum collected count, or compare the collected count against a recorded baseline. - **Distinguish a few exit values if it earns its keep** - one for "blocking cases failed", another for "the suite could not run" - so the pipeline can tell a real regression from broken wiring without parsing logs. Keep the vocabulary small; the pipeline only branches on what it agreed to understand. - **Never leave an advisory period open-ended.** If a stage is temporarily non-blocking, the exemption carries a name and an expiry.
- Your suite writes a results document but the stage still passes on a red run. Where do you look first?At what the stage actually invokes. Most often a wrapper script runs the suite, then a reporting or publishing step, and returns the last step's status. Reproduce it by invoking the entry point against a case you know fails and inspecting the status it returns. Then move artefact handling into a step that always runs and never owns the verdict.
- Is it worth using several distinct non-zero values rather than a single one?Sometimes. One value for “blocking cases failed” and another for “the suite could not run” lets the pipeline branch without parsing logs - for instance re-invoking the whole run only for infrastructure faults, never for assertion failures. Keep the vocabulary to a handful; anything richer belongs in the results document, since the pipeline can only branch on what it agreed to understand.
- How would you prove the entry point still fails the stage after someone refactors it?Add a check that invokes the entry point on a fixed, deliberately failing case and asserts a non-zero status, plus one that points the selection at nothing and asserts the empty-run status. Both run alongside everything else, so the gate's own gate cannot rot quietly.
A smoke alarm wired to a lamp that was never connected: the sensor still detects smoke and the log it writes is accurate, but nothing in the building ever reacts.
saying these in an interview costs you the question
- Thinks the results document, not the exit status, fails the stage
- Exits zero when the suite crashes before any case runs
- Treats a run that collected zero cases as a pass
- Wraps the suite in a script returning the last step's status
- Leaves a stage non-blocking indefinitely after an incident