skip to content

A CI job writes `executor.json` into the Allure results directory in a step that runs after the upload step. What does the stored run look like, and why does this ordering bug survive so long unnoticed?

level: middleimportance: nice to knowfreq 38%

answer

  1. the push is a snapshot
  2. two authors, one directory
  3. late writes never leave the runner
  4. an optional file fails silently

basics

~20 s

The stored run holds every test and every attachment but not that file, because the upload copied the directory as it stood at that instant. Nothing errors, every test-level detail looks right, so the gap is only noticed when someone needs it.

solid answer

~50 s

An upload is a **snapshot**: it archives the results directory exactly as it stands when the step runs, and anything written into that directory afterwards never leaves the runner, which is then reclaimed. So the store receives a run that is complete in every test-level respect — every `-result.json`, every `-container.json`, every `-attachment` — and missing only the whole-run file the job was supposed to contribute. That is why the bug hides. Nothing fails: whole-run files like `executor.json` are optional inputs, and a run without one still renders and still reports its outcomes correctly. The gap only surfaces weeks later, when somebody opens a stored run and cannot tell which build produced it. The fix is ordering, not content: every file the job contributes must be written before the push, ideally in the same step that performs it.

go deeper

for a junior

Know that the upload copies the directory as it is at that moment, so any file written afterwards never leaves the runner and is lost with the workspace.

for a middle

Explain the split between per-test files written by the adaptor and whole-run files contributed by the job, and why only the second group can arrive after the snapshot.

for a senior

Show that you would close the gap structurally — write and push in one step, plus a pre-push assertion — rather than fixing the step order once and trusting it to stay.

for a principal

Take the position that a shared publish step owning the whole handoff beats every team rediscovering this ordering rule, and decide what the org treats as mandatory in a stored run.

## The upload is a snapshot, not a subscription The delivery step for a results directory copies what is on disk at the moment it runs. It does not watch the directory, it does not re-read it later, and there is no second chance — once the job ends, the workspace is reclaimed and anything written after the snapshot is gone. That makes the ordering of writes against the push a real, load-bearing property of the job, and it is one that nothing in the tooling enforces. ## Two kinds of file, written by two different actors This bug exists because a results directory has two populations of file with different authors: - **Per-test files**, written by the adaptor **inside the test process**: `-result.json` for each test, `-container.json` for the fixtures around them, `-attachment` files for the bytes. These are finished by the time the suite exits, so they are always present when the push runs. - **Whole-run files**, contributed by the **job** rather than by any test: `executor.json`, `environment.properties`, `categories.json`. Nothing in the test run produces them. Some step in the pipeline has to write them, and where that step sits relative to the push is a choice somebody makes — often without realising it is a choice. Only the second group can be late. And because the first group is never late, the run looks fine. ## Why the gap is silent Three things conspire: 1. **The push succeeds.** The directory is non-empty and well formed. There is nothing for the upload to complain about. 2. **The missing file is optional.** A results directory without `executor.json` is perfectly valid input; consumers simply have nothing to show for that section rather than raising an error. Absence is not a failure condition anywhere on this path. 3. **Everything a reader checks first is intact.** Counts, statuses, durations, failure messages and attachments are all per-test files, all present. Someone opening the run sees a normal run. So the defect has no symptom on the day. Its symptom appears the first time someone asks a question the missing file was there to answer — which build, which job, which environment produced this run — and by then dozens of stored runs have the same hole and none of them can be repaired, because the workspaces are long gone. ## Where the ordering actually breaks The common shapes are all mundane: - The publish step was added first, and the step that writes the whole-run files was appended later at the end of the job, where new steps naturally get added. - The job was refactored and the steps were reordered by someone reasoning about the suite, not about the snapshot. - The whole-run files are written by a step that only runs conditionally, so the ordering is correct on some runs and not on others — which produces the worst version of this, an intermittent gap nobody can reproduce. ## Fixes, in order of how well they hold 1. **Write and push in one step.** If the same step that performs the upload also writes the whole-run files immediately before archiving, the two cannot be separated by a later edit. This is the only fix that survives someone reordering the job a year from now. 2. **Assert the directory's contents before the push.** Check that the files the job promised are actually there, and fail the publish step if they are not. This turns a silent gap into a red step and costs almost nothing. 3. **Keep the ordering explicit in the job.** If the writes must stay in their own step, make the dependency visible rather than relying on the reader inferring it from step order. The general rule is worth stating on its own: **anything the job contributes to a results directory must land before the snapshot.** The push is the deadline for the whole directory, not just for the parts the tests wrote, and the only reliable way to keep a deadline is to not let anything be scheduled after it.

  • Where in the job would you put the write so this ordering cannot drift again?
    Inside the step that performs the upload, immediately before it archives the directory. Keeping the write and the snapshot in one unit means no later edit can insert anything between them or move one without the other. A separate step is fine only if a pre-push assertion backs it up.
  • What check would catch this before a human eventually notices?
    Assert on the directory just before the push: it exists, it is non-empty, and it contains the whole-run files this job promised to contribute. Fail the publish step when the assertion does not hold. It is one command, and it converts a defect with no same-day symptom into an immediately red step.

saying these in an interview costs you the question

  • Thinks the upload rereads the directory or picks up later writes
  • Assumes a missing whole-run file makes the push fail
  • Expects the next run's upload to carry the stragglers
  • Believes the results server can reconstruct the missing file