skip to content

One step in an otherwise hermetic build graph downloads a reference data file - how do you find it and what replaces it?

level: seniorimportance: should knowfreq 42%

answer

  1. the claim is transitive across the graph
  2. detect by denying, not by reading
  3. an unlisted input in the lineage
  4. name the content, not the location
  5. fetch phase before the sealed phase

basics

~20 s

Find it by running the build in a sandbox with outbound access denied and seeing which step fails. Replace the fetch with a declared input pinned by content digest, fetched and verified in a separate phase before the sealed build runs.

solid answer

~60 s

One escape hatch invalidates the property for the whole graph. Every downstream target now depends on whatever that host served at that moment, so the artifact is no longer a function of the declared inputs, and an attacker in that network position - or on that upstream host - can change what you shipped without touching a line of your source. Detection is best done by denial, not by inspection: run the build with egress blocked at the sandbox and see what breaks; that is definitive where grepping build rules for shell-outs is not. Complement it with a comparison of the recorded declared inputs against what the step actually opened. The replacement is to make the file a first-class input: record its URL and expected digest in the build definition, fetch it in an explicit pre-build phase that verifies the digest and fails on mismatch, or vendor it into an internal repository and reference it by digest. Then the build step runs sealed, and a change in that data becomes a reviewable change to a declaration.

go deeper

for a junior

Understand that a build step downloading a file at build time means the artifact depends on something nobody declared, and that the fix is to turn that file into a normal, pinned dependency.

for a middle

Explain why the property is transitive across the graph and describe the two-phase shape: fetch and verify digests first, then run the build with no outbound access.

for a senior

Show the diagnostic instinct - detect by denying egress rather than reading rules - and the replacement design, including a mirrored internal copy referenced by digest and a CI guard that keeps the hole closed.

for a principal

Be ready to say what you tell stakeholders while the gap is open: a partial hermeticity claim is not a weaker guarantee, and the honest options are to close it or to stop making the claim until it is closed.

## Why one step matters for the whole graph A hermetic build graph makes a single claim: the output is determined by the declared inputs. That claim is not per-step, it is transitive. If one rule fetches a file at build time, every target downstream of that rule inherits an undeclared input, and the claim is false for all of them. Teams often reason that the fetch is "just reference data" - a currency table, a holiday calendar, a set of tolerance constants - which is exactly what makes it dangerous, because that data is usually consumed directly into the numbers the artifact produces. Consider a pricing library at a trading firm: a generated rule pulls a reference data file at build time and compiles it in. Nobody reviews that file, it has no version in the repository, and there is generally no integrity check on the download because the fetch was added as a convenience. Whoever controls that host, or anyone able to intercept the connection from the build network, can change what the shipped library computes. The source tree looks untouched, the review history looks clean, and two builds of the same commit differ in ways nothing in the repository explains. The asset at risk here is not customer data - it is the correctness of a priced artifact and the trade-secret logic around it, and the attacker position is a compromised upstream host or someone with a network position between the runner and it. ## Finding it **Deny and observe.** The reliable method is to execute the build inside a sandbox that permits no outbound connections and watch what fails. This is definitive: anything that needs the network says so by breaking. Reading build rules is not, because a fetch can hide behind a generated rule, a wrapper script, a plugin, a package manager invoked as a subprocess, or a test fixture that quietly warms a cache. **Diff the declaration against reality.** If the build system records the inputs it believes each action consumed, compare that record against what the action actually opened. A file that appears in the output's lineage but in nobody's declared input list is the tell. **Watch for the accidental variants.** The obvious `curl` in a rule is the easy case. The harder ones are a package manager run inside a build step, a code generator that checks for updates, a test that reaches a staging service, and a toolchain plugin that downloads its own components on first use. Note that some builds do legitimately need to reach the network, and deciding which and under what policy is a separate discipline from this one. The point here is that a build claiming hermeticity cannot have unlisted fetches at all. ## Replacing it The general move is to turn the fetch into a declared, content-addressed input and move the fetching out of the sealed phase. 1. **Name the content, not the location.** Record the URL together with an expected cryptographic digest of the file. The URL says where to get a copy; the digest says which bytes are acceptable. A fetch that returns different bytes fails loudly instead of silently changing your artifact. 2. **Fetch in a distinct phase.** Resolve and download every external input first, verify each digest, and place the results in an input tree. Then run the build with no outbound access. A missing input becomes a hard failure at fetch time, in a phase where a failure is easy to read. 3. **Prefer a copy you control.** For anything that matters, mirror the file into an internal repository and reference the internal copy by digest. That removes the dependence on an external host's availability and on its willingness to keep serving the same bytes. 4. **Make updates reviewable.** Bumping the digest becomes a normal change: a diff, a reviewer, a history. That is the real win. The data stops being ambient and starts being versioned, which also means you can answer "which reference data went into the release we shipped in March". 5. **Guard the regression.** Keep running the build with egress denied in CI so the next convenient download fails immediately rather than being discovered a year later. ## The judgment to show The instinct to look for is refusing to accept a partial claim. "Hermetic except for one step" is not a weaker guarantee, it is no guarantee, because the exception is exactly where an adversary or an accident lands. The right response is either to close the hole or to stop claiming the property until it is closed.

  • Why is blocking egress a better detection method than auditing the build rules?
    Because a fetch hides in places an audit misses: a generated rule, a wrapper script, a package manager invoked as a subprocess, a plugin downloading its own components, a test hitting a staging service. Denying the network makes every one of them announce itself by failing. The audit tells you what you thought was there; the denial tells you what is there.
  • The fetched file has no published digest and changes weekly upstream. What do you do?
    Pull it deliberately rather than incidentally: fetch it on a schedule outside the build, review the diff, store the accepted copy in an internal repository, and have the build reference that copy by digest. The data still updates weekly, but each update is an explicit, reviewable change with a history, instead of an invisible input that varies per build.
  • What evidence would show that a past release was built with an undeclared fetch?
    The build's own record of resolved inputs is the first place to look: an artifact in the output's lineage that appears in no declared input list. Failing that, rebuild the same commit with egress denied and see whether it still builds, and compare two rebuilds for differences that nothing in the repository explains.

saying these in an interview costs you the question

  • Treats one network step as a minor exception to hermeticity
  • Relies on grepping build files to find fetches
  • Adds a retry or a cache instead of declaring the input
  • Pins the URL but never the expected content digest
  • Assumes reference data is harmless because it is not code

context

resolving links…