How would you roll out `docker buildx` SBOM and provenance attestations across every service image?

### Start by naming what you actually get The honest benefit of turning attestations on everywhere is **answerability**. Given a running image digest — say the invoice-rendering worker capped at 128 MiB that started failing at 03:14 — one registry query tells you which source revision built it, on which builder, when, and what a generator saw inside it. Today that question is usually answered by archaeology across CI logs that have aged out. That is worth real money during an incident and at audit time. What you do **not** get by producing attestations is any security guarantee. An unsigned record in a registry is writable by anyone who can push to the repository. If the plan is to gate deployments on these claims, that is a different, larger programme with its own signing identity and enforcement point, and it should be sequenced after production, not conflated with it. Saying this out loud early is what stops the effort being sold as something it is not. ### The rollout has a hard technical precondition Attestations only exist inside an image index, so the first question for every pipeline is not "which flags" but "can this build path even carry them". Three shapes work: a `docker-container` buildx builder pushing to a registry with `--push`; the OCI exporter writing an index to disk; or the engine's containerd image store. The shape that does not work is the classic Docker image store — a build that ends with `--load`, or an artefact shipped as a `docker save` tarball, loses the attestations with no error that anybody reads. In a large estate this is where most of the effort goes, because it means touching builder provisioning on the runners, not just the build command. ### Set it once, centrally Do not ask forty teams to add two flags. Put the defaults in whatever is shared: the CI build action, the container-build template, the internal wrapper script. Teams then inherit `--sbom=true` and a provenance mode, and opting out becomes a visible, reviewable exception rather than the natural consequence of nobody hearing about the initiative. Pin the SBOM generator image explicitly (`--sbom=generator=<image>@sha256:...`) in that shared place, so the inventory for the same source produces comparable output over time instead of shifting when the default generator moves. ### Choose modes by audience, not by ambition - **Minimal provenance everywhere.** Cheap, and it carries the fields that answer the triage question. - **`mode=max` where build arguments have been audited.** Maximum provenance publishes build-arg values; on an internal repository with clean pipelines that is fine and genuinely useful, and on a public one it is a disclosure waiting to happen. Pair the switch with a check that fails a build whose build-arg names look secret-shaped, and with a push to move credentials to `RUN --mount=type=secret`. - **SBOM on anything published outside the team.** Generation costs build minutes on large images; measure it on the biggest image you own before promising it is free. ### Expect the breakage to be in the tooling, not the images Running containers are unaffected — an attested image pulls and runs exactly as before. What breaks is everything that assumed a tag resolves to one image: registry UIs that display a stray unknown-platform row and generate support tickets, promotion scripts that copy a manifest rather than an index, retention rules that count members, and older or self-hosted registries with quirks around index members. So roll out by blast radius: one pipeline family first, then the noisy internal ones, then anything customer-facing. Watch the registry, not the build logs. ### Sequence and stopping point A defensible plan is four steps. **One:** move builds onto index-capable builders and confirm with `docker buildx imagetools inspect --raw` that the extra manifests actually arrive. **Two:** default minimal provenance in the shared build path and let it soak. **Three:** add SBOM generation for published images, with the generator pinned. **Four:** enable `mode=max` on the subset of repositories whose build arguments have been audited. Only after that does it make sense to ask who signs the attestations and where a check runs. The judgement an interviewer is listening for is that you stop at the point where the cost turns. Attestations on every image are a small, permanent tax on the build path with a clear triage payoff. Making them load-bearing — signed, verified, blocking — is an organisational commitment with break-glass paths, exceptions that never expire, and third-party images that will never comply. Those are worth doing, and they are not this task; treating them as one is how a six-week change becomes an eighteen-month programme that ships nothing.