skip to content

Signing & Content Trust

Docker's own answer to 'is this the image I built': DOCKER_CONTENT_TRUST with Notary key roles, and cosign signatures stored beside the image in the registry. Asked because a signature binds to a digest, never to a tag, and candidates routinely mix the two up.

part ofDockeroverview, primer and where to startread it →
on this pageshow

questions

4

What does cryptographically signing a container image give you that referencing the image by its `sha256:` digest does not — and what does digest pinning give you that a signature does not?

level: middleimportance: must knowfreq 45%

answer

  1. digest = integrity + immutability, no provenance
  2. signature = authenticated identity over the digest
  3. tag = mutable pointer, never the trust subject
  4. verify → resolve → deploy by digest
  5. unconstrained verify (any signer) proves nothing

basics

~20 s

A digest guarantees integrity: you got exactly those bytes, unchanged. It says nothing about who produced them or whether they were approved. A signature adds authenticated provenance: identity X asserts this digest is good. You need both — verify the signature, then deploy by digest.

solid answer

~50 s

They answer different questions. **Digest** (`image@sha256:…`) is content addressing. It guarantees **integrity and immutability**: whatever you pull hashes to that value or the client rejects it, and unlike a tag it cannot be repointed later. What it cannot tell you is *whether that content should be trusted* — an attacker who slips a malicious image into your pipeline is perfectly happy for you to pin its digest. **Signature** binds a **verified identity and an assertion** to that digest: 'the release workflow of repo X, at this time, produced/approved this digest.' That is provenance and authorisation, which no hash can supply. The two compose. The strong deployment pattern is: verify a signature from an expected identity over a digest, then run **that digest**, never the tag. Pinning without verification trusts whoever filled your pipeline; signing without pinning lets a mutable tag drift to something you never verified.

code

bash · 7 lines
bash
DIGEST=$(crane digest registry.example.com/acme/api:1.4)
cosign sign --yes "registry.example.com/acme/api@${DIGEST}"

cosign verify \
  --certificate-identity-regexp '^https://github.com/acme/api/\.github/workflows/release\.yml@refs/tags/.*$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "registry.example.com/acme/api@${DIGEST}"

go deeper

for a junior

Know that a digest identifies exact content while a signature says who vouched for it, and that production should deploy by digest.

for a middle

Explain immutability versus provenance, why signatures are made over digests, and the verify-then-deploy-by-digest sequence.

for a senior

Discuss identity policy, verification failure modes, mutation of tag to digest at admission, and where attestations extend the model.

for a principal

Own the trust model: which identities count, key/identity rotation, fail-closed policy and break-glass, and how the guarantee holds across mirrors and air gaps.

## Two different questions Supply-chain trust for images has two separable parts: *are these the bytes I mean?* and *should I trust those bytes?* Confusing them is the single most common error in this area. ## What a digest is An OCI image reference like `alpine@sha256:beef…` is **content-addressed**. The digest is the SHA-256 of the image manifest, which itself lists the digests of the config and every layer. Pulling by digest is self-verifying: the client hashes what it received and refuses a mismatch. Consequences: - **Immutability.** A tag is a mutable pointer — `myapp:1.4` can be re-pushed tomorrow at a different digest. A digest cannot be repointed; the same reference always means the same bytes. - **Reproducibility.** Builds and deployments become deterministic across time and mirrors. - **Tamper evidence in transit.** A malicious mirror cannot substitute content for a digest-pinned pull. What it does **not** provide: any statement about origin or approval. A digest is a fingerprint of a stranger. If an attacker gets a build into your registry and your automation pins its digest, everything downstream is faithfully reproducing the compromise. ## What a signature is A signature is a cryptographic statement made **over the digest** by a key or identity: 'this digest was produced by / approved by me.' Verification tells you two things a hash cannot: **who** (authenticated identity) and **that they asserted something** (provenance, and optionally richer attestations — SBOM, SLSA build provenance, vulnerability-scan results). Crucially, signing over the digest is what makes signatures survive re-tagging and mirroring: move `sha256:beef…` to another registry under a different tag and the signature still verifies, because it never mentioned the tag. ## Where each fails alone - **Digest only:** integrity without authorisation. Reproducibly deploys whatever got in. - **Signature only, resolved by tag:** you might verify at one moment and deploy something else. A tag can move between the verification step and the pull; worse, a system that verifies 'the current `latest`' has no stable subject at all. Signature checks must terminate in a digest. - **Signature with a weak identity policy:** verifying that *something* signed the image proves nothing. `cosign verify` without constraining the signer identity and issuer is close to useless — it accepts any signature from anyone the trust root recognises. ## The pattern that works 1. Build produces an image; CI signs it **by digest**. 2. Deploy-time policy resolves the tag to a digest, verifies the signature against an **expected identity** (a specific workflow, key, or subject/issuer pair), and checks any required attestations. 3. The workload manifest records the **digest**, and that is what runs. Admission controllers commonly automate this: they intercept the workload, verify, and *mutate* the tag reference into the verified digest so that what runs is what was verified. That mutation step is the concrete bridge between the two concepts. ## Trust root, not magic Signature verification is only as strong as the trust root and policy: which keys or which identities are acceptable, how they are distributed and rotated, and whether verification fails closed. A signature answers 'who says so'; you still have to decide whose word counts — that decision is policy, and it is where the real engineering is. ## A useful framing for interviews Digest = *what*. Signature = *who says it is okay*. Tag = *convenience, and a mutable one*. Ship all three: humans read tags, policies verify signatures, machines run digests.

  • If we already verify signatures at deploy time, why still pin the digest in the manifest?
    Because a tag can move between verification and pull, so the thing that runs may not be the thing you verified — and on a later restart or reschedule the tag may resolve differently again. Recording the digest makes the deployed artifact stable and auditable. In practice an admission controller verifies and rewrites the tag to the verified digest for exactly this reason.
  • Does re-tagging or copying an image to another registry invalidate its signature?
    No — the signature is made over the image digest, not over the repository or tag, so the same content verifies anywhere. What can break is *locating* the signature: it is stored as a separate object in the source repository, so a copy that moves only the image leaves the signature behind. Use a copy that carries attached artifacts.

A digest is a tamper-proof seal on a parcel: you know nothing was swapped in transit. A signature is the sender's ID on the label: you know who packed it. A sealed parcel from an unknown sender is still not something you want to open.

saying these in an interview costs you the question

  • Claiming a digest proves the image is trustworthy or came from your build system.
  • Running `cosign verify` without constraining signer identity/issuer and calling the image verified.
  • Verifying a tag and then deploying that same tag, leaving a window for the tag to move.
  • Thinking retagging or mirroring breaks a signature, rather than merely misplacing it.
  • Treating signing as a replacement for scanning and provenance attestations rather than a carrier for them.

context

open as a page

Explain cosign's Sigstore 'keyless' signing flow: how a CI job signs a container image without holding any long-lived private key, where the resulting signature is stored, and what a verifier must check.

level: seniorimportance: should knowfreq 38%

basics

~20 s

CI presents its OIDC identity token; Fulcio returns a short-lived certificate binding that identity to an ephemeral key; cosign signs the image digest, logs the signature in the Rekor transparency log, and discards the private key. Verifiers check the certificate chain, the logged inclusion time, and the expected signer identity and issuer.

open as a page

You are asked to make signature verification mandatory before any container image runs in production, across many teams and with substantial third-party images in use. How do you roll that out, and what will break?

level: principalimportance: should knowfreq 28%

basics

~20 s

Enforce at admission, not in CI: a policy engine verifies signatures over digests against expected signer identities and rewrites tags to verified digests. Roll out in audit mode first, sign your own builds, mirror and re-sign third-party images internally, then fail closed per namespace with a documented break-glass path.

open as a page

What does setting the environment variable `DOCKER_CONTENT_TRUST=1` actually change about `docker push` and `docker pull`, how does the key hierarchy behind it work, and why have most teams moved to something else?

level: middleimportance: nice to knowfreq 26%

basics

~20 s

It turns on Docker Content Trust: pushes get signed via a Notary server and pulls or FROM references refuse unsigned or unverified tags. It uses TUF roles — an offline root key, a per-repository targets key, plus server-held snapshot and timestamp keys. Teams moved on because it is tag-scoped, Docker-client-only and painful to operate.

open as a page