skip to content

How do you verify cosign signatures inside an air-gapped enclave that cannot reach Fulcio or Rekor?

level: seniorimportance: nice to knowfreq 30%

answer

  1. signing needs the network, verifying need not
  2. carry the proof with the artifact
  3. cache the trust root beforehand
  4. bundle: signature, certificate, log entry
  5. initialize from a staged mirror

basics

~20 s

Sign outside the enclave and carry the proof in. A bundle ships the signature, the signing certificate and the transparency-log entry beside the artifact, and a trust root cached in advance lets cosign check them with no network call.

solid answer

~50 s

Split signing from verifying. Keyless signing needs an OIDC provider, the certificate authority and the log, so it happens on the connected side, where each artifact is signed with `--bundle` so its signature, certificate and log entry travel as one file. Verification inside the enclave then needs two things: the per-artifact bundle, and a local copy of the trust root — the CA's root and intermediate certificates and the log's public key — which cosign holds in a local cache you populate with `cosign initialize` pointed at a mirrored trust-root repository. With both present, verification is pure arithmetic: chain the certificate to the cached root, check the identity and issuer against what you expect, check the signature over the digest, check the log entry against the cached key. The operational work is refreshing that cached root before its metadata expires and re-importing it across the gap.

go deeper

for a junior

Know that verification needs two inputs: the signature material shipped with the artifact, and a local copy of the trust root, and that neither has to be fetched at verification time.

for a middle

Explain what a bundle contains and why carrying the log entry inside it is what removes the network call during verification.

for a senior

Show the operating discipline: staging and refreshing the trust root on a schedule, verifying at the point of use, and failing closed when signature material is absent.

for a principal

Decide whether an isolated estate mirrors an external chain of trust or operates its own signing infrastructure, and be able to defend the running cost of that choice.

## Signing needs the network; verifying does not have to An air-gapped estate — say a mirror of OS packages pulled into a classified enclave — cannot talk to a public certificate authority, an identity provider or a transparency log. That rules out *signing* inside the enclave with the keyless flow, because every step of it is a network exchange. It does not rule out *verifying*, because verification is arithmetic over material you can carry in. ### Do the signing on the connected side Packages are signed where the build runs, outside the gap. Signing each with `--bundle` produces one file per artifact carrying the signature, the signing certificate and the transparency-log entry, rather than three loose strings that someone will lose while moving media across the boundary. Artifact and bundle then cross together, and the import process treats a package without a bundle as a package that cannot be admitted. ### Cache the trust root before you need it The second half is the root of trust: the certificate authority's root and intermediate certificates, and the public key of the transparency log. cosign fetches these from a signed trust-root repository and caches them locally; `cosign initialize` is the command that populates or refreshes that cache, and it accepts a mirror, which is how you feed it from a copy staged inside the enclave rather than from the public internet. This material is signed and time-bounded — that is deliberate, because it is how a compromised or stale root is prevented from being trusted forever. The practical consequence for an air gap is a chore with a deadline: someone must periodically refresh the trust root on the connected side, move it across, and re-initialise inside, or verification will one day start failing on artifacts that are perfectly good. Discovering that on the day the metadata expires, during a deployment, is the classic way this bites. ### What verification then actually does With bundle and cached root present, cosign can complete every check locally: the certificate chains to the cached root; the identity and issuer in the certificate match the ones you pass; the signature is valid over the artifact's digest; the log entry in the bundle is consistent and signed by the key you cached. No call leaves the enclave. Note that this only works because the bundle carried the log entry — a bare signature and certificate would force a lookup you cannot make. ### The threat this closes The interesting adversary here is not an anonymous outsider — there is no route in — but a compromised operator of the mirror, or someone on the enclave's own network who can write to the package repository. They can replace an RPM with a modified one. What they cannot do is produce a bundle that verifies against the expected signing identity, because obtaining a certificate for that identity requires the very network exchange the enclave forbids. So the control works precisely because signing is hard to do inside; the asset protected is the integrity and availability of a controlled estate that has no external help if it is subverted. Two design rules follow. First, verify at the point of use — at install, or before a service loads the file — not only once at import, because import-time verification protects only the boundary and says nothing about what happened to the repository afterwards. Second, fail closed on missing material: a deleted bundle must block the install, never be read as "nothing to verify". ### The alternatives, and when they are better If artifacts genuinely originate *inside* the enclave, keyless signing is not available to them at all and you have two options: run a private Sigstore deployment within the enclave, which gives you the same shape at the cost of operating a CA, an identity provider and a log; or sign with a key pair, accepting the key-management burden that keyless was invented to remove. Neither is wrong — the choice turns on how many artifacts originate inside and whether you can operate the services responsibly.

  • What breaks when the cached trust root is never refreshed?
    Its metadata is time-bounded on purpose, so that a stale or compromised root cannot be trusted indefinitely. Left alone, it eventually expires and verification starts failing on artifacts that are entirely valid. Treat refreshing and re-importing it as a scheduled operational task with an owner, not something to discover during a deployment.
  • Why should you verify at install time and not only when packages cross the boundary?
    Import-time verification protects the boundary and nothing after it. Anyone who can write to the internal repository afterwards — a compromised mirror operator, someone on the local network — is entirely unaffected by a check that already happened. Verifying at the point of use makes tampering inside the enclave detectable at the moment it matters.
  • Could you run keyless signing inside the enclave instead?
    Only by standing up a private Sigstore deployment there: your own certificate authority, identity provider and transparency log. That is a real option for an estate producing many internal artifacts, but you now operate those services and own their availability and key protection. For a handful of artifacts, a key pair is usually the honest trade.

It is like carrying a sealed certificate of authenticity across a border where you cannot phone the issuer to confirm it, so you bring a copy of the issuer's seal with you and check the wax yourself.

saying these in an interview costs you the question

  • Thinks the artifact bytes are stored in the transparency log
  • Assumes verification always requires a live network call
  • Never refreshes the cached trust root, then blames cosign
  • Verifies only at import and not at install time
  • Believes keyless signing can be performed with no network access

context