How would you enforce, at Kubernetes admission time, that workloads may only run container images from approved registries, pinned by digest, and carrying a valid signature?
answer
- allowlist prefix on containers + initContainers + ephemeralContainers
- startsWith, not contains — anchor the registry host
- tags are mutable → verify then mutate to digest (TOCTOU)
- cosign keyed vs keyless (Fulcio identity, Rekor log)
- verification = registry call in the admission path → cache + timeout + failurePolicy
basics
~20 sUse an admission policy engine. A simple CEL or YAML rule can require the image reference to start with an approved registry prefix and to contain an @sha256: digest. Signature and provenance checks need an engine that can reach the registry — for example Kyverno's verifyImages with cosign — which verifies the signature and rewrites the tag to the verified digest.
solid answer
~60 sThree layered rules: 1. **Registry allowlist.** A pure field check on `spec.containers[*].image` (plus initContainers and ephemeralContainers): the reference must start with an approved registry/repository prefix. This is cheap enough for a built-in ValidatingAdmissionPolicy with CEL. Watch for tricks like `evil.io/registry.company.com/x`; anchor the match at the start and require the host segment. 2. **Digest pinning.** Require `@sha256:` in the reference so the running content is immutable — a tag can be repointed after your check passed. Better than rejecting tags is to *resolve and mutate*: the verification step replaces `repo:tag` with `repo@sha256:...`, closing the time-of-check/time-of-use gap between admission and kubelet pull. 3. **Signature/attestation verification.** Only an engine that can call the registry can do this: Kyverno `verifyImages` (cosign keyed or keyless, plus attestations such as SLSA provenance), or Gatekeeper with an external image-verification provider. Verification is a network call inside admission, so cache results, set a tight timeout, and think hard about `failurePolicy`. Pair it with cluster hygiene: `imagePullPolicy`, controlled `imagePullSecrets`, and background scans to catch pods admitted before the policy existed.
code
yaml · 23 linesapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-images
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 15
rules:
- name: only-signed-company-images
match:
any:
- resources:
kinds: ["Pod"]
verifyImages:
- imageReferences:
- "registry.company.com/*"
mutateDigest: true
required: true
attestors:
- entries:
- keyless:
subject: "https://github.com/company/*/.github/workflows/release.yaml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"go deeper
Know that an admission policy can restrict which registries images come from, and that a digest identifies exact image content while a tag can move.
Describe the three layers — allowlist, digest pinning, signature verification — and that signature checks need an engine like Kyverno rather than plain CEL.
Emphasise verify-then-mutate-to-digest for the TOCTOU gap, keyless signing bound to a pipeline identity, caching, failurePolicy and blast radius, plus background scans for pre-existing workloads.
Frame it as supply-chain policy: mirror-and-re-sign so one trust root covers third-party images, attestation predicates (SLSA level, scan freshness) as the next rung, an exception process, and the availability coupling you are accepting to the registry.
## Why admission is the right control point Everything that runs in a cluster arrives as a pod spec containing image references. Admission is the single choke point where every such spec passes through the API server before it is persisted, which makes it the natural place to answer "is this artifact allowed to run here?". Scanning in CI is necessary but not sufficient: nothing stops someone applying a manifest pointing at an unscanned image, and nothing in CI observes what actually lands on the cluster. ## Layer 1 — where the image came from The simplest and highest-value rule is a registry allowlist on every image field: `spec.containers`, `spec.initContainers`, and `spec.ephemeralContainers` (a frequently forgotten bypass — an attacker with exec rights can attach an arbitrary debug image). Express it as a prefix match on the approved registry host and repository path. Two classic mistakes: matching with `contains()` instead of `startsWith()`, so `attacker.example/registry.company.com/tool` passes; and forgetting that Docker Hub images have implicit hosts (`nginx` means `docker.io/library/nginx`), so a bare name may not match your rule at all and must be handled explicitly. This rule is a pure function of the object, so it belongs in a built-in ValidatingAdmissionPolicy — no webhook, no availability risk. ## Layer 2 — pinning what will actually run A tag is a mutable pointer. If you validate `repo:1.4.2` at admission, nothing prevents that tag being repointed to different content before the kubelet pulls it, or on a later node restart. Digests (`repo@sha256:...`) are content-addressed and immutable, so pinning eliminates that time-of-check/time-of-use window and also makes the running content auditable. Requiring digests by rejection is blunt and unpopular with developers. The better pattern, which Kyverno's image verification does automatically, is **mutate to digest**: the policy resolves the tag once, verifies the artifact it resolved to, then rewrites the pod spec to the digest it verified. The pod runs exactly the bytes that passed the check. ## Layer 3 — proving who built it Provenance means a cryptographic claim about the artifact, produced by your build system and stored alongside the image in the registry (an OCI referrer artifact): - **Signature** — cosign/sigstore signs the image digest. Verification is either **keyed** (you hold a public key, simple, but key distribution and rotation are yours) or **keyless** (the signature carries a short-lived certificate from Fulcio binding it to an OIDC identity such as a specific GitHub Actions workflow, logged in the Rekor transparency log). Keyless lets a policy say "signed by *our release pipeline*", which is a far stronger statement than "signed by some key". - **Attestations** — signed statements *about* the image: SLSA build provenance (which source commit, which builder), an SBOM, or a vulnerability-scan result. A policy can require an attestation to exist, be signed by the expected identity, and satisfy a predicate — e.g. the scan is under 30 days old and reports no critical CVEs. Because verification requires fetching signatures and possibly transparency-log entries, it cannot run in the API server's sandboxed CEL. It needs a webhook-based engine: Kyverno `verifyImages` (built in), or Gatekeeper delegating to an external verification provider. ## Operating it without breaking the cluster Verification puts a **network call to an external registry inside the pod-admission path**. Consequences to plan for: - **Latency and caching.** Cache verification results by digest with a sensible TTL; otherwise a scale-up of 50 replicas triggers 50 verifications. - **failurePolicy.** Fail-closed means a registry outage stops all pod creation, including your own recovery workloads; fail-open means anything can run during that outage. The usual compromise is fail-closed with tight scoping, exclusions for `kube-system` and the policy namespace, and short timeouts. - **Pull credentials.** The verifier needs registry access of its own, configured separately from workload `imagePullSecrets`. - **Mirrors and third-party images.** Base images, ingress controllers, and vendor operators are rarely signed by you. The workable pattern is to mirror them into your own registry and re-sign on the way in, so one policy — "signed by us" — covers everything. - **Existing workloads.** Admission only sees new writes. Run the engine's background scan/PolicyReport to find running pods that would fail, and treat that list as the migration backlog. ## What this does not give you Digest pinning and signatures prove *integrity and origin*, not *safety*: a faithfully signed image full of critical CVEs still passes. That is why attestation-based rules (scan freshness, SLSA level) are the natural next step, and why runtime controls remain necessary.
- Why is requiring a digest more than a style preference — what attack does a tag allow?A tag is a mutable pointer in the registry. A policy that validates repo:1.4.2 at admission time says nothing about the bytes the kubelet pulls later, because whoever can push to that repository can repoint the tag between admission and pull, or before a node restart re-pulls it. Pinning to an immutable content digest closes that time-of-check/time-of-use gap; resolving and mutating to the verified digest during admission does it without burdening developers.
- Your image-verification webhook is fail-closed and your external registry becomes unreachable. What is the blast radius, and how do you limit it in advance?Every pod creation matching the policy fails, so scale-ups, rescheduling after node loss, and rollouts all stall while running pods continue. Limit it in advance by caching verification results by digest, mirroring images into a registry inside your failure domain, excluding kube-system and the policy engine's own namespace, keeping the webhook timeout short, and having a documented break-glass procedure to relax the policy under an incident with a mandatory post-incident audit of what was admitted.
The allowlist checks the supplier's name on the crate, the digest is a tamper-evident seal on the exact crate, and the signature is the supplier's notarised letter proving they packed it. Only all three together tell you what is inside came from where you think.
saying these in an interview costs you the question
- Checking only spec.containers and missing initContainers and ephemeralContainers.
- Using a contains() match on the registry name, which an attacker-controlled path segment defeats.
- Believing a signature proves the image is free of vulnerabilities — it proves origin and integrity only.
- Assuming scanning in CI is sufficient, when nothing prevents applying a manifest that points elsewhere.
- Ignoring that verification adds a registry round trip inside the admission path, with real latency and outage coupling.