Your team scans images in CI and the last build passed clean, yet weeks later the same deployed image shows new critical CVEs. Why does this happen, and what base-image patch-and-rebuild strategy keeps images current?

seniorshould knowfreq 40%

answer

image is frozen; CVE feed keeps moving
clean build = clean on that day only
rebuild on cadence (nightly/weekly)
pin by digest BUT auto-bump (Renovate/Dependabot)
distroless/slim + multi-stage; re-scan deployed -> trigger rebuild

basics

~20 s

New CVEs are disclosed continuously against packages already baked into your image, and a passing build only reflects the feed on build day. Fix it by rebuilding regularly (scheduled/nightly), pinning base images by digest but bumping them on a cadence, using minimal/distroless bases to shrink the surface, and re-scanning deployed images so new CVEs trigger a rebuild.

solid answer

~60 s

An image is a frozen snapshot; the world's vulnerability knowledge is not. A build that passed clean only means 'no known CVEs against these versions on that day'. Weeks later, new CVEs get disclosed against the very OS packages and libraries already inside the image, so a re-scan lights up red though nothing in your repo changed.

The strategy is to stop treating images as build-once artifacts:

Rebuild on a cadence. Schedule periodic rebuilds (e.g. nightly/weekly) so images pick up patched base layers even without code changes.
Pin, but bump. Pin the base image by digest for reproducibility, but automate raising that digest (Renovate/Dependabot) so pins don't rot into ancient bases.
Shrink the surface. Use minimal or distroless bases and multi-stage builds; fewer packages means fewer future CVEs to match.
Re-scan what's deployed. Continuously scan running images / stored SBOMs against fresh feeds; a new critical triggers a rebuild-and-redeploy, ideally automatically.

So cadence + minimal base + continuous re-scan turns 'green at merge' into 'stays current'.

An image is a **frozen** snapshot; the world's vulnerability knowledge is not. A build that passed clean only means 'no known CVEs against these versions **on that day**'. Weeks later, new CVEs get disclosed against the very OS packages and libraries already inside the image, so a re-scan lights up red though nothing in your repo changed. The strategy is to stop treating images as build-once artifacts: 1. **Rebuild on a cadence.** Schedule periodic rebuilds (e.g. nightly/weekly) so images pick up patched base layers even without code changes. 2. **Pin, but bump.** Pin the base image by **digest** for reproducibility, but automate raising that digest (Renovate/Dependabot) so pins don't rot into ancient bases. 3. **Shrink the surface.** Use minimal or **distroless** bases and multi-stage builds; fewer packages means fewer future CVEs to match. 4. **Re-scan what's deployed.** Continuously scan running images / stored SBOMs against fresh feeds; a new critical triggers a rebuild-and-redeploy, ideally automatically. So cadence + minimal base + continuous re-scan turns 'green at merge' into 'stays current'.

code

dockerfile · 12 lines

dockerfile

# Pin by digest for reproducibility; automation bumps this over time
FROM debian:12-slim@sha256:aabbcc...
# ...build...

# Or shrink surface with distroless final stage (multi-stage)
FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app .
FROM gcr.io/distroless/static-debian12
COPY --from=build /app /app
ENTRYPOINT ["/app"]

go deeper

for a junior

Understand that new CVEs appear over time so a clean image needs rebuilding to stay clean.

for a middle

Explain immutability vs a moving feed, and name cadence + minimal base as the fixes.

for a senior

Design the loop: scheduled rebuilds, digest-pin-plus-auto-bump, distroless, continuous re-scan triggering rebuilds.

for a principal

Own it as a program with severity-based remediation SLAs, automated base-bump-to-deploy pipelines, and fleet-wide freshness metrics.

Why a clean build goes stale

A container image is immutable: once built, its bytes never change. Vulnerability knowledge, by contrast, grows every day. A CVE is a newly discovered flaw in software that already existed; the openssl or glibc in your image was carrying the bug all along, the world just hadn't catalogued it yet. So 'the scan passed' means only 'no CVE known on build day matched these versions'. Time passes, researchers publish new CVEs against those exact versions, and a re-scan of the identical image now reports criticals, without a single line of your code or your Dockerfile changing. This surprises people because they treat scanning as a one-time build check; it is really a moving target.

The core insight: images must be re-baked, not just re-scanned

Re-scanning tells you an image is stale; only rebuilding on updated base layers actually fixes it (assuming the upstream distro/library has shipped a patch). The operational job is to keep the gap between 'a fix exists' and 'we've deployed it' small.

Strategy components

1. Scheduled rebuild cadence

Don't rely solely on code pushes to trigger builds. Run a scheduled pipeline (nightly or weekly, plus on-demand for urgent CVEs) that rebuilds images so they pull the latest patched base and dependency versions. This is the single biggest lever: most CVE exposure is in the base OS layer, which distros patch continuously.

2. Pin by digest, but automate the bump

Reproducibility wants a pinned base (FROM debian:12-slim@sha256:...) so builds are deterministic and not silently changed by a mutable tag. But a static pin rots: it freezes you on an old, increasingly-vulnerable base. Resolve the tension with automation, Renovate or Dependabot that opens PRs to advance the digest to the newest patched build. You keep determinism per-build and freshness over time.

3. Minimize the attack surface

Every package you ship is future CVE exposure. Reduce it:

Minimal bases: -slim, Alpine (musl caveats aside), or distroless images that contain only your app and its runtime, no shell, no package manager.
Multi-stage builds: compile/build in a fat stage, copy only artifacts into a tiny final stage. Fewer components means fewer matches on every future scan and less to patch.

4. Continuous re-scanning drives the cadence

Pair rebuilds with continuous scanning of deployed images or their stored SBOMs (see the SBOM question) against fresh feeds. When a new critical appears in something you're running, it should trigger an out-of-band rebuild-and-redeploy, not wait for the next scheduled cycle. Mature setups automate this: new critical -> bump base -> rebuild -> scan -> deploy.

5. Set remediation SLAs

Decide, by severity, how fast you patch: e.g. critical within 24-48h, high within a week. The cadence and automation exist to meet those SLAs. This is where the practice becomes a program rather than a cron job.

Common failure modes

Build-once images that live for months and accumulate unpatched CVEs.
Pinning without bumping: a digest pin that nobody ever advances, freezing an ancient base.
Fat base images (full ubuntu/node) that maximise future CVE surface.
Scan-at-merge only, with no re-scan of what's actually deployed, so drift is invisible.
Manual rebuilds that only happen when someone remembers, missing the SLA.

The one-liner for interviews

'Images are immutable but the CVE feed isn't, so a clean build decays. Keep them current by rebuilding on a cadence off minimal, digest-pinned-but-auto-bumped bases, and let continuous re-scanning of deployed images trigger rebuilds against your severity SLAs.'

## Why a clean build goes stale A container image is **immutable**: once built, its bytes never change. Vulnerability knowledge, by contrast, grows every day. A CVE is a *newly discovered* flaw in software that already existed; the openssl or glibc in your image was carrying the bug all along, the world just hadn't catalogued it yet. So 'the scan passed' means only 'no CVE **known on build day** matched these versions'. Time passes, researchers publish new CVEs against those exact versions, and a re-scan of the identical image now reports criticals, **without a single line of your code or your Dockerfile changing**. This surprises people because they treat scanning as a one-time build check; it is really a moving target. ## The core insight: images must be re-baked, not just re-scanned Re-scanning tells you an image is stale; only **rebuilding** on updated base layers actually fixes it (assuming the upstream distro/library has shipped a patch). The operational job is to keep the gap between 'a fix exists' and 'we've deployed it' small. ## Strategy components ### 1. Scheduled rebuild cadence Don't rely solely on code pushes to trigger builds. Run a **scheduled pipeline** (nightly or weekly, plus on-demand for urgent CVEs) that rebuilds images so they pull the latest patched base and dependency versions. This is the single biggest lever: most CVE exposure is in the base OS layer, which distros patch continuously. ### 2. Pin by digest, but automate the bump Reproducibility wants a **pinned** base (`FROM debian:12-slim@sha256:...`) so builds are deterministic and not silently changed by a mutable tag. But a static pin **rots**: it freezes you on an old, increasingly-vulnerable base. Resolve the tension with automation, **Renovate** or **Dependabot** that opens PRs to advance the digest to the newest patched build. You keep determinism per-build and freshness over time. ### 3. Minimize the attack surface Every package you ship is future CVE exposure. Reduce it: - **Minimal bases:** `-slim`, Alpine (musl caveats aside), or **distroless** images that contain only your app and its runtime, no shell, no package manager. - **Multi-stage builds:** compile/build in a fat stage, copy only artifacts into a tiny final stage. Fewer components means fewer matches on every future scan and less to patch. ### 4. Continuous re-scanning drives the cadence Pair rebuilds with **continuous scanning** of deployed images or their stored SBOMs (see the SBOM question) against fresh feeds. When a new critical appears in something you're running, it should **trigger** an out-of-band rebuild-and-redeploy, not wait for the next scheduled cycle. Mature setups automate this: new critical -> bump base -> rebuild -> scan -> deploy. ### 5. Set remediation SLAs Decide, by severity, how fast you patch: e.g. critical within 24-48h, high within a week. The cadence and automation exist to meet those SLAs. This is where the practice becomes a program rather than a cron job. ## Common failure modes - **Build-once images** that live for months and accumulate unpatched CVEs. - **Pinning without bumping:** a digest pin that nobody ever advances, freezing an ancient base. - **Fat base images** (full `ubuntu`/`node`) that maximise future CVE surface. - **Scan-at-merge only**, with no re-scan of what's actually deployed, so drift is invisible. - **Manual rebuilds** that only happen when someone remembers, missing the SLA. ## The one-liner for interviews 'Images are immutable but the CVE feed isn't, so a clean build decays. Keep them current by rebuilding on a cadence off minimal, digest-pinned-but-auto-bumped bases, and let continuous re-scanning of deployed images trigger rebuilds against your severity SLAs.'

If pinning the base image by digest is best for reproducibility, doesn't that guarantee it goes stale?
Yes, if left alone. That's why you pair the pin with automation (Renovate/Dependabot) that opens PRs advancing the digest to the newest patched build. You get deterministic, tamper-evident builds today and freshness over time, instead of choosing between reproducibility and currency.
How does using a distroless or slim base reduce future scanning pain?
Fewer packages in the image means fewer components for scanners to match against future CVE disclosures, so fewer findings and fewer forced rebuilds. Distroless images drop the shell and package manager entirely, removing whole classes of vulnerable and exploitable tooling from the runtime.
What should happen when continuous re-scanning flags a new critical in a deployed image?
It should trigger an out-of-band rebuild against the patched base, re-scan, and redeploy, ideally automatically, to meet your remediation SLA rather than waiting for the next scheduled cycle. The re-scan detects; the rebuild-and-redeploy remediates.

An image is a sealed tin of food with a fixed recipe; new CVEs are like recalls announced after canning. The tin never changes, so you must re-can with corrected ingredients, not just re-read the recall list.

saying these in an interview costs you the question

Assuming a passing build stays clean over time
Treating images as build-once artifacts with month-long lifespans
Pinning a base digest and never advancing it
Choosing fat base images that maximise future CVE surface
Only scanning at merge and never re-scanning what's deployed
Thinking re-scanning alone fixes anything (you must rebuild)

Your team scans images in CI and the last build passed clean, yet weeks later the same deployed image shows new critical CVEs. Why does this happen, and what base-image patch-and-rebuild strategy keeps images current?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

Why a clean build goes stale

The core insight: images must be re-baked, not just re-scanned

Strategy components

1. Scheduled rebuild cadence

2. Pin by digest, but automate the bump

3. Minimize the attack surface

4. Continuous re-scanning drives the cadence

5. Set remediation SLAs

Common failure modes

The one-liner for interviews

▾common follow-ups3 questions

▾analogy~20 s

▾red flags6 flags

§ context

▾related questions6 questions

▾related topics4 topics

Why a clean build goes stale

The core insight: images must be re-baked, not just re-scanned

Strategy components

1. Scheduled rebuild cadence

2. Pin by digest, but automate the bump

3. Minimize the attack surface

4. Continuous re-scanning drives the cadence

5. Set remediation SLAs

Common failure modes

The one-liner for interviews

answer

go deeper

context