An autoscaled service deployed from a reviewed digest runs different bytes on nodes added hours later - how do you diagnose it?

## The principle behind the symptom A pin is only as strong as the **last place the reference is resolved**. A reference passes through many hands - a developer's file, a build job, a deploy tool, a cluster controller, the node agent that actually pulls, and the local cache on that node. If any step downstream of your pin still holds a mutable name, the pin was decorative: the name gets resolved again, later, under conditions nobody reviewed. That is what a multi-tenant autoscaling service exposes so cruelly. The reviewed digest was true at deploy time. Then a scale-out event, a node replacement, a spot reclaim or a rolling eviction creates new instances from the *stored* spec, hours later, and each of those pulls resolves the name against the registry as it is now. ## Diagnose in this order **1. Freeze and collect, do not roll.** The instinct to redeploy destroys the only evidence you have. For every running instance record the resolved image digest the runtime reports and the instance start time. Mixed digests across one workload is the finding. **2. Read the persisted spec, not the pipeline.** Ask what the platform has actually stored for this workload. If it says `svc:release-2026.04`, you have your answer: the digest existed only in the pipeline's logs or in a pre-flight check, and was discarded before anything durable was written. **3. Correlate against the tag history.** Registries record when a tag was remapped and by which identity. Match the timestamp against the moment the first divergent instance started. That distinguishes a deliberate push from a stale mirror or a restored backup. **4. Explain the partial spread.** People misread "only half the fleet" as randomness or a registry fault. It is neither. Instances started before the remap keep running the bytes they already have; nodes holding a cached copy may not re-pull at all depending on pull policy; only fresh pulls after the remap fetch new content. Age of the instance, not configuration, decides who is affected. If the split does **not** follow start time, look elsewhere - a mirror serving stale content, or two clusters pointed at different registries. **5. Establish what actually ran.** For a multi-tenant API the security question is not only "is it malicious" but "what did unreviewed code have access to": the service account, the tenant data reachable through it, and any credential mounted into it. Assume the unreviewed bytes had everything the workload has, and scope the response from there. ## The fix, in order of durability - **Put the digest in the persisted spec.** Whatever writes the deployment - the pipeline, a controller reconciling from git, a template - must emit `svc@sha256:...`. If a tool resolves a tag to a digest for verification and then applies the tag anyway, the resolution is thrown away at exactly the moment it matters. - **Gate the deployment.** A check that refuses any workload whose image reference is not digest-pinned makes the failure mode loud instead of silent, and turns "we agreed to pin" into something enforced rather than remembered. - **Close the registry side too.** Forbidding remaps of existing tags means the substitution is rejected at the source; restricting who holds push rights shrinks the set of identities that could have done it. Neither replaces the digest in the spec, because a mirror, another registry or a restore sits outside those rules. - **Detect drift continuously.** Compare the digests actually running against the digests approved for that release, on a schedule. This is the control that would have found the problem in minutes instead of at the next incident. ## The trap to name explicitly "We pin in CI" is the single most common false comfort here. Resolving a digest during the build or the deploy proves what *that step* saw; it says nothing about what the platform will resolve when it creates instance number forty at three in the morning. The question to ask of any pinning story is always: **what string is stored in the thing that scale-out reads?** If the answer is a tag, there is no pin.