The CI runner pod behind your alert is already deleted — how do you keep pivoting?

## Why the pivot dead-ends Ephemeral infrastructure breaks the usual triage rhythm. On a workstation you pivot on the host and keep pivoting for weeks. A CI runner pod may have lived four minutes; by the time an alert reaches an analyst the object is deleted, its address is reassigned, and its name resolves to nothing. Pivoting on the pod produces zero results, and zero results feel like an answer when they are actually an artefact of your join key having evaporated. The move is to change the key. ## Durable entities in a build estate Rank candidate entities by whether something *other than the dead workload* recorded them. - **The identity.** The service account the runner authenticated as persists as a cluster object and appears on every API call it made. It survives the pod entirely and is usually the widest surviving surface. - **The image digest.** A content address, so it names the exact artefact regardless of where it ran. It appears in registry pull records, in deployment records, and on any other host that ran the same content — which is what turns one dead pod into an estate-wide question. - **The pipeline job id.** It ties the activity to a specific build: which repository, which commit, which triggering event, which secrets that job was entitled to. This is the pivot that converts "something odd happened in the cluster" into "this build did it". - **The git ref and commit.** Durable by design, and it tells you what the job was *supposed* to do. - **The node.** The pod is gone; the machine that hosted it is not, and its own telemetry covers the period. - **Artefacts produced.** Anything the job pushed — images, packages, signatures — outlives the job and can be examined. The general principle generalises past containers: prefer the entity that a *different* system independently wrote down, because that record is not destroyed when the workload is. ## What genuinely died with the pod Be precise about the loss, because vagueness here turns into overclaiming later: - the container filesystem, including anything the intruder wrote; - process memory and the live process tree; - environment as it actually stood, including any credential materialised at runtime; - anything the workload did that emitted no record anywhere. You can bound activity from what durable systems kept. You cannot recover what only the pod held. The correct sentence in the case notes is "the runner was deleted at 02:19; no host-level artefacts were preserved, so activity is reconstructed from control-plane and registry records only" — a stated gap that a reviewer can weigh. The wrong sentence is any version of "nothing was found on the host", which reads as a negative finding and is not one. ## Turning the next occurrence into evidence The useful senior move is forward-looking. If the pipeline still schedules runners from the same image and the same identity, the behaviour will recur — and the next instance is evidence you can actually hold. Options, all decisions to take early because each cycle destroys the artefact: - arrange for the next matching pod to be held rather than reaped, so it can be examined; - capture volatile state before the object is removed; - retain the image locally so the artefact cannot be repointed away from you; - if you must let it run, at minimum know in advance which durable records you will be reading afterwards. This is also where an ephemeral estate quietly punishes a slow queue: a case that sits for two hours has lost evidence a case picked up in ten minutes would have kept, and that is worth saying plainly when someone asks why the pivot dead-ended. ## The trap to name in an interview The failure this question is really probing is concluding *clean* from *gone*. An intruder who lives inside short-lived compute benefits directly from the analyst who pivots on the pod, gets nothing, and closes the case. The right answer changes the key, states the gap, and sets up the next occurrence.