A suspect Kubernetes pod may reschedule at any moment - what do you capture, and what can wait?

## The sorting rule Under a reschedule deadline the useful question is not "what evidence would I like" but **"what does the next container lifecycle event destroy, and what is stored somewhere that outlives it"**. Sort every candidate artefact into those two buckets and spend the window entirely on the first. Almost everyone under time pressure does the reverse, because the durable things are the easy ones to grab. ## The perishable set, and why each item earns its place **The suspect process's memory.** A loader that runs in memory has no on-disk copy anywhere: not in the image, not in the writable layer, not in a registry. If you take one thing, take this. It is also the artefact that answers the questions you will be asked afterwards — what the code did, what it was configured to talk to, what it had already decrypted or collected. **The container's identity.** Container id, pod name and **pod UID**, namespace, node, start time, the image reference *and its sha256 digest*, the command and arguments it was started with. This is small, fast, and it is what makes everything else interpretable months later. Tags are mutable — `payments-api:1.4.2` can point to different bytes next week — so a tag is a name, while a digest is the actual thing that ran. **The writable layer.** Whatever was dropped: staged archives, added tooling, scripts, partial output. In a container this is usually tiny compared with a host disk, so it is a cheap grab with a high hit rate. Memory tells you what was executing; the layer tells you what was staged. Neither implies the other. **Live network and process state.** The socket table with peers, the process tree with parents, open file descriptors. Plus the **pod IP to pod mapping**: cluster pod IPs come from a pool and are recycled, so a flow record for `10.42.3.17` becomes uninterpretable once the mapping is gone. That single line of metadata is what keeps all your network evidence attributable. **The pod object itself.** The live spec and status — mounted volumes, projected tokens, environment references, service account, node assignment, restart history. Once the object is garbage-collected, reconstructing what the workload had access to becomes archaeology. ## The set that can wait - **The image.** Immutable and pullable by digest whenever you like. Analysing it during the window is a common and expensive mistake. - **Kubernetes audit events.** Written at the API server, retained centrally, unaffected by the pod's life. They will tell you who called what, including whether an interactive `pods/exec` was requested — later. - **Runtime sensor and EDR telemetry, flow records, proxy and DNS logs.** Already off the node. Subject to retention, not to the pod's lifetime — check the retention window, but do not spend the deadline on them. - **Cloud control-plane logs** for whatever the pod's identity touched. Same reasoning. ## The move that beats capturing faster The single highest-value action is often not collection at all: **stop the thing that will destroy the pod**. Pause the Deployment's rollout so a reconcile does not replace it, exclude the node from autoscaler scale-down, cordon the node so a subsequent drain is less likely, and tell the on-call platform engineer what you are doing before they do the obvious thing. Cordoning marks the node unschedulable and leaves the running pods alone — it does not evict anything, which is exactly why it is safe to do immediately and why it is not, on its own, containment. Removing the deadline converts a five-minute scramble into an unhurried collection, and a truncated capture is worth far less than a complete one: a process image taken while the process is running has no single point in time to begin with, and a truncated one may hold only fragments. ## Doing this while the pod is still dangerous Capturing is not a reason to leave a live foothold untouched. The two are separable: deny the pod's egress with a network policy, revoke the service account's and any cloud credentials it holds, and block its known peers at the egress gateway. The container keeps running — and stays capturable — while the operator's reach is cut. Whether that is enough, and who authorises the wait, is the harder decision that follows. ## What this looks like when it is answered badly Weak answers start with the image because it is the familiar artefact, describe collecting logs that were never at risk, or spend the window arguing in the incident channel while an autoscaler quietly removes the node. The strongest answers are short: stop the clock, record the identity and digest, take memory, take the layer, then argue about everything else with the pod still alive.