Answering For It

## The question behind the question An interviewer asking this is checking one thing: do you understand that **containerised remediation is destructive by default**, and can you name precisely what it destroys? In a Kubernetes estate the normal reaction to a flagged workload — delete the pod, let the controller replace it — is also the fastest possible evidence-destruction procedure, and it happens in seconds, often automatically, usually before a human has looked at anything. ## What lives only inside the container A running container is a process (or a small tree of them) in its own namespaces on a node, with a thin writable layer stacked over the read-only image layers. Three categories of material exist nowhere else: - **Process memory.** The address space of the running process holds the decoded payload of a memory-only loader, its configuration, any keys or tokens it pulled in, decrypted buffers, and the arguments of whatever it is doing right now. A loader that fetches and executes in memory never writes the payload to a filesystem, so this is not one copy of the implant — it is the *only* copy. - **The writable layer.** Anything the process dropped: staged archives, scripts, added binaries, temporary output. It is removed when the container instance goes away. - **Live kernel-side state for those namespaces.** Established sockets and their peers, the process tree and parent relationships, open file descriptors, mounts. Together these answer "what was it talking to, and what started it". Delete the pod and all three are gone with no copy anywhere. This is different from a VM or a laptop, where the disk survives a reboot and can be imaged tomorrow — the container's equivalent of "the disk" is discarded as part of normal operation. ## What survives, and what it actually proves The durable record is second-hand, and each piece proves something narrower than people assume: | What survives | What it proves | What it does not | |---|---|---| | Kubernetes audit events | that an API request was made, by which identity, against which resource and subresource | anything about what happened *inside* a container; for a `pods/exec`, the requested command is in the request URI, but the session content is not recorded | | Container stdout shipped off the node | what the process chose to write to stdout | anything it chose not to write, and anything about its memory | | Runtime-sensor / EDR telemetry | that a rule fired and that certain events were observed | that something malicious happened — a detection firing is a detection firing | | Flow records (NetFlow/IPFIX) | that bytes moved between endpoints, how many, when | what those bytes were — flow records carry no payload | | The image in the registry | what was *supposed* to be running | the runtime-injected implant, which by definition was not in the image | That table is the whole answer to "the SIEM has it": the SIEM has records of emissions and observations. It does not have the artefact. You cannot hash what you never copied, cannot pull strings from it, cannot compare it against a sample from another victim, and cannot hand it to anyone who might identify it. ## The ephemeral-workload twist On a Kubernetes node the deadline is not set by you. A liveness probe restart, an autoscaler scale-down, a rollout, a node drain, or an operator's `delete pod` all end the container, and the ones that are automated do not wait for the incident channel. Meanwhile the replacement pod is clean, so you also lose forward visibility: you cannot watch what the intruder does next in that workload, because the workload they were in no longer exists. That is a second loss, and it is the one people forget — the intruder's access path (a stolen credential, an exposed endpoint, a vulnerable dependency in the image) is untouched by the deletion, so they can come back into a pod you are no longer watching. ## What "deleting the pod cut them off" gets wrong Deleting the pod ends that process. It does not end the intrusion. A projected, bound service-account token stops being accepted once the pod object it is bound to is gone, which helps; but a long-lived Secret-based token, a cloud credential lifted from the node's instance metadata, a key exfiltrated minutes earlier, or the vulnerability that granted execution in the first place are all unaffected. Treating deletion as eradication is the classic wrong answer here: containment stops damage, eradication removes the adversary's paths back, and the two are different phases. ## What to do instead, in the first minute You do not need a forensics programme to avoid the worst of this. Before anything destroys the pod: record the pod's identity and object (name, UID, namespace, node, the spec as it stands), pin the **image digest** rather than the tag, and stop whatever is about to recreate it. Those cost seconds and they preserve the ability to reconstruct the baseline and to interpret every other record you collect later. The expensive part — capturing the running process — is a decision with a cost, and it is the one that has to be argued for while the clock runs.