A flagged Kubernetes pod was deleted and replaced before anyone captured it - what is gone for good?
answer
- what only that container held
- shipped logs record emissions
- the loader never touched disk
- no copy means nothing to hash
basics
~20 sEverything that existed only inside that container: the process memory holding a memory-only loader, its live network and process state, and the writable layer's dropped files. Shipped logs survive, but they record what was emitted, never the code that ran.
solid answer
~50 sDeleting the pod destroys the only copy of anything that never touched durable storage. A memory-only loader lives in the process address space, so its decoded payload, in-memory configuration, keys and established sockets go with the container, along with whatever it dropped into the writable layer. What survives is second-hand: Kubernetes audit records at the API server, container stdout already shipped off the node, runtime-sensor telemetry, flow records. Those prove that events were emitted - an exec was requested, bytes moved - but they never contain the artefact, so you cannot hash it, analyse it, or match it against anything later. The replacement pod is clean, so you lose forward visibility too, while the access path that got them in is untouched. Pin the image digest and the pod object early; those at least preserve what was supposed to be running.
code
json · 16 lines{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"stage": "ResponseComplete",
"verb": "create",
"user": { "username": "system:serviceaccount:ci:deploy-bot" },
"sourceIPs": ["10.42.7.19"],
"objectRef": {
"resource": "pods", "subresource": "exec",
"namespace": "payments", "name": "api-7d9c8-fz2kq"
},
"requestURI": ".../pods/api-7d9c8-fz2kq/exec?command=sh&container=api&stdin=true&tty=true",
"responseStatus": { "code": 101 },
"stageTimestamp": "2026-03-11T02:41:07Z"
}go deeper
Be ready to name what exists only inside a running container - process memory, live sockets, files dropped into the writable layer - and to say plainly that centrally shipped logs are not a copy of the code.
Explain why a memory-only loader leaves nothing on disk to collect later, and which durable records survive the pod - Kubernetes audit, node-shipped stdout, runtime and flow telemetry - and what each one actually proves.
Show that you act in the first minute: pin the image digest and the pod object, stop whatever is about to recreate the pod, and only then argue about how long a capture may take.
Own the consequence at estate scale. If remediation destroys evidence by default, the organisation cannot answer scope questions after any container intrusion; decide whether that is an accepted risk or a platform change you fund.
## The question behind the question An interviewer asking this is checking one thing: do you understand that **containerised remediation is destructive by default**, and can you name precisely what it destroys? In a Kubernetes estate the normal reaction to a flagged workload — delete the pod, let the controller replace it — is also the fastest possible evidence-destruction procedure, and it happens in seconds, often automatically, usually before a human has looked at anything. ## What lives only inside the container A running container is a process (or a small tree of them) in its own namespaces on a node, with a thin writable layer stacked over the read-only image layers. Three categories of material exist nowhere else: - **Process memory.** The address space of the running process holds the decoded payload of a memory-only loader, its configuration, any keys or tokens it pulled in, decrypted buffers, and the arguments of whatever it is doing right now. A loader that fetches and executes in memory never writes the payload to a filesystem, so this is not one copy of the implant — it is the *only* copy. - **The writable layer.** Anything the process dropped: staged archives, scripts, added binaries, temporary output. It is removed when the container instance goes away. - **Live kernel-side state for those namespaces.** Established sockets and their peers, the process tree and parent relationships, open file descriptors, mounts. Together these answer "what was it talking to, and what started it". Delete the pod and all three are gone with no copy anywhere. This is different from a VM or a laptop, where the disk survives a reboot and can be imaged tomorrow — the container's equivalent of "the disk" is discarded as part of normal operation. ## What survives, and what it actually proves The durable record is second-hand, and each piece proves something narrower than people assume: | What survives | What it proves | What it does not | |---|---|---| | Kubernetes audit events | that an API request was made, by which identity, against which resource and subresource | anything about what happened *inside* a container; for a `pods/exec`, the requested command is in the request URI, but the session content is not recorded | | Container stdout shipped off the node | what the process chose to write to stdout | anything it chose not to write, and anything about its memory | | Runtime-sensor / EDR telemetry | that a rule fired and that certain events were observed | that something malicious happened — a detection firing is a detection firing | | Flow records (NetFlow/IPFIX) | that bytes moved between endpoints, how many, when | what those bytes were — flow records carry no payload | | The image in the registry | what was *supposed* to be running | the runtime-injected implant, which by definition was not in the image | That table is the whole answer to "the SIEM has it": the SIEM has records of emissions and observations. It does not have the artefact. You cannot hash what you never copied, cannot pull strings from it, cannot compare it against a sample from another victim, and cannot hand it to anyone who might identify it. ## The ephemeral-workload twist On a Kubernetes node the deadline is not set by you. A liveness probe restart, an autoscaler scale-down, a rollout, a node drain, or an operator's `delete pod` all end the container, and the ones that are automated do not wait for the incident channel. Meanwhile the replacement pod is clean, so you also lose forward visibility: you cannot watch what the intruder does next in that workload, because the workload they were in no longer exists. That is a second loss, and it is the one people forget — the intruder's access path (a stolen credential, an exposed endpoint, a vulnerable dependency in the image) is untouched by the deletion, so they can come back into a pod you are no longer watching. ## What "deleting the pod cut them off" gets wrong Deleting the pod ends that process. It does not end the intrusion. A projected, bound service-account token stops being accepted once the pod object it is bound to is gone, which helps; but a long-lived Secret-based token, a cloud credential lifted from the node's instance metadata, a key exfiltrated minutes earlier, or the vulnerability that granted execution in the first place are all unaffected. Treating deletion as eradication is the classic wrong answer here: containment stops damage, eradication removes the adversary's paths back, and the two are different phases. ## What to do instead, in the first minute You do not need a forensics programme to avoid the worst of this. Before anything destroys the pod: record the pod's identity and object (name, UID, namespace, node, the spec as it stands), pin the **image digest** rather than the tag, and stop whatever is about to recreate it. Those cost seconds and they preserve the ability to reconstruct the baseline and to interpret every other record you collect later. The expensive part — capturing the running process — is a decision with a cost, and it is the one that has to be argued for while the clock runs.
- The image the pod ran from is still in the registry - doesn't that give you the malware?Only if the malicious code was in the image. A memory-only loader is fetched or injected at runtime, so the image is the clean baseline rather than the implant. Pinning it by sha256 digest is still worth doing immediately: it tells you exactly what was supposed to be running, and diffing against it shows what was added. It answers 'was this a supply-chain compromise', not 'what was the payload'.
- Does deleting the pod at least cut the intruder off?Not reliably. It ends that process, not the access path. A projected bound service-account token stops being accepted once the pod object it is bound to is gone, but a long-lived Secret-based token, a cloud credential lifted from instance metadata, or the vulnerability that granted execution are all unaffected. The controller creates a clean replacement, and the same weakness is reachable again seconds later - in a pod nobody is watching.
- The node still has the container's stdout log files - is that the same as having the container?No. Kubelet keeps pod log files on the node only while the pod object exists, and they contain whatever the process chose to write to stdout - nothing about its memory, its child processes, or its sockets. They are a cheap thing to grab early, not a substitute for the container itself.
Scrapping a crashed car before anyone inspects it. The traffic cameras still show it was on the road at 02:00, but the thing that would tell you why it crashed has been melted down.
saying these in an interview costs you the question
- Says the SIEM already has everything we need
- Assumes the container image contains the implant
- Treats the clean replacement pod as the same evidence
- Thinks deleting the pod ends the intrusion
- Believes a fired runtime alert proves what executed