Platform engineering auto-deletes flagged pods, so every intrusion loses its evidence - what do you negotiate?

## Why the default exists and why "stop doing it" loses Auto-remediation that deletes a flagged pod is a good control. It restores a healthy workload in seconds, it needs no human, and it reduces pages for the team that owns the platform. That team is measured on availability and on toil. A security ask that reads "stop deleting compromised pods" asks them to trade their own metric for someone else's, at 03:00, on the word of a detection that might be a false positive. They can refuse, and if they cannot refuse openly they will route around it later. Treat this as a negotiation with an owner who has a legitimate position, not as a policy you can assert. ## Reframe: preserve *and* remediate, don't choose The proposal that survives contact is one that keeps everything the platform team gets today: 1. **Remediation still happens immediately.** The controller creates the replacement pod; service is restored on the same timeline as now. 2. **The original is quarantined, not destroyed.** It stays on its node, the node is cordoned so nothing new lands there, a deny-all egress policy is applied, and the identity it holds is revoked. The container keeps running, isolated and stripped, so it remains capturable. 3. **A hard, platform-enforced expiry.** After a fixed window the held pod is reaped automatically unless someone re-authorises. The guard rail must be enforced by the system, not by a human promise, because the failure mode is a forgotten hold occupying capacity for a month. 4. **A capacity cap.** A maximum number of concurrent holds, visible on the platform team's own dashboard, with who invoked each and why. Now the conversation is about a bounded, self-releasing resource cost rather than about disabling their control. ## Automate the cheap evidence so the human decision shrinks The decision that fails at 03:00 is the one that requires an expert to be awake and to win an argument. So move everything cheap into the remediation path itself: on deletion, emit the pod object, the container runtime metadata, the image reference **and digest**, the restart history and the container filesystem to an evidence bucket, automatically, for every flagged pod. None of that is contentious, none of it delays remediation meaningfully, and it removes most of the routine loss. What remains — holding a live process so its memory can be captured — is expensive and genuinely needs a decision, so make sure that is the *only* thing anyone has to argue about. ## Pre-authorise, with a budget Standing authority beats escalation. Write it down: the incident lead may quarantine up to N pods or one node's worth of capacity, for up to the enforced window, without further approval; beyond that it escalates to the service owner. Name who may invoke it out of hours. The point is that nobody negotiates capacity with a sleeping executive while an intruder is live, and equally that the hold cannot quietly grow into an outage nobody agreed to. ## Fund it with cases, not with principles "Best practice" is the weakest argument available. Bring the record: the container intrusions in the last year where remediation destroyed the implant, and the specific consequence in each — the report claim that had to be softened from established to consistent-with, the scope question a customer asked that could only be answered with inference, the detection you could not write because you never had the artefact. Engineering leadership funds a capability that closes a named gap; it does not fund forensic tidiness. ## Accept exemptions rather than pretend Some workloads cannot bear it — a capacity-critical path, a regulated environment where a held pod is itself a problem, a node type too scarce to hold. A documented exemption with an agreed alternative (automatic filesystem and metadata capture only, no live hold) is worth more than a universal policy that is quietly ignored. The exemption list is also the honest statement of residual risk, and it belongs where a risk owner signs it. ## Measure whether it worked Two numbers make this reviewable: the proportion of container intrusions where the running artefact was actually recovered, and the time from detection to quarantine. If recovery does not improve, the mechanism exists but is not reachable in practice — usually because invoking it needs a person who is not there. If holds routinely hit the expiry with nobody having captured anything, the capture capability is the gap, not the policy. Either way, report it against the cases you used to fund it. ## The failure modes to name in an interview Demanding a veto over another team's automation; designing a hold with no expiry; making security the only party who can release it; assuming access to the cluster is the same as authority to change how it behaves; and building a beautiful mechanism whose invocation depends on a specialist being awake.