Your production Kubernetes workloads run distroless images as non-root with read-only root filesystems, and on-call engineers complain they cannot debug incidents. How would you give them a debugging capability without undoing that hardening?

## Frame the tension honestly Hardening removes tools from images on purpose: smaller attack surface, fewer CVEs to patch, no shell for an attacker to pivot with. Debuggability is not free either — the answer is not to re-add a package manager to the production image, and not to tell on-call to cope. The capability should be **attachable on demand, privileged deliberately, and observable afterwards**. ## Layer 1 — make most incidents shell-free Every request for a shell is a signal that some evidence was not exported. Before granting broad interactive access, close the common gaps: structured logs on stdout at the right level with correlation IDs; per-endpoint request/error/duration metrics; a diagnostics endpoint exposing configuration, build info, dependency status and thread or goroutine dumps; and on-demand profiles or heap dumps that the process writes to object storage rather than to a local disk someone must copy off. Keep this scoped — long-term dashboards and alerting are the observability platform's job — but the last-mile diagnostics endpoints belong to the service. A team that measures "incidents that required a shell" usually finds the number falls once these exist. ## Layer 2 — a sanctioned debug path Standardize on ephemeral containers rather than mutable images: - One or two **approved debug images**, built and scanned like any other artifact, hosted in the internal registry so they pull fast and are available on air-gapped clusters. Pre-pulling them onto nodes removes the failure mode where a node with a broken runtime or full disk cannot fetch tools. - A documented decision tree: live pod misbehaving goes to `kubectl debug --target`; container not staying up goes to `--copy-to` with a changed command; node-level symptoms go to `kubectl debug node/`; anything reachable by port goes to `kubectl port-forward` instead of a shell. - Security-context profiles chosen consciously: `--profile=restricted` by default and elevated profiles only where the namespace's Pod Security admission level permits them. Expect that a `restricted` namespace rejects privileged debug containers — that is the control working, and you need a decided answer (usually a dedicated namespace for node debugging) rather than a surprise mid-incident. - Known limits written down: ephemeral containers cannot be removed, carry no resource requests and can push a tight node over allocatable, and copies are not the failing instance. ## Layer 3 — access as break-glass, not a standing grant `pods/exec` and `pods/ephemeralcontainers` mean code execution inside a workload with its service-account token and mounted secrets; node debug means root on the machine and a path to cluster-admin. Standing grants for these in production are hard to defend. A workable model: read-only plus `pods/log` and `pods/portforward` by default; an escalation role bound for a fixed window through an approved, logged mechanism; node debug held by the platform team only. Pair with an audit policy that records those subresources in full, ship those events somewhere durable, and review them periodically — the review is what makes the grant sustainable, not the grant itself. ## Layer 4 — hygiene after the incident A debugged pod is a modified pod: the ephemeral container is in its spec permanently, and anything the responder changed inside the container is invisible to Git. The rule should be that pods touched interactively are recycled once the incident is over, and that any fix applied by hand is immediately reproduced as a manifest or image change. Forgotten node-debugger pods and hand-patched replicas are two of the most common findings in cluster reviews, and both are process failures rather than tooling ones. ## Trade-offs a principal should name The curated-toolbox model costs image maintenance and scanning, and adds friction during an incident. Break-glass access adds an approval step that can be resented at 3am, so the mechanism must be fast or people will route around it. Pre-pulling debug images consumes node disk. And no debug tooling helps if the node itself is unreachable — the fallback of cloud-console or SSH access must exist and be tested, or the whole scheme has an untested single point of failure.