At Metadata audit level, a Kubernetes entry shows a create on pods/exec into production — what can you establish?

## Kubernetes audit levels in one paragraph The API server evaluates every request against an audit policy whose rules select a **level**: `None` (record nothing), `Metadata` (record the request's metadata but neither request nor response body), `Request` (metadata plus the request body), and `RequestResponse` (metadata plus both bodies). Events are also emitted at **stages** — `RequestReceived`, `ResponseStarted`, `ResponseComplete`, `Panic` — so one call can produce more than one record. Most production clusters run Metadata as the default with narrow exceptions, for two reasons: bodies multiply log volume, and bodies of `Secret` and `ConfigMap` writes would put the secret material itself into the audit log, turning the audit trail into a credential store. ## What the Metadata-level exec entry gives you For a `kubectl exec` the API server records a `create` on resource `pods` with subresource `exec`. From that single record you can state: - **Who**: `user.username` and `user.groups` as authenticated by the API server, plus `impersonatedUser` if `--as` was used. - **Where from**: `sourceIPs` and the self-reported `userAgent` (`kubectl/v1.31.2`, a CI client, a controller). - **What target**: `objectRef` naming the namespace, pod name and container. - **When**: `requestReceivedTimestamp` and `stageTimestamp`, which bracket how long the session ran. - **Whether it was allowed**: `responseStatus`, and for a successful exec the protocol upgrade rather than a 403. - **What was asked for**: this is the part people miss. Because `kubectl exec` passes its arguments as **query parameters on the request URI**, and `requestURI` is metadata, the initial argv survives at Metadata level — you can usually read `command=%2Fbin%2Fbash`, the container name and `tty=true`, which is enough to say an interactive shell was requested rather than a one-shot command. ## What no audit level will give you The exec session is not a request body. The call **upgrades the connection to a bidirectional stream** carrying stdin, stdout and stderr, and that stream is not part of the audited request. So: - **Raising the rule to `Request` adds nothing** for exec — there is no JSON body to capture. - **Raising it to `RequestResponse` adds nothing either**, while paying the full cost of body capture across everything else the rule matches. Anyone who answers "I would turn up the audit level to see the commands" has just accepted a large volume and secret-exposure cost for zero additional evidence on the exact behaviour they cared about. That is the trap in the question. ## Where the missing evidence actually lives To reconstruct what happened inside the session you need a different observation surface: - **In-container process telemetry** from a runtime sensor watching process creation inside the pod. - **Node-level process records**, if the node still exists and its telemetry was shipped. - **A session-recording access broker** in front of the cluster, which is the only control that captures keystrokes by design. And here the ephemerality bites: if the pod has since restarted or the node was scaled away, none of those exist retroactively. The audit entry may be the only account of the session that survives, which is why the policy decision — which namespaces get audited at what level, and whether exec is routed to a detection — has to be made before the incident. ## Reaching a verdict With the session content unavailable, the verdict is built on identity and corroboration: map the username to a person or a workload, check whether a change record, incident bridge or deployment covers the timestamp window, look at what else that identity did in the same window across the cluster and the identity provider, and ask the named operator. In the common case an on-call engineer opened a shell in a production pod during a live degradation, which looks exactly like an intruder and is not one. That is a **benign true positive**: the detection was correct about the behaviour, and the behaviour was authorised. Closing it as a false positive would be wrong, and tuning the rule away because of it would delete a genuinely valuable detection. The durable improvement is not a higher audit level — it is making exec into production rare and attributable: an approval path, short-lived access, and a recorded broker session so the next entry arrives with the transcript already attached.