You must be able to show from Kubernetes audit records who read a database-credentials Secret, who opened kubectl exec sessions, and who changed RBAC bindings. What audit Policy rules do you write?

seniorshould knowfreq 45%

answer

three questions, three rules
values never belong in the log
subresources, not verbs
full body only where rare
exclusions can hide attackers

basics

~20 s

Record Secret get, list and watch at Metadata, pods/exec, attach and portforward at Metadata for any verb, and RBAC object writes at RequestResponse, all placed above any exclusion or broad rule. Never exclude service-account traffic wholesale.

solid answer

~40 s

I pin three rules near the top of the policy. First, secrets at Metadata - it records user, source IP, verb, namespace, name and the allow/deny decision, never the value; a list or watch has no name, so it means every Secret in scope was readable. Second, pods/exec, pods/attach and pods/portforward at Metadata without a verb filter; the requestURI carries the command, though not what was typed afterwards. Third, create, update, patch, delete and deletecollection on roles, rolebindings, clusterroles and clusterrolebindings in rbac.authorization.k8s.io at RequestResponse, so the exact grant is reconstructable. I add serviceaccounts/token at Metadata. Then I check that no None rule above them excludes a whole group such as system:serviceaccounts, because a stolen token would vanish, and I test each rule by generating the event.

I pin three rules near the top of the policy. First, `secrets` at `Metadata` - it records user, source IP, verb, namespace, name and the allow/deny decision, never the value; a `list` or `watch` has no name, so it means every Secret in scope was readable. Second, `pods/exec`, `pods/attach` and `pods/portforward` at `Metadata` without a verb filter; the `requestURI` carries the command, though not what was typed afterwards. Third, `create`, `update`, `patch`, `delete` and `deletecollection` on `roles`, `rolebindings`, `clusterroles` and `clusterrolebindings` in `rbac.authorization.k8s.io` at `RequestResponse`, so the exact grant is reconstructable. I add `serviceaccounts/token` at `Metadata`. Then I check that no `None` rule above them excludes a whole group such as `system:serviceaccounts`, because a stolen token would vanish, and I test each rule by generating the event.

code

yaml · 20 lines

yaml

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts/token"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  # Narrow noise exclusions and broader rules go below this line
  - level: Metadata

go deeper

for a junior

Recall the three things investigators ask: who read a Secret, who opened a shell into a Pod, and who changed permissions.

for a middle

Explain which resources and subresources those rules match, and why Secrets stay at Metadata while RBAC writes get full bodies.

for a senior

Demonstrate the traps: list without names, verb filters on exec, and broad None exclusions that hide stolen ServiceAccount tokens; prove each rule fires.

for a principal

Own the policy as a reviewed control with named consumers, and decide which gaps, such as in-session activity, are closed by other telemetry instead.

The questions an investigation will ask

When a credential leaks, the audit log is usually the only record inside Kubernetes that a Secret was read. Consider an internal wiki with an embedded database in namespace wiki; its database password lives in the Secret wiki-db-credentials and the database runs in Pod wiki-0. After a suspicious login to the database, the investigation needs three answers:

Who read wiki-db-credentials, from where, and was it allowed?
Did anyone open a shell or port-forward into wiki-0?
Did anyone grant themselves or a ServiceAccount new permissions beforehand?

The audit policy must be written so that each of those is answerable before the incident.

Rule 1 - Secret reads at Metadata

Match resources: [{group: "", resources: ["secrets"]}] with no verb filter, or with get, list, watch plus the write verbs.
Use Metadata. RequestResponse would copy every returned value into the log, and for a list that is every Secret in the namespace.
The event gives user.username, user.groups, sourceIPs, userAgent, verb, objectRef.namespace, objectRef.name, responseStatus.code and the authorization.k8s.io/decision annotation.
A list or watch has no objectRef.name. Treat a successful one as a read of every Secret it covered - the log cannot tell you which item the caller used.
Do not narrow with resourceNames: you want reads of Secrets you have not thought of yet.

Rule 2 - interactive access to Pods

Match pods/exec, pods/attach and pods/portforward (the subresource form resource/subresource).
Do not pin the verb: the verb recorded for a streaming request depends on how the client connects, and a verb filter can silently miss one transport.
Metadata is enough. The requestURI includes the query string, so for exec it shows the command and container parameters. The session's keystrokes and output are not recorded by Kubernetes audit.
Consider keeping the RequestReceived stage for this rule so an open session is visible before it ends.

Rule 3 - RBAC writes at RequestResponse

Match group rbac.authorization.k8s.io, resources roles, rolebindings, clusterroles, clusterrolebindings, verbs create, update, patch, delete, deletecollection.
Use RequestResponse so the event contains the submitted and resulting object: which subjects were bound to which role.
These writes are rare, so the detail costs little.
Add serviceaccounts/token (TokenRequest) at Metadata: minting a token for a ServiceAccount is how bound permissions get used from outside the cluster.

Where the rules go and what breaks them

Hazard	Effect	Guard
A broad body-level rule above rule 1	Secret values land in the log	Put rule 1 first
`None` for `userGroups: ["system:serviceaccounts"]`	Reads with a stolen SA token vanish	Exclude named controllers by `users`, and only for named resources
`None` for all `get` requests	Secret reads vanish	Never exclude by verb alone
Verb filter on exec	One client transport is missed	Match the subresource only
Policy changed without testing	A typo matches nothing	Generate each event and look for it

Impersonation is covered automatically: when someone uses --as, the event's user is the real caller and impersonatedUser is the identity assumed.

What each rule costs

Rule	Level	Relative rate	Body size
Secret access	`Metadata`	Moderate - controllers and workloads read Secrets	None
Exec, attach, port-forward	`Metadata`	Low - humans and a few tools	None
RBAC writes	`RequestResponse`	Very low outside cluster bootstrap and GitOps syncs	Small objects
Token requests	`Metadata`	Moderate - the kubelet requests projected tokens for Pods	None

The expensive part of an audit policy is almost never these rules; it is broad body-level rules on high-rate traffic. That is why the high-signal rules can stay generous while the rest of the policy is trimmed.

Proving the rules work

After rolling the policy out, perform each action with a test identity - read the Secret, open an exec session, create a RoleBinding - and confirm the matching events appear with the expected level. A rule that has never been seen firing is an assumption, not a control. Detection logic and correlation over these records belong to the SIEM side; the audit policy's job is that the records exist.

## The questions an investigation will ask When a credential leaks, the audit log is usually the only record inside Kubernetes that a Secret was read. Consider an internal wiki with an embedded database in namespace `wiki`; its database password lives in the Secret `wiki-db-credentials` and the database runs in Pod `wiki-0`. After a suspicious login to the database, the investigation needs three answers: 1. Who read `wiki-db-credentials`, from where, and was it allowed? 2. Did anyone open a shell or port-forward into `wiki-0`? 3. Did anyone grant themselves or a ServiceAccount new permissions beforehand? The audit policy must be written so that each of those is answerable *before* the incident. ## Rule 1 - Secret reads at Metadata - Match `resources: [{group: "", resources: ["secrets"]}]` with no verb filter, or with `get`, `list`, `watch` plus the write verbs. - Use `Metadata`. `RequestResponse` would copy every returned value into the log, and for a `list` that is every Secret in the namespace. - The event gives `user.username`, `user.groups`, `sourceIPs`, `userAgent`, `verb`, `objectRef.namespace`, `objectRef.name`, `responseStatus.code` and the `authorization.k8s.io/decision` annotation. - A `list` or `watch` has no `objectRef.name`. Treat a successful one as a read of **every** Secret it covered - the log cannot tell you which item the caller used. - Do not narrow with `resourceNames`: you want reads of Secrets you have not thought of yet. ## Rule 2 - interactive access to Pods - Match `pods/exec`, `pods/attach` and `pods/portforward` (the subresource form `resource/subresource`). - Do not pin the verb: the verb recorded for a streaming request depends on how the client connects, and a verb filter can silently miss one transport. - `Metadata` is enough. The `requestURI` includes the query string, so for exec it shows the `command` and `container` parameters. The session's keystrokes and output are **not** recorded by Kubernetes audit. - Consider keeping the `RequestReceived` stage for this rule so an open session is visible before it ends. ## Rule 3 - RBAC writes at RequestResponse - Match group `rbac.authorization.k8s.io`, resources `roles`, `rolebindings`, `clusterroles`, `clusterrolebindings`, verbs `create`, `update`, `patch`, `delete`, `deletecollection`. - Use `RequestResponse` so the event contains the submitted and resulting object: which subjects were bound to which role. - These writes are rare, so the detail costs little. - Add `serviceaccounts/token` (TokenRequest) at `Metadata`: minting a token for a ServiceAccount is how bound permissions get used from outside the cluster. ## Where the rules go and what breaks them | Hazard | Effect | Guard | |---|---|---| | A broad body-level rule above rule 1 | Secret values land in the log | Put rule 1 first | | `None` for `userGroups: ["system:serviceaccounts"]` | Reads with a stolen SA token vanish | Exclude named controllers by `users`, and only for named resources | | `None` for all `get` requests | Secret reads vanish | Never exclude by verb alone | | Verb filter on exec | One client transport is missed | Match the subresource only | | Policy changed without testing | A typo matches nothing | Generate each event and look for it | Impersonation is covered automatically: when someone uses `--as`, the event's `user` is the real caller and `impersonatedUser` is the identity assumed. ## What each rule costs | Rule | Level | Relative rate | Body size | |---|---|---|---| | Secret access | `Metadata` | Moderate - controllers and workloads read Secrets | None | | Exec, attach, port-forward | `Metadata` | Low - humans and a few tools | None | | RBAC writes | `RequestResponse` | Very low outside cluster bootstrap and GitOps syncs | Small objects | | Token requests | `Metadata` | Moderate - the kubelet requests projected tokens for Pods | None | The expensive part of an audit policy is almost never these rules; it is broad body-level rules on high-rate traffic. That is why the high-signal rules can stay generous while the rest of the policy is trimmed. ## Proving the rules work After rolling the policy out, perform each action with a test identity - read the Secret, open an exec session, create a RoleBinding - and confirm the matching events appear with the expected level. A rule that has never been seen firing is an assumption, not a control. Detection logic and correlation over these records belong to the SIEM side; the audit policy's job is that the records exist.

A Kubernetes audit event shows a successful list of secrets in namespace wiki by a ServiceAccount. Which Secrets did that caller obtain?
Potentially all of them. A `list` event has no object name and, at `Metadata`, no response body, so the log cannot enumerate the returned items. You must assume every Secret in `wiki` the caller could list was disclosed and rotate accordingly. That uncertainty is also why `list` on Secrets deserves the same scrutiny as `get` in RBAC.
Why not simply audit everything in a Kubernetes cluster at RequestResponse and filter later?
Two reasons. It copies every Secret and token value that crosses the API into the log, turning the audit store into the most sensitive dataset in the organisation. And response bodies for lists and watches are large, so volume and API server write cost grow sharply. Targeted detail on rare, high-value writes gives the forensic value without either cost.
What does a Kubernetes audit record of kubectl exec not tell you?
It shows who opened the session, into which Pod and container, from where, with which initial command in the `requestURI`, and when it ended. It does not capture what was typed or printed inside the session. Command-level activity inside the container needs runtime or host telemetry, which is outside the API server's view.

saying these in an interview costs you the question

Auditing Secrets at RequestResponse is needed to prove what was read
A list on Secrets only exposes Secret names
Audit logs record every command typed in an exec session
Excluding all ServiceAccount traffic is safe because it is just controllers
Filtering exec rules on the create verb is always sufficient
Impersonated requests are logged only under the assumed identity

You must be able to show from Kubernetes audit records who read a database-credentials Secret, who opened kubectl exec sessions, and who changed RBAC bindings. What audit Policy rules do you write?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The questions an investigation will ask

Rule 1 - Secret reads at Metadata

Rule 2 - interactive access to Pods

Rule 3 - RBAC writes at RequestResponse

Where the rules go and what breaks them

What each rule costs

Proving the rules work

▾common follow-ups3 questions

▾red flags6 flags

The questions an investigation will ask

Rule 1 - Secret reads at Metadata

Rule 2 - interactive access to Pods

Rule 3 - RBAC writes at RequestResponse

Where the rules go and what breaks them

What each rule costs

Proving the rules work

answer

go deeper