How does a Kubernetes audit Policy choose what to record for an API request, and what does each audit level capture?

middlemust knowfreq 55%

answer

ordered list, earliest match decides
unmatched means nothing written
four levels, growing payload
bodies can copy Secret data
narrow rules before broad ones

basics

~20 s

kube-apiserver checks the audit Policy's rules in order; the first rule matching the request's user, verb, resource and namespace sets its level, and unmatched requests are not logged. Metadata records who did what; Request adds the request body; RequestResponse adds the response.

solid answer

~40 s

Auditing only happens when kube-apiserver is started with --audit-policy-file and a backend. The audit.k8s.io/v1 Policy holds an ordered list of rules; each can match on users, userGroups, verbs, resources (group, resource, optional resourceNames), namespaces or nonResourceURLs, and the first matching rule wins. A request that matches nothing gets level None and is not logged. Metadata records user, groups, source IPs, user agent, verb, the object reference and the response code. Request adds the request body, and RequestResponse adds the response body too. That is why Secrets get a Metadata rule placed above any broader rule: a body-level rule would copy Secret values into the log.

Auditing only happens when kube-apiserver is started with `--audit-policy-file` and a backend. The `audit.k8s.io/v1` `Policy` holds an ordered list of `rules`; each can match on `users`, `userGroups`, `verbs`, `resources` (group, resource, optional `resourceNames`), `namespaces` or `nonResourceURLs`, and the **first matching rule wins**. A request that matches nothing gets level `None` and is not logged. `Metadata` records user, groups, source IPs, user agent, verb, the object reference and the response code. `Request` adds the request body, and `RequestResponse` adds the response body too. That is why Secrets get a `Metadata` rule placed above any broader rule: a body-level rule would copy Secret values into the log.

code

yaml · 22 lines

yaml

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Secret and ConfigMap contents must never reach the log
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  # Probe endpoints are pure noise
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*"]
  # Record wiki workload changes with the full object
  - level: RequestResponse
    namespaces: ["wiki"]
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "apps"
        resources: ["deployments", "statefulsets"]
  # Everything else: who, what, when, outcome
  - level: Metadata

go deeper

for a junior

Remember that auditing is off until kube-apiserver gets a policy file and a backend, and name the four levels in order of growing detail.

for a middle

Explain first-match-wins evaluation, the fields a rule can match on, and exactly what Metadata, Request and RequestResponse add to an event.

for a senior

Show that rule order is the design: sensitive resources pinned to Metadata above any body-level rule, narrow exclusions, and a Metadata catch-all so nothing silently drops.

for a principal

Frame the policy as a deliberate tradeoff between forensic completeness, secret exposure in the log and volume, and own how it is reviewed and tested before rollout.

What an audit event is

The Kubernetes API server can write a chronological, structured record of the requests it handles. Each record is an audit event of kind Event in the audit.k8s.io/v1 API group. It answers who (user, impersonatedUser, sourceIPs, userAgent), what (verb, requestURI, objectRef with resource, namespace, name and subresource), when (requestReceivedTimestamp, stageTimestamp) and with what outcome (responseStatus, plus annotations such as authorization.k8s.io/decision).

None of this happens by default. kube-apiserver needs two things:

--audit-policy-file pointing at a Policy object, which decides what is recorded.
At least one backend: --audit-log-path for a file (- means standard out) or --audit-webhook-config-file for a remote receiver.

With a backend but no policy file, the server records nothing. A policy file with zero rules, or one that fails validation, is rejected when the server loads it.

How rules match

The Policy has an ordered rules list. For every request the server walks the list top to bottom and stops at the first rule whose conditions all match. The fields a rule can match on are:

users - exact usernames, such as system:kube-scheduler.
userGroups - any group the requester belongs to, such as system:nodes.
verbs - get, list, watch, create, update, patch, delete and so on.
resources - a list of group plus resources, where "" is the core group, pods/exec names a subresource, and resourceNames narrows to named objects.
namespaces - the request's namespace; an empty string matches cluster-scoped objects.
nonResourceURLs - paths such as /healthz*, for requests that are not about an API object.

An omitted field means "any". A rule with only level matches everything, which makes it a natural catch-all as the last rule. If no rule matches, the event level is None.

The four levels

Level	What the event contains	Typical use
`None`	Nothing is recorded	Health probes, known high-volume noise
`Metadata`	User, verb, object reference, source IPs, user agent, response status, timestamps	Secrets, ConfigMaps, token requests, the default catch-all
`Request`	Metadata plus `requestObject`	Writes where the submitted object matters but the response is bulky
`RequestResponse`	Request plus `responseObject`	Changes you must reconstruct exactly, such as RBAC edits

Two details matter. A get has no request body, so Request on a read adds nothing over Metadata; RequestResponse on a read copies the returned object. And bodies are only attached to resource requests, not to non-resource URLs.

Why order is the whole design

Take an internal wiki with an embedded database running in namespace wiki, whose credentials live in the Secret wiki-db-credentials. Suppose the policy starts with a broad rule "everything in namespace wiki at RequestResponse" so that workload changes are recorded in full. A read of that Secret matches the broad rule first, and the Secret's data is written into the audit log in plain base64 - the log is now a second copy of the credential.

The fix is ordering, not a different level on the broad rule:

put a Metadata rule for secrets (and usually configmaps and serviceaccounts/token) above every body-level rule;
put deliberate None exclusions next, and keep them narrow;
put detailed rules for the workloads you care about after that;
end with a Metadata catch-all so nothing silently drops to None.

With that policy, when someone patches the wiki Deployment to raise its container's CPU request from a 0.35-core request (350m) to 2, the event carries the patch body and the resulting object, so the change can be reconstructed exactly. A read of the Secret still records who read it, from which address and whether it was allowed - without the value.

Reading an event

A Metadata event for the Secret read above carries, among other fields:

level: Metadata and stage: ResponseComplete;
verb: get and requestURI with the full path;
user.username and user.groups of the caller, plus impersonatedUser if --as was used;
objectRef with resource: secrets, namespace: wiki, name: wiki-db-credentials;
responseStatus.code, such as 200 for an allowed read or 403 for a denied one;
annotations with the authorizer's decision and reason.

That is enough to answer who touched which object, when, from where and whether it worked. Everything beyond it - the object itself - is what the higher levels add, and what you must decide deliberately per resource.

Operational notes

The policy file is read when kube-apiserver starts; changing it means restarting the API server, and on a kubeadm-style control plane the file must be mounted into the static Pod.
Policy-level omitStages and omitManagedFields set defaults for every rule; a rule can add its own.
Test a new policy on a non-production cluster first: a misspelled resource or user name passes validation and simply matches nothing.

## What an audit event is The Kubernetes API server can write a chronological, structured record of the requests it handles. Each record is an **audit event** of kind `Event` in the `audit.k8s.io/v1` API group. It answers who (`user`, `impersonatedUser`, `sourceIPs`, `userAgent`), what (`verb`, `requestURI`, `objectRef` with resource, namespace, name and subresource), when (`requestReceivedTimestamp`, `stageTimestamp`) and with what outcome (`responseStatus`, plus annotations such as `authorization.k8s.io/decision`). None of this happens by default. kube-apiserver needs two things: 1. `--audit-policy-file` pointing at a `Policy` object, which decides *what* is recorded. 2. At least one backend: `--audit-log-path` for a file (`-` means standard out) or `--audit-webhook-config-file` for a remote receiver. With a backend but no policy file, the server records nothing. A policy file with zero rules, or one that fails validation, is rejected when the server loads it. ## How rules match The `Policy` has an ordered `rules` list. For every request the server walks the list top to bottom and stops at the **first rule whose conditions all match**. The fields a rule can match on are: - `users` - exact usernames, such as `system:kube-scheduler`. - `userGroups` - any group the requester belongs to, such as `system:nodes`. - `verbs` - `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` and so on. - `resources` - a list of `group` plus `resources`, where `""` is the core group, `pods/exec` names a subresource, and `resourceNames` narrows to named objects. - `namespaces` - the request's namespace; an empty string matches cluster-scoped objects. - `nonResourceURLs` - paths such as `/healthz*`, for requests that are not about an API object. An omitted field means "any". A rule with only `level` matches everything, which makes it a natural catch-all as the last rule. If no rule matches, the event level is `None`. ## The four levels | Level | What the event contains | Typical use | |---|---|---| | `None` | Nothing is recorded | Health probes, known high-volume noise | | `Metadata` | User, verb, object reference, source IPs, user agent, response status, timestamps | Secrets, ConfigMaps, token requests, the default catch-all | | `Request` | Metadata plus `requestObject` | Writes where the submitted object matters but the response is bulky | | `RequestResponse` | Request plus `responseObject` | Changes you must reconstruct exactly, such as RBAC edits | Two details matter. A `get` has no request body, so `Request` on a read adds nothing over `Metadata`; `RequestResponse` on a read copies the returned object. And bodies are only attached to resource requests, not to non-resource URLs. ## Why order is the whole design Take an internal wiki with an embedded database running in namespace `wiki`, whose credentials live in the Secret `wiki-db-credentials`. Suppose the policy starts with a broad rule "everything in namespace `wiki` at `RequestResponse`" so that workload changes are recorded in full. A read of that Secret matches the broad rule first, and the Secret's data is written into the audit log in plain base64 - the log is now a second copy of the credential. The fix is ordering, not a different level on the broad rule: - put a `Metadata` rule for `secrets` (and usually `configmaps` and `serviceaccounts/token`) **above** every body-level rule; - put deliberate `None` exclusions next, and keep them narrow; - put detailed rules for the workloads you care about after that; - end with a `Metadata` catch-all so nothing silently drops to `None`. With that policy, when someone patches the wiki Deployment to raise its container's CPU request from a 0.35-core request (`350m`) to `2`, the event carries the patch body and the resulting object, so the change can be reconstructed exactly. A read of the Secret still records who read it, from which address and whether it was allowed - without the value. ## Reading an event A `Metadata` event for the Secret read above carries, among other fields: - `level: Metadata` and `stage: ResponseComplete`; - `verb: get` and `requestURI` with the full path; - `user.username` and `user.groups` of the caller, plus `impersonatedUser` if `--as` was used; - `objectRef` with `resource: secrets`, `namespace: wiki`, `name: wiki-db-credentials`; - `responseStatus.code`, such as 200 for an allowed read or 403 for a denied one; - `annotations` with the authorizer's decision and reason. That is enough to answer who touched which object, when, from where and whether it worked. Everything beyond it - the object itself - is what the higher levels add, and what you must decide deliberately per resource. ## Operational notes - The policy file is read when kube-apiserver starts; changing it means restarting the API server, and on a kubeadm-style control plane the file must be mounted into the static Pod. - Policy-level `omitStages` and `omitManagedFields` set defaults for every rule; a rule can add its own. - Test a new policy on a non-production cluster first: a misspelled resource or user name passes validation and simply matches nothing.

In a Kubernetes audit Policy, what happens if a narrow None rule for a noisy controller sits below a broad Metadata catch-all?
The None rule never fires. Rules are evaluated in order and the first match wins, so a rule with only `level: Metadata` matches every request before the evaluator reaches the exclusion. The controller's traffic is logged at `Metadata`, and volume stays high. Exclusions and specific rules must come before the catch-all, which belongs at the very end.
Why is `Request` level on a Kubernetes Secret still a leak even though a `get` has no request body?
Because writes do carry a body. A `create`, `update` or `patch` on a Secret sends the data itself, and at `Request` level that body is stored as `requestObject`. Anyone who can read the audit log can then read the credential. Keeping Secrets at `Metadata` records the write - who, when, which object, what result - without the value.
Which Kubernetes audit event field tells you whether a request was allowed, and why is a denied request still worth recording?
The `responseStatus` code shows the outcome, and the `authorization.k8s.io/decision` and `authorization.k8s.io/reason` annotations show what the authorizer decided and why. Denied requests matter because an attacker probing with a stolen credential produces a run of 403s before finding something that works; the failed attempts are often the earliest trace.

It works like a firewall rule table read top to bottom: the first line that fits the packet decides its fate, so a broad line placed too high swallows every specific line below it.

saying these in an interview costs you the question

The API server audits every request by default once the cluster is running
The most detailed matching rule wins regardless of where it sits
RequestResponse on Secrets is fine because the data is only base64
Metadata level records the request body but not the response
A request that matches no rule is logged at Metadata

How does a Kubernetes audit Policy choose what to record for an API request, and what does each audit level capture?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What an audit event is

How rules match

The four levels

Why order is the whole design

Reading an event

Operational notes

▾common follow-ups3 questions

▾analogy~10 s

▾red flags5 flags

What an audit event is

How rules match

The four levels

Why order is the whole design

Reading an event

Operational notes

answer

go deeper