skip to content

API Audit Logging

7 roadmaps4 questions

The API server records only what an audit policy tells it to: rules match on user, verb, resource and namespace, and each picks a level from None up to RequestResponse. Audit is the only trace that a Secret was ever read.

part ofKubernetesoverview, primer and where to startread it →
on this pageshow

explore

questions

4

How does a Kubernetes audit Policy choose what to record for an API request, and what does each audit level capture?

level: middleimportance: must knowfreq 55%

answer

  1. ordered list, earliest match decides
  2. unmatched means nothing written
  3. four levels, growing payload
  4. bodies can copy Secret data
  5. narrow rules before broad ones

basics

~20 s

kube-apiserver checks the audit Policy's rules in order; the first rule matching the request's user, verb, resource and namespace sets its level, and unmatched requests are not logged. Metadata records who did what; Request adds the request body; RequestResponse adds the response.

solid answer

~40 s

Auditing only happens when kube-apiserver is started with `--audit-policy-file` and a backend. The `audit.k8s.io/v1` `Policy` holds an ordered list of `rules`; each can match on `users`, `userGroups`, `verbs`, `resources` (group, resource, optional `resourceNames`), `namespaces` or `nonResourceURLs`, and the **first matching rule wins**. A request that matches nothing gets level `None` and is not logged. `Metadata` records user, groups, source IPs, user agent, verb, the object reference and the response code. `Request` adds the request body, and `RequestResponse` adds the response body too. That is why Secrets get a `Metadata` rule placed above any broader rule: a body-level rule would copy Secret values into the log.

code

yaml · 22 lines
yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Secret and ConfigMap contents must never reach the log
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  # Probe endpoints are pure noise
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*"]
  # Record wiki workload changes with the full object
  - level: RequestResponse
    namespaces: ["wiki"]
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "apps"
        resources: ["deployments", "statefulsets"]
  # Everything else: who, what, when, outcome
  - level: Metadata

go deeper

for a junior

Remember that auditing is off until kube-apiserver gets a policy file and a backend, and name the four levels in order of growing detail.

for a middle

Explain first-match-wins evaluation, the fields a rule can match on, and exactly what Metadata, Request and RequestResponse add to an event.

for a senior

Show that rule order is the design: sensitive resources pinned to Metadata above any body-level rule, narrow exclusions, and a Metadata catch-all so nothing silently drops.

for a principal

Frame the policy as a deliberate tradeoff between forensic completeness, secret exposure in the log and volume, and own how it is reviewed and tested before rollout.

## What an audit event is The Kubernetes API server can write a chronological, structured record of the requests it handles. Each record is an **audit event** of kind `Event` in the `audit.k8s.io/v1` API group. It answers who (`user`, `impersonatedUser`, `sourceIPs`, `userAgent`), what (`verb`, `requestURI`, `objectRef` with resource, namespace, name and subresource), when (`requestReceivedTimestamp`, `stageTimestamp`) and with what outcome (`responseStatus`, plus annotations such as `authorization.k8s.io/decision`). None of this happens by default. kube-apiserver needs two things: 1. `--audit-policy-file` pointing at a `Policy` object, which decides *what* is recorded. 2. At least one backend: `--audit-log-path` for a file (`-` means standard out) or `--audit-webhook-config-file` for a remote receiver. With a backend but no policy file, the server records nothing. A policy file with zero rules, or one that fails validation, is rejected when the server loads it. ## How rules match The `Policy` has an ordered `rules` list. For every request the server walks the list top to bottom and stops at the **first rule whose conditions all match**. The fields a rule can match on are: - `users` - exact usernames, such as `system:kube-scheduler`. - `userGroups` - any group the requester belongs to, such as `system:nodes`. - `verbs` - `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` and so on. - `resources` - a list of `group` plus `resources`, where `""` is the core group, `pods/exec` names a subresource, and `resourceNames` narrows to named objects. - `namespaces` - the request's namespace; an empty string matches cluster-scoped objects. - `nonResourceURLs` - paths such as `/healthz*`, for requests that are not about an API object. An omitted field means "any". A rule with only `level` matches everything, which makes it a natural catch-all as the last rule. If no rule matches, the event level is `None`. ## The four levels | Level | What the event contains | Typical use | |---|---|---| | `None` | Nothing is recorded | Health probes, known high-volume noise | | `Metadata` | User, verb, object reference, source IPs, user agent, response status, timestamps | Secrets, ConfigMaps, token requests, the default catch-all | | `Request` | Metadata plus `requestObject` | Writes where the submitted object matters but the response is bulky | | `RequestResponse` | Request plus `responseObject` | Changes you must reconstruct exactly, such as RBAC edits | Two details matter. A `get` has no request body, so `Request` on a read adds nothing over `Metadata`; `RequestResponse` on a read copies the returned object. And bodies are only attached to resource requests, not to non-resource URLs. ## Why order is the whole design Take an internal wiki with an embedded database running in namespace `wiki`, whose credentials live in the Secret `wiki-db-credentials`. Suppose the policy starts with a broad rule "everything in namespace `wiki` at `RequestResponse`" so that workload changes are recorded in full. A read of that Secret matches the broad rule first, and the Secret's data is written into the audit log in plain base64 - the log is now a second copy of the credential. The fix is ordering, not a different level on the broad rule: - put a `Metadata` rule for `secrets` (and usually `configmaps` and `serviceaccounts/token`) **above** every body-level rule; - put deliberate `None` exclusions next, and keep them narrow; - put detailed rules for the workloads you care about after that; - end with a `Metadata` catch-all so nothing silently drops to `None`. With that policy, when someone patches the wiki Deployment to raise its container's CPU request from a 0.35-core request (`350m`) to `2`, the event carries the patch body and the resulting object, so the change can be reconstructed exactly. A read of the Secret still records who read it, from which address and whether it was allowed - without the value. ## Reading an event A `Metadata` event for the Secret read above carries, among other fields: - `level: Metadata` and `stage: ResponseComplete`; - `verb: get` and `requestURI` with the full path; - `user.username` and `user.groups` of the caller, plus `impersonatedUser` if `--as` was used; - `objectRef` with `resource: secrets`, `namespace: wiki`, `name: wiki-db-credentials`; - `responseStatus.code`, such as 200 for an allowed read or 403 for a denied one; - `annotations` with the authorizer's decision and reason. That is enough to answer who touched which object, when, from where and whether it worked. Everything beyond it - the object itself - is what the higher levels add, and what you must decide deliberately per resource. ## Operational notes - The policy file is read when kube-apiserver starts; changing it means restarting the API server, and on a kubeadm-style control plane the file must be mounted into the static Pod. - Policy-level `omitStages` and `omitManagedFields` set defaults for every rule; a rule can add its own. - Test a new policy on a non-production cluster first: a misspelled resource or user name passes validation and simply matches nothing.

  • In a Kubernetes audit Policy, what happens if a narrow None rule for a noisy controller sits below a broad Metadata catch-all?
    The None rule never fires. Rules are evaluated in order and the first match wins, so a rule with only `level: Metadata` matches every request before the evaluator reaches the exclusion. The controller's traffic is logged at `Metadata`, and volume stays high. Exclusions and specific rules must come before the catch-all, which belongs at the very end.
  • Why is `Request` level on a Kubernetes Secret still a leak even though a `get` has no request body?
    Because writes do carry a body. A `create`, `update` or `patch` on a Secret sends the data itself, and at `Request` level that body is stored as `requestObject`. Anyone who can read the audit log can then read the credential. Keeping Secrets at `Metadata` records the write - who, when, which object, what result - without the value.
  • Which Kubernetes audit event field tells you whether a request was allowed, and why is a denied request still worth recording?
    The `responseStatus` code shows the outcome, and the `authorization.k8s.io/decision` and `authorization.k8s.io/reason` annotations show what the authorizer decided and why. Denied requests matter because an attacker probing with a stolen credential produces a run of 403s before finding something that works; the failed attempts are often the earliest trace.

It works like a firewall rule table read top to bottom: the first line that fits the packet decides its fate, so a broad line placed too high swallows every specific line below it.

saying these in an interview costs you the question

  • The API server audits every request by default once the cluster is running
  • The most detailed matching rule wins regardless of where it sits
  • RequestResponse on Secrets is fine because the data is only base64
  • Metadata level records the request body but not the response
  • A request that matches no rule is logged at Metadata
open as a page

You must be able to show from Kubernetes audit records who read a database-credentials Secret, who opened kubectl exec sessions, and who changed RBAC bindings. What audit Policy rules do you write?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Record Secret get, list and watch at Metadata, pods/exec, attach and portforward at Metadata for any verb, and RBAC object writes at RequestResponse, all placed above any exclusion or broad rule. Never exclude service-account traffic wholesale.

open as a page

On a busy Kubernetes cluster, how do you balance audit coverage against API server latency and audit volume, and how do you choose between the log and webhook audit backends and their modes?

level: principalimportance: should knowfreq 30%

basics

~20 s

Budget volume through the policy: drop or thin high-rate system traffic, omit RequestReceived, keep bodies only for rare writes. Then choose backend modes by what must win when the sink fails: blocking protects completeness, batch protects latency and may drop events.

open as a page

Why can a single Kubernetes API request produce several audit events, and why do audit Policies often omit the RequestReceived stage?

level: middleimportance: nice to knowfreq 25%

basics

~20 s

kube-apiserver can emit one audit event per stage - RequestReceived, ResponseStarted for long-running calls, ResponseComplete, and Panic - all sharing one auditID. ResponseComplete already carries the outcome, so omitting RequestReceived roughly halves volume for ordinary requests.

open as a page