skip to content

Which fields in a Kubernetes audit record of a Secret read are worth pivoting on?

level: juniorimportance: must knowfreq 68%

answer

  1. Every value is a possible search
  2. Who, what it touched, from where
  3. One field only links one request
  4. Addresses get reassigned; identities persist
  5. A 200 proves a token was accepted

basics

~20 s

The entity fields: the service-account username, the object's namespace and name, the source IP, and the user agent. Each becomes a search across other sources inside a time window. The audit ID only links stages of that one request.

solid answer

~50 s

A pivot is taking a value out of one record and searching every other source for it over a bounded window, so one alert grows into a picture. In this record the pivotable entities are the authenticated identity (`system:serviceaccount:ci:runner`), the object (`secrets/registry-push` in namespace `ci`), the source address, and the user agent as a weak hint. The identity is the strongest: it answers "what else did this credential touch around 02:14". The object answers "who else read this Secret, and how often is it read at all". The address answers only "what else came from there while that address was assigned". The audit ID is not a pivot — it correlates the stages of this single API call and no other system records it. Note what the record proves: the API server accepted a token bound to that account and returned the object. It does not prove a person was present or that the intended pipeline made the call.

code

json · 17 lines
json
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "9f2c8e1a-...",
  "stage": "ResponseComplete",
  "verb": "get",
  "requestURI": "/api/v1/namespaces/ci/secrets/registry-push",
  "user": { "username": "system:serviceaccount:ci:runner",
            "groups": ["system:serviceaccounts", "system:serviceaccounts:ci"] },
  "sourceIPs": ["10.42.7.19"],
  "userAgent": "kubectl/v1.29.4",
  "objectRef": { "resource": "secrets", "namespace": "ci", "name": "registry-push" },
  "responseStatus": { "code": 200 },
  "requestReceivedTimestamp": "2026-05-12T02:14:07.881Z",
  "stageTimestamp": "2026-05-12T02:14:07.894Z"
}

go deeper

for a junior

Be ready to point at a log record and name the values you could search for in other sources, saying in one sentence what each search would answer.

for a middle

Expect to explain why identities and object names pivot well while addresses and user agents are weak, and how the event timestamp bounds every search you run.

for a senior

Show that you pick the pivot by what result would change your verdict, and that you state what the record cannot prove before you build a story on it.

for a principal

Own the design consequence: pivots are only as sharp as the identities the estate issues, so broadly shared service accounts make every investigation ambiguous — that is a cost worth arguing in budget terms.

## What an entity pivot is An alert names a handful of concrete things: an account, a host, a process, a file hash, an address, an artefact. A **pivot** is taking one of those values, searching other sources for the same value over a bounded time window, and reading what comes back. Triage is mostly a sequence of pivots: the alert is one record, and the picture you eventually escalate or close is the set of records the pivots pulled in around it. So the first skill is looking at a record and seeing which values are *entities* — things other systems also write down — and which are only local decoration. ## The entities in this record **`user.username` — the authenticated identity.** This is what the API server accepted a credential for. Pivoting on it answers *what else did this identity do*: every other API call it made in the window, and, joined to build-system job records, which pipelines run as it. It is usually the strongest pivot in the record because other systems record identities too. **`objectRef` — the thing acted on.** Namespace `ci`, resource `secrets`, name `registry-push`. Pivoting here inverts the question: *who else touches this object, and how unusual is any read at all*. If that Secret is read forty times an hour by the deploy pipeline, one more read at 02:14 is not remarkable; if it is read twice a month, the timestamp is the finding. **`sourceIPs` — the address the API server saw.** Weak, and the reason is worth internalising: in a cluster this is often a pod address handed out by the CNI and handed to someone else minutes later, and in other estates it is a NAT or proxy address shared by everything. It is pivotable only *inside the window in which that assignment held*, and only if you can resolve the assignment as it stood at the event's timestamp rather than now. **`userAgent`** (`kubectl/v1.29.4`) is a hint, not an entity: it is attacker-controlled and trivially common. It is useful for the narrow question "did this identity normally call the API with a client library and suddenly call it with kubectl", which is a *behaviour* observation, not a join key. **Timestamps** define the window every other pivot runs in. Start tight — minutes either side — and widen deliberately. **`auditID`** is the one that looks like a pivot and is not. The API server assigns it per request so that the `RequestReceived` and `ResponseComplete` stages of the same call can be stitched together. Nothing outside the API server has ever seen it, so pivoting on it across other sources returns nothing by construction. ## What the record proves — and what it does not `responseStatus.code: 200` means the API server authenticated a token bound to that service account, authorised the `get`, and served the object. That is a statement about a **credential being accepted**, not about a human being present, and not about which pod or job held the token — many pods can mount the same service account, and a stolen token works exactly as well as a legitimate one. This matters because the direction of the claim survives into your write-up. "The CI service account read the registry Secret at 02:14" is defensible. "The CI pipeline read the registry Secret at 02:14" is a conclusion you have not yet earned; getting from the first sentence to the second is precisely what the next pivot is for. Equally, the audit level shown is `Metadata`, which means the request and response bodies were not recorded. The record tells you the Secret was served; it does not contain what was in it. ## Choosing where to start Two cheap heuristics carry a junior a long way: - **Prefer the specific entity to the shared one.** An identity used by one pipeline beats an identity used by forty; a unique object name beats a namespace. - **Say out loud what result would change your mind before you run the search.** "If this account made no other unusual calls in the ten minutes around 02:14, I close it" is a plan; "let me look around" is not. And bound every pivot in time. An unbounded search on a value that gets reused turns one alert into a pile of unrelated records, which feels like progress and is the opposite.

  • What does a 200 response on that audit record actually prove?
    That the API server authenticated a token bound to that service account, authorised the read, and served the object. It proves a credential was accepted and the Secret was returned — not that a human acted, and not which pod held the token, since any pod mounting that account presents the same identity.
  • Why is the auditID useless as a pivot across other log sources?
    The API server mints it per request purely to stitch that call's stages together. No other system has ever seen the value, so a search for it elsewhere returns nothing by construction. Pivots need values that two or more systems independently record — an identity, an image digest, an address, a hostname.
  • Which pivot would you run first on an alert of 'unusual Secret read at 02:14'?
    The service-account identity over a tight window either side of 02:14: every other API call that credential made. It is specific, it directly bounds what else the credential reached, and its result changes the next step. The source address comes second because it may be shared or already reassigned.

saying these in an interview costs you the question

  • Treats a source IP as a stable machine identity
  • Pivots on the audit ID expecting hits elsewhere
  • Reads a 200 response as proof a person acted
  • Runs pivots with no time window at all
  • Assumes a service account maps to one pipeline

context