skip to content

Widening the Picture

1 roadmap12 questions

One host and one account are never the whole story, and every extra pivot costs time you may not have. This is where interviewers put a live process chain on the whiteboard and watch you read it.

on this pageshow

explore

questions

12

Which fields in a Kubernetes audit record of a Secret read are worth pivoting on?

level: juniorimportance: must knowfreq 68%

answer

  1. Every value is a possible search
  2. Who, what it touched, from where
  3. One field only links one request
  4. Addresses get reassigned; identities persist
  5. A 200 proves a token was accepted

basics

~20 s

The entity fields: the service-account username, the object's namespace and name, the source IP, and the user agent. Each becomes a search across other sources inside a time window. The audit ID only links stages of that one request.

solid answer

~50 s

A pivot is taking a value out of one record and searching every other source for it over a bounded window, so one alert grows into a picture. In this record the pivotable entities are the authenticated identity (`system:serviceaccount:ci:runner`), the object (`secrets/registry-push` in namespace `ci`), the source address, and the user agent as a weak hint. The identity is the strongest: it answers "what else did this credential touch around 02:14". The object answers "who else read this Secret, and how often is it read at all". The address answers only "what else came from there while that address was assigned". The audit ID is not a pivot — it correlates the stages of this single API call and no other system records it. Note what the record proves: the API server accepted a token bound to that account and returned the object. It does not prove a person was present or that the intended pipeline made the call.

code

json · 17 lines
json
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "9f2c8e1a-...",
  "stage": "ResponseComplete",
  "verb": "get",
  "requestURI": "/api/v1/namespaces/ci/secrets/registry-push",
  "user": { "username": "system:serviceaccount:ci:runner",
            "groups": ["system:serviceaccounts", "system:serviceaccounts:ci"] },
  "sourceIPs": ["10.42.7.19"],
  "userAgent": "kubectl/v1.29.4",
  "objectRef": { "resource": "secrets", "namespace": "ci", "name": "registry-push" },
  "responseStatus": { "code": 200 },
  "requestReceivedTimestamp": "2026-05-12T02:14:07.881Z",
  "stageTimestamp": "2026-05-12T02:14:07.894Z"
}

go deeper

for a junior

Be ready to point at a log record and name the values you could search for in other sources, saying in one sentence what each search would answer.

for a middle

Expect to explain why identities and object names pivot well while addresses and user agents are weak, and how the event timestamp bounds every search you run.

for a senior

Show that you pick the pivot by what result would change your verdict, and that you state what the record cannot prove before you build a story on it.

for a principal

Own the design consequence: pivots are only as sharp as the identities the estate issues, so broadly shared service accounts make every investigation ambiguous — that is a cost worth arguing in budget terms.

## What an entity pivot is An alert names a handful of concrete things: an account, a host, a process, a file hash, an address, an artefact. A **pivot** is taking one of those values, searching other sources for the same value over a bounded time window, and reading what comes back. Triage is mostly a sequence of pivots: the alert is one record, and the picture you eventually escalate or close is the set of records the pivots pulled in around it. So the first skill is looking at a record and seeing which values are *entities* — things other systems also write down — and which are only local decoration. ## The entities in this record **`user.username` — the authenticated identity.** This is what the API server accepted a credential for. Pivoting on it answers *what else did this identity do*: every other API call it made in the window, and, joined to build-system job records, which pipelines run as it. It is usually the strongest pivot in the record because other systems record identities too. **`objectRef` — the thing acted on.** Namespace `ci`, resource `secrets`, name `registry-push`. Pivoting here inverts the question: *who else touches this object, and how unusual is any read at all*. If that Secret is read forty times an hour by the deploy pipeline, one more read at 02:14 is not remarkable; if it is read twice a month, the timestamp is the finding. **`sourceIPs` — the address the API server saw.** Weak, and the reason is worth internalising: in a cluster this is often a pod address handed out by the CNI and handed to someone else minutes later, and in other estates it is a NAT or proxy address shared by everything. It is pivotable only *inside the window in which that assignment held*, and only if you can resolve the assignment as it stood at the event's timestamp rather than now. **`userAgent`** (`kubectl/v1.29.4`) is a hint, not an entity: it is attacker-controlled and trivially common. It is useful for the narrow question "did this identity normally call the API with a client library and suddenly call it with kubectl", which is a *behaviour* observation, not a join key. **Timestamps** define the window every other pivot runs in. Start tight — minutes either side — and widen deliberately. **`auditID`** is the one that looks like a pivot and is not. The API server assigns it per request so that the `RequestReceived` and `ResponseComplete` stages of the same call can be stitched together. Nothing outside the API server has ever seen it, so pivoting on it across other sources returns nothing by construction. ## What the record proves — and what it does not `responseStatus.code: 200` means the API server authenticated a token bound to that service account, authorised the `get`, and served the object. That is a statement about a **credential being accepted**, not about a human being present, and not about which pod or job held the token — many pods can mount the same service account, and a stolen token works exactly as well as a legitimate one. This matters because the direction of the claim survives into your write-up. "The CI service account read the registry Secret at 02:14" is defensible. "The CI pipeline read the registry Secret at 02:14" is a conclusion you have not yet earned; getting from the first sentence to the second is precisely what the next pivot is for. Equally, the audit level shown is `Metadata`, which means the request and response bodies were not recorded. The record tells you the Secret was served; it does not contain what was in it. ## Choosing where to start Two cheap heuristics carry a junior a long way: - **Prefer the specific entity to the shared one.** An identity used by one pipeline beats an identity used by forty; a unique object name beats a namespace. - **Say out loud what result would change your mind before you run the search.** "If this account made no other unusual calls in the ten minutes around 02:14, I close it" is a plan; "let me look around" is not. And bound every pivot in time. An unbounded search on a value that gets reused turns one alert into a pile of unrelated records, which feels like progress and is the opposite.

  • What does a 200 response on that audit record actually prove?
    That the API server authenticated a token bound to that service account, authorised the read, and served the object. It proves a credential was accepted and the Secret was returned — not that a human acted, and not which pod held the token, since any pod mounting that account presents the same identity.
  • Why is the auditID useless as a pivot across other log sources?
    The API server mints it per request purely to stitch that call's stages together. No other system has ever seen the value, so a search for it elsewhere returns nothing by construction. Pivots need values that two or more systems independently record — an identity, an image digest, an address, a hostname.
  • Which pivot would you run first on an alert of 'unusual Secret read at 02:14'?
    The service-account identity over a tight window either side of 02:14: every other API call that credential made. It is specific, it directly bounds what else the credential reached, and its result changes the next step. The source address comes second because it may be shared or already reassigned.

saying these in an interview costs you the question

  • Treats a source IP as a stable machine identity
  • Pivots on the audit ID expecting hits elsewhere
  • Reads a 200 response as proof a person acted
  • Runs pivots with no time window at all
  • Assumes a service account maps to one pipeline

context

resolving links…

open as a page

An alert fires on powershell.exe with an encoded command line — what does its parent chain add to the verdict?

level: juniorimportance: must knowfreq 72%

basics

~20 s

The parent chain says who asked for it. Encoded PowerShell under a document application means a user opened something that executed code; the same command under a management agent's service means scheduled automation. The lineage, not the child, carries the verdict.

open as a page

One unexplained entry in a Linux server's ~/.ssh/authorized_keys: what defines the scope of your investigation?

level: juniorimportance: must knowfreq 72%

basics

~20 s

Scope has three axes: how many hosts carry the same artefact, which accounts it grants access to, and over what time window you can see. One host with no accounts and no time bound is a finding, not a scope.

open as a page

A build alert gives you a token id, an image digest and a source address — which pivot runs first?

level: seniorimportance: must knowfreq 50%

basics

~20 s

The one whose result would change your next decision, weighted by how selective it is. In a build-infrastructure intrusion that is normally the image digest, then the token id; a shared or NAT'd source address answers least and usually runs last.

open as a page

Two alerts a week apart share the CI service account ci-runner — what does that shared name prove?

level: middleimportance: should knowfreq 55%

basics

~20 s

Only that both events authenticated as the same identity. If every pipeline uses that account, the link is nearly worthless. It becomes evidence when a second, more selective entity such as an image digest is shared too.

open as a page

The same encoded PowerShell runs under WINWORD.EXE on one host and under an RMM agent on 4,000 — how does ancestry decide each verdict?

level: middleimportance: should knowfreq 62%

basics

~20 s

Compare the roots, not the leaves. One chain is rooted at an interactive user's document application; the other at the service control manager starting a management agent as SYSTEM in session 0. Identical children, different causes, opposite verdicts.

open as a page

An authorized_keys entry's mtime is 40 days old but auditd retains 14 days — how do you set your lookback window?

level: middleimportance: should knowfreq 52%

basics

~20 s

Set the window per source, not once for the case. auditd answers only its 14 days; reach further with config-management history, archived sshd authentication logs, bastion logs and host backups, and record whatever stays unreachable as a stated gap.

open as a page

A Sysmon Event ID 1 record names a parent that had already exited — how do you rebuild that lineage?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Join upward on the parent process GUID, never on the parent process id, because Windows reuses ids and a dead parent's number can be occupied by something unrelated. If the parent's own creation record is missing, the grandparent is unknowable and you say so.

open as a page

Four hours into an unexplained SSH key case with no second host and no execution — how do you close it?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Close it inconclusive rather than leaving it open. Record the population, accounts and per-source window you actually covered, the question still unanswered, the residual risk in plain words, an owner who accepts it, a dated re-check and a trigger that reopens the case.

open as a page

Scoping an unexplained authorized_keys entry, you find a config-management run deployed it — why not close as benign?

level: seniorimportance: should knowfreq 46%

basics

~20 s

Because the run explains the mechanism, not the authorisation. The next question is which commit put the key in the configuration repository, who wrote it and who reviewed it. That answer also widens the scope to every host in the group.

open as a page

The CI runner pod behind your alert is already deleted — how do you keep pivoting?

level: seniorimportance: nice to knowfreq 34%

basics

~20 s

Pivot on entities that outlive the workload: the identity it ran as, the image digest, the pipeline job id, the node. The pod's filesystem and memory are gone, and that is an evidence gap to state, not a clean result.

open as a page

Desktop engineering says the RMM chain on the alerted host 'is us' — is that enough to close the case?

level: seniorimportance: nice to knowfreq 38%

basics

~20 s

Not by itself. The owner's word establishes that such automation exists, not that this particular execution was theirs. Ask for something checkable — a job record with a run identifier and window, or the script that matches the decoded argument — then close.

open as a page