skip to content

Your broker's audit records land on a stream in the same cluster, writable by the principals they describe. What does that cost you?

level: seniorimportance: must knowfreq 50%

answer

  1. a witness who was in the building
  2. independence, then survivability
  3. different custody, not just elsewhere
  4. reach beyond the time to notice
  5. which grant allowed it, and when

basics

~20 s

Two properties. A principal with write access to the trail can edit the account of itself, so the records stop being evidence; and the trail dies with the cluster it describes. Ship it off-cluster, append-only, under different custody.

solid answer

~50 s

It costs independence and survivability. A trail that any administrator of the cluster can write to is not evidence about those administrators — anyone able to edit the account of their own actions makes the record a convenience rather than a proof, and that is the one job an audit trail has. It also shares the fate of its subject: if the cluster is lost, rebuilt or restored to an earlier point, the explanation goes with the thing it was meant to explain. The repair is three-part: ship records to a destination whose credentials the cluster's administrators do not hold; make that destination append-only for the identity shipping to it, so records can be added and not amended or removed; and set the audit retention period from how long it takes to notice an incident, not from convenience — misuse is usually found months after it starts.

go deeper

for a junior

Remember the core idea: a record kept inside the system it describes, writable by the people it describes, cannot be used to answer questions about those people.

for a middle

Explain both failures — editability by the subjects and loss alongside the cluster — and know that the audit retention period is a separate decision from how long streams keep their data.

for a senior

Demonstrate the full repair: separate custody, append-only for the shipping identity, retention derived from detection time, grant-table history kept alongside, and a periodic check that records still arrive.

for a principal

Set the separation-of-custody rule for the estate and defend the retention number on detection grounds when it is challenged on cost, while stating plainly what the trail still cannot prove.

## The two properties a co-located trail gives up **Independence.** An audit trail exists to answer a question about people who had power over the system. If the trail lives on the cluster, then a principal with write grants over that stream — which in practice means every administrator, and often the bootstrap identity as well — can append to it, rewrite it, or remove the part that concerns them. Nothing about that is exotic; it is the ordinary consequence of storing the account of an action inside the thing the actor controls. The record can still be useful operationally, for finding a misconfigured client or an unexpected source address. It is not evidence about the people who could edit it, and those are exactly the people an investigation is about. **Survivability.** The trail shares the fate of its subject. A cluster that is lost, rebuilt, or restored to an earlier point in time takes its own explanation with it — and the moment you most want the trail is a moment when the cluster is in one of those states. A trail stored elsewhere survives the event it describes; a trail stored on the cluster is a witness who was in the building. A third property is usually damaged at the same time, and it is the one teams notice last. ## Reach: the audit retention period is set by how long it takes to notice The useful length of an audit trail is not a storage decision, it is a detection decision. Quiet misuse — a grant handed out for a migration and never withdrawn, an integration reading a stream nobody remembered it could reach — is typically discovered long after it begins, often by an unrelated review rather than by an alarm. A trail reaching back a week can only ever explain the last week, so when the discovery is months old the trail contributes nothing. The rule to carry is: **the audit retention period must exceed the time it plausibly takes you to notice this class of problem.** Derive it from how such things have actually come to light in your organisation, then round up. Note carefully that this has nothing to do with how long the streams themselves keep their data — those are separate lifetimes with separate owners, and it is common and correct for a trail to outlive by a long way the records whose access it documents. ## The repair, in order 1. **Move the destination off the cluster.** Ship records to a store the broker's administrators cannot write to. The point is not distance, it is different custody — a destination whose credentials are held by a different set of people. 2. **Make it append-only for the shipping identity.** The identity that delivers records needs permission to add and nothing else: no amend, no delete, no lifecycle change. If that identity can also remove records, you have moved the problem rather than solved it. 3. **Set the audit retention period from detection time.** See above. Write the number down with the reasoning, because it will be challenged on cost and the reasoning is the defence. 4. **Keep the grant table's history alongside the trail.** Decision records say what was allowed. They do not say which grant allowed it, or that the grant was added the previous afternoon. Without a history of the grant table, a review can reconstruct what happened and not whether it should have been permitted at the time. 5. **Verify the path, periodically.** A shipping path that has silently stopped looks exactly like a cluster on which nothing interesting happened. Confirm that records produced today arrive at the destination today. ## What this looks like when it is wrong | Symptom | What has actually failed | |---|---| | The trail is complete but everyone with access to the cluster could edit it | Independence: useful operationally, worthless as evidence | | The trail was on the cluster that was restored to an earlier point | Survivability: the gap covers precisely the window of interest | | Records reach back three weeks; the grant was abused for seven months | Reach: the audit retention period was set by convenience | | The trail shows the access but not which grant permitted it | No grant-table history: what happened is known, whether it was allowed is not | | Records stop arriving in the destination and nobody notices for a quarter | Unverified path: silence is indistinguishable from quiet | ## The honest limits Off-cluster, append-only storage raises the cost of tampering; it does not make tampering impossible, because someone administers the destination too. The aim is separation of the two roles, so that falsifying the record requires compromising two sets of custody rather than one. Similarly, a trail proves what the broker decided — it cannot prove what a permitted principal did with what it read, and it cannot record an access that happened through a path the broker never evaluated.

  • Why keep a history of the grant table alongside the audit trail?
    Because decision records say what was allowed, not what should have been. Reading a year-old access against today's grant table tells you nothing about whether the grant existed then, who added it, or whether it was added the day before the access. Without that history a review reconstructs events but cannot judge them.
  • How do you set the audit retention period honestly?
    From detection time, not from cost. Look at how problems of this class have actually surfaced in your organisation — usually through an unrelated review, months in — take the longest realistic interval, and round up. Record the reasoning with the number, because the number gets challenged on cost and the reasoning is what defends it.
  • Does shipping records off the cluster make the trail tamper-proof?
    No, it makes tampering require two compromises instead of one. Someone administers the destination as well. The goal is separation of custody: the people who can change the cluster cannot change its record, and the people who can change the record have no reason to. Append-only permissions for the shipping identity harden that further.

A trail on the cluster it describes is a shop's own till roll kept in the till, with the keys held by whoever works the counter. It records honest days perfectly well, which is the reason nobody questions it, and it is worth nothing on the one day that matters.

saying these in an interview costs you the question

  • Calling a trail evidence when its subjects can write to it
  • Setting the audit retention period from storage cost alone
  • Assuming the trail survives a cluster restored to an earlier point
  • Reading old decision records against today's grant table
  • Never checking that records still arrive at the destination
  • Believing off-cluster storage makes tampering impossible