skip to content

Payloads on a long-retained stream were encrypted by their writers, so a year on, what does custody of those encryption keys decide?

level: principalimportance: should knowfreq 33%

answer

  1. the key outranks the grant table
  2. retroactive access to written history
  3. losing it is a durability event
  4. history stays bound to its original key
  5. custody must outlive the retention window

basics

~20 s

Custody decides who can read the stream's history, including records written long before the current custodians arrived. It is an access decision and a durability decision at once: withdraw or lose the encryption key and the bytes survive as bytes nobody can open, whatever the grant table says.

solid answer

~50 s

Once payloads are opaque to the cluster, the grant table no longer decides who reads history — the encryption key does. Whoever holds the key the records were written under can read every record written under it, including everything produced before they took the role, and can deny that to anyone else regardless of broker-side grants. That makes custody a standing organisational decision rather than a setup step: who holds it, who inherits it when a team is reorganised or dissolved, and what happens to a year of history if nobody does. It is also a durability question, because losing the key leaves the records present and permanently unreadable. And because retained records were already written, moving to a new encryption key only affects future writes; the old history stays bound to the old key.

go deeper

for a junior

Recall the basic fact: if the records were encrypted before reaching the broker, only whoever holds that encryption key can read them, no matter what the cluster allows.

for a middle

Explain why a key change does not reach records already written, and what a record must carry outside the ciphertext for a consumer to read across several keys.

for a senior

Show that you treat key custody as a durability dependency as well as an access one, and that you would test recovering it rather than assume it.

for a principal

Own the estate-wide call: name the custodian, state what that choice concedes about who can read history, and check the arrangement survives longer than the retention window it protects.

## Why custody is the real access decision On a cluster where the broker reads plaintext, the question "who can read this stream's history?" is answered by the grant table, and it can be answered again tomorrow by editing it. Once the writers encrypt payloads, that stops being true. The grant decides who can obtain the bytes; the encryption key decides who can understand them. Two consequences follow that most teams do not plan for: - **Reading history is retroactive.** Whoever holds the key today reads records written a year ago, by people they never met, for purposes nobody remembers. Handing someone the key is not a forward-looking grant; it is access to everything in the window. - **Denial is retroactive too.** Withdrawing the key removes access to that same history even from a principal whose broker-side grant is untouched. The two control surfaces can disagree, and the key wins. ## Custody is also a durability decision This is the half that surprises people. If the key the records were written under is lost — a team dissolved, a store rebuilt, a custodian gone — the records remain exactly where they were. Every copy is intact, retention has not expired, the cluster serves them happily, and they are meaningless. There is no operational recovery from the broker side, because the broker never had anything to recover with. So the design questions for a stream carrying opaque payloads are the same ones you would ask about a backup: 1. Who can produce the key today, and who can produce it if that person and their team are gone? 2. Is the custody arrangement itself recoverable, or does it have a single point of failure that nobody has exercised? 3. Does the retention window on this stream exceed the lifetime of the arrangement that protects it? If history is kept for a year and keys are managed by a team that reorganises every six months, the answer is already no. ## Moving to a new encryption key Stored records are already written, and a broker's history is append-only: there is no operation that re-encrypts what is there. That produces a specific, permanent shape: - New writes use the new encryption key; **everything already retained stays bound to the key it was written under**. - A reader that must cover the whole window therefore has to hold more than one key, and must be able to tell which one applies. That requires the record to carry, outside the ciphertext, an indicator of the key it was written under. - "Re-encrypting the history" is possible only by reading it out and writing it somewhere else as new records, which produces new records with new positions — a migration, not an in-place change, and it is rarely worth it. The practical rule: the number of keys a consumer must be prepared for is set by how often you change keys and by how long the stream keeps history. ## Who should hold it | Custodian | What it gets you | What it costs | | --- | --- | --- | | The producing team | Access follows the data's owner; narrowest reader set | Central teams cannot help debug; history is orphaned if the team dissolves | | A central platform team | Durable, staffed, recoverable custody | The platform team can read everything, which was often the point of encrypting | | A rented tier's operator | No machinery for you to build or run | Their staff are inside the boundary; you have re-created the problem | | A consumer-side group per stream | Readers hold exactly what they read | Duplication, and no single answer to "who can read this history?" | There is no free option, and saying so is the answer to this question. The point of writer-side encryption was to exclude the operator; choosing an operator-adjacent custodian for convenience quietly undoes it, and a lead is expected to notice that trade rather than discover it in an audit. ## What varies Platforms differ in ways that change the sums. Some keep history for days, so custody only has to outlive a week and the problem is small; others keep months or years, and on those the custody arrangement must outlive people. Rented clusters differ in whether you can supply the key that protects their storage at all — where you can, it changes who can read the volumes but not who the broker serves plaintext to, so it is not a substitute for writer-side encryption. And platforms differ in what a record may carry outside the ciphertext, which decides how gracefully you can run several keys at once. ## Answering it well Name three things: custody decides retroactive read access; it is a durability dependency, because the bytes survive and the meaning does not; and old history cannot be re-encrypted in place, so every key decision is bounded below by the retention window. Then state who you would make the custodian and what you accept by doing so.

  • Why can retained history not simply be re-encrypted under the new key?
    A broker's history is append-only; there is no operation that rewrites records in place, and the cluster holds ciphertext it cannot open in any case. The only route is reading the records out and writing them as new records elsewhere, which creates new positions and a migration for every reader. In practice you keep the old key instead.
  • What has to be true for a consumer to read across a key change?
    It must hold every key covering the window it reads, and each record must carry outside the ciphertext an indicator of which one applies. Without that indicator, a consumer holding three keys cannot tell which opens an old record, and the safe fallback of trying each is both slow and a sign the design is missing a field.
  • A team that owned a stream's encryption keys is dissolved. What is the exposure?
    Two, in opposite directions. If custody passes informally to whoever is nearby, the reader set has silently widened over history nobody reviewed. If it passes to nobody, a year of retained records becomes permanently unreadable while still occupying storage and still counting against capacity. Both are found late, usually during an incident.
  • Does the broker-side grant table still matter once payloads are opaque?
    Yes, and dropping it is a common overreach. Grants still decide who can connect, write, and consume the bytes at all, which governs load, cost and the ability to disrupt a stream. What they no longer decide, alone, is who can understand the records.

saying these in an interview costs you the question

  • Treats the grant table as still deciding who reads opaque history
  • Assumes a broker can re-encrypt retained records under a new key
  • Ignores that losing the encryption key permanently loses the history
  • Hands custody to the platform team without noticing the boundary moved
  • Plans a key change without an indicator of which key a record used
  • Assumes custody arrangements naturally outlive a long retention window