skip to content

A byte ceiling of five hundred gigabytes is configured on a stream with twenty-four partitions — what does it actually bound?

level: middleimportance: should knowfreq 58%

answer

  1. bytes of what, exactly?
  2. the scope multiplies the number
  3. partition count is the multiplier
  4. skew makes the front edge ragged

basics

~20 s

It depends on the scope the platform counts it against. Where a size bound is per partition of a stream — the common case on platforms that split one — twenty-four partitions mean up to twelve terabytes, not five hundred gigabytes.

solid answer

~40 s

A byte ceiling is always stated against a scope, and the figure means nothing until you name it. On platforms that split a stream into partitions, the size bound is commonly enforced **per partition of a stream**, so twenty-four partitions multiply it: five hundred gigabytes becomes twelve terabytes of stored history. Other designs count it per stream as a whole, and queue-shaped ones count it against a **per-subscriber backlog**, where the bytes grow with the number of subscribers rather than with the partition count. Some platforms have no size bound at all and express retention only as an age. The operational consequence is that the same configured number produces different capacity on different platforms, and that changing the partition count later silently rescales the storage this one setting implies.

go deeper

for a junior

Remember to ask 'per what?' whenever someone quotes a size bound. The number alone does not tell you how much history a stream will hold.

for a middle

Explain the multiplication: a per-partition scope means the stream may hold the ceiling times the partition count, and uneven routing means the bound bites the busiest parts first.

for a senior

Point out the change that rescales storage without anyone editing retention — raising the partition count — and describe how you would catch it in capacity review.

for a principal

Argue for expressing retention policy as a span in days at a stated peak rate, with the per-scope byte figure derived from it, so the estate's capacity plan survives a partition-count change.

## The number is per-scope, always A byte ceiling (a size bound) says: *this scope may hold at most this many bytes, and beyond that the oldest records become removable.* The word doing all the work is **scope**. An operator who can quote the ceiling but not the scope cannot answer the only question anyone actually asks — how much capacity does this stream need? The scopes in circulation: | scope | what the figure bounds | what multiplies it | |---|---|---| | per partition of a stream | the bytes held by one partition | the partition count | | per stream | the bytes held by the whole stream | nothing | | per subscriber backlog | the bytes one subscriber has not taken yet | the number of subscribers | | none offered | — | retention is expressed only as an age | ## The worked case Five hundred gigabytes, twenty-four partitions, per-partition scope: - worst case stored history = 24 × 500 GB = **12 TB**; - and that is the bound on *this* stream's history alone, before anything else on the cluster is counted. The figure is a worst case, not a prediction: a partition only reaches its ceiling if enough records are routed to it. Which brings the second trap — **partitions are rarely filled evenly**. If routing concentrates most records into a few partitions, those few reach the ceiling and start removing history while the quiet ones sit far below it. The result is a stream whose readable span differs between its own partitions: one part of the history reaches back days, another only hours. A reader that reads all of them sees a ragged front edge. ## Why the scope bites twice The first bite is at sizing time — someone reads the ceiling as a stream-wide number, plans capacity for five hundred gigabytes and provisions a twenty-fourth of what the configuration permits. The second bite is later, and it is worse because nobody edits a retention setting to cause it: the stream's partition count is raised for throughput reasons, and the same untouched byte ceiling now permits proportionally more stored history. One change made for a completely different reason rescales the storage this stream may occupy. The corollary is worth stating in an interview: **on a per-partition scope, a byte ceiling is not a capacity figure — the product of it and the partition count is.** On a per-subscriber scope the same structure appears with a different multiplier: adding a subscriber that reads slowly adds a whole ceiling's worth of bytes that cannot be removed while it still owes itself those records. ## What this scope question is not A few neighbouring questions sound similar and are answered elsewhere: - How many copies of each record the cluster keeps is a durability decision; this bound counts one logical history, and whatever the copy set does to the final figure is that other question's ground. - What happens when the data volume genuinely runs out is a different failure entirely — a byte ceiling exists precisely so that it does not. - Moving closed segments off the local volume so the device stops being the bound is another subject again. Keep the answer on what the configured number counts. ## Stating it well A strong answer sounds like: *"Five hundred gigabytes of what? If it is per partition of a stream, twenty-four partitions means the stream may hold twelve terabytes, and if routing is skewed the ceiling binds on the hot partitions first, so the retention window — the span of history still readable, which is the replay budget anyone recovering will spend — is not even the same across the stream. I would restate the policy as a span in days and derive the per-scope ceiling from the peak write rate, so the number in the configuration is a guard rather than the policy."* ## What varies between platforms - Split-stream designs almost always enforce a size bound at the level of the part, because that is the level at which segments roll closed. - Designs built around a per-subscriber backlog count bytes per subscriber, and their equivalent question is "how many subscribers", not "how many partitions". - Some platforms offer only an age bound, and there is no size figure to misread — the capacity risk moves entirely to your monitoring of bytes held. - Where a hosted offering states one storage number for a whole stream, check whether it is describing a bound it enforces or capacity it provisions; they behave differently under a surge.

  • The partition count on this stream is doubled for throughput reasons. What happens to retention?
    On a per-partition scope, nothing was edited but the permitted stored history doubles, because the same ceiling now applies to twice as many parts. The age bound is unchanged, so the practical effect is that the size bound is far less likely to bind — capacity planning has to be redone even though no retention setting was touched.
  • Why can two partitions of the same stream hold different spans of history?
    Because the size bound is enforced per partition and records are rarely routed evenly. A partition receiving most of the traffic reaches its byte ceiling and removes its oldest records while a quiet partition is nowhere near its own, so the earliest position still on the store differs between them.

saying these in an interview costs you the question

  • Assumes a byte ceiling counts the whole stream without checking the scope
  • Quotes the ceiling as a capacity figure with no partition count attached
  • Thinks every partition of a stream holds the same span of history
  • Believes changing the partition count leaves storage unaffected
  • Treats the ceiling as a prediction rather than a worst case