Partition & Queue Count Planning

Parallel reading is not one mechanism. Brokers in this class come in two shapes, and the shape decides whether there is a number to pick at all - which is exactly why this gets asked before a stream exists rather than after. ## Shape one: a stream split into parts Platforms that split a stream divide it into a fixed number of **partitions** when the stream is created. Each partition is an independent part of the stream with its own stored records, and the common cooperating-reader arrangement hands **each partition to one reader at a time**. Two consequences follow at once: - Readers up to the partition count each receive a share of the partitions and do work. - Readers beyond the partition count receive no partitions. They are warm standbys - genuinely useful when one of the others dies, worth nothing for throughput. So the count chosen up front is a **parallelism ceiling**: a cap on how many reader processes can be making progress at once on that stream, fixed before a single record was written. Adding machines to the reading side stops helping precisely there, and no amount of reader-side tuning moves it. One honest qualification: some split-stream platforms also offer a mode in which several readers share the records of a single part. That mode buys parallelism back and gives up the one-reader-per-part assignment, so it is a different trade rather than an exception to the arithmetic - and where it exists you should say which mode you mean. ## Shape two: one shared queue A queue-shaped broker has no structural count to choose. There is one work list, and every record goes to whichever connected reader asks for work next. Start five readers and five get work; start fifty and fifty do. Nothing inside the queue caps that. That does not make the parallelism unbounded. It means the bound is **not in the broker**: it sits in whatever each reader touches while handling a record - a database and its row contention, a connection pool, an external service's allowed call rate, a file system. Past that point, extra readers convert waiting in the queue into waiting at the shared resource, usually with contention costs on top. ## The two shapes side by side | | split stream | shared queue | |---|---|---| | unit handed out | a partition, held by one reader at a time | a single record, to whichever reader asks next | | number chosen up front | yes - the partition count | none | | readers beyond the ceiling | receive nothing; standbys only | still receive work until something downstream saturates | | where the ceiling lives | inside the stream's own shape | outside the broker, in the shared resources readers touch | | moving the ceiling | a structural change to the stream | start or stop reader processes | ## Why this is a first-screen question It separates candidates who have only added readers from candidates who know why adding readers sometimes does nothing. The failure it predicts is concrete and common: a team scales the reading side of a split stream from four processes to twelve, watches throughput stay flat, and starts profiling application code - when the stream was created with four parts and eight processes are holding nothing. On a queue the same team scales to twelve, throughput does rise, and then flattens for an entirely different reason that lives in the database behind the readers. The practical habits that follow: 1. Before scaling readers on a split stream, check the partition count first. It is the cheapest possible diagnosis. 2. Before scaling readers on a shared queue, find the shared resource each record touches and measure how much of it is left. 3. When writing down capacity for a new stream, record which shape you are on - the number means something different in each. ## What this question is not about It is not about ordering, keys or how a record is routed to a particular part; that is its own subject and it is not needed to answer this. It is also not about how you would execute a change to the count on a cluster that is already running. Here the question is only: for each shape, what sets the number of readers that can usefully work at once, and is that number something you chose?