A single record exceeds the maximum record size a broker node accepts — what happens to that write and to everything else?
answer
- one record, not the whole client
- a size check, not a capacity check
- deterministic, so the retry fails identically
- shrink the record or raise every hop
basics
~20 sThe node refuses that one write with a size error and keeps serving everything else on the same connection. Retrying the identical record never succeeds: it is a deterministic check, so the record must shrink or a ceiling must rise.
solid answer
~50 sA record-size ceiling is a per-hop maximum on one record's bytes, and it is checked deterministically — a record either fits or it does not. When it does not, the hop refuses that single write and returns a size error; the connection stays open and neighbouring records are unaffected. That makes it easy to separate from a rate ceiling (where the same record would succeed once the measured rate falls) and from a saturated node (where everything slows at once). Because the check depends only on size, backoff and retry are wasted: the second attempt carries exactly the same bytes. Some writer libraries apply their own ceiling locally and fail the call before anything reaches the network, which is why the error sometimes arrives with no round trip. The remedies are to send a smaller record, or to raise the ceiling at every hop the record must clear.
go deeper
Recall that a maximum record size exists and that exceeding it refuses that one write with a size error, immediately and repeatably, while other records keep flowing.
Explain why retry is useless here: the check compares bytes against a configured number, so nothing about load or timing can change the outcome. Know that the writer's library may enforce its own copy of the ceiling before the network.
Show the triage: read the reason from the error rather than the clock, look at the tail of the size distribution, and separate this refusal from a rate ceiling and from a node that cannot keep up.
Frame the standing question: what size of record the platform is willing to carry for everybody, and what the writing applications are expected to do with a record that will not fit.
## What a record-size ceiling is A **record-size ceiling** is the largest single record that one hop on a record's path will accept, counted in bytes of the record as it goes on the wire — the payload plus whatever key and metadata travel with it. Every platform in this class has such a ceiling somewhere; what differs is where it is set, exactly which bytes it counts, and how many hops enforce one. This question is about the hop that takes the write: the node the writer is connected to. The check is **deterministic**. The node compares one number against another number. Current load, the depth of the node's request queue, that client's recent traffic and the time of day play no part. That single property explains nearly everything about how the failure behaves. ## What the writer actually sees - The write for **that record** fails, with an error whose reason is size. - **Neighbouring records are unaffected** — a small record sent immediately afterwards on the same connection succeeds. - The **connection stays up**. Nothing is torn down and no other client on the node is disturbed. - The failure is **immediate**, not a timeout, and the record is not silently shortened to fit: it is stored whole or refused whole. - **Retrying changes nothing.** Backoff, jitter and a bigger retry budget all fail identically, because the retry carries the same bytes. Where the check runs varies by design. Many writer libraries carry their own ceiling and refuse the call locally, so the error appears with no network round trip at all; others learn about it only from the node's response. Both are the same class of failure with the same remedy, but the difference matters when you are reading a stack trace and asking whether the node was ever involved. ## Telling it apart from the other refusals An operator's real task here is mapping a symptom back to the ceiling that produced it. Several refusals look alike at the call site: | Symptom at the writer | Which ceiling is in play | Would the same call succeed later? | |---|---|---| | One record refused, its neighbours flow | the record-size ceiling | No — identical bytes fail identically | | Writes delayed or refused in bursts | a rate ceiling on that principal | Yes, once the measured rate falls | | Everything slows or times out together | the node cannot keep up | Yes, when the load drops | | New connections refused, existing ones fine | a ceiling on concurrent connections | Yes, when connections are released | The distinguishing evidence is **selectivity plus repeatability**: one record failing the same way every time while its neighbours succeed is a size ceiling and little else. ## Why it is discovered late, and usually under load The outlier payload is rare by definition — the order with a thousand lines, the document that happens to carry an embedded image, the end-of-month reconciliation record. A stream can run for a year inside a ceiling nobody chose deliberately, and then meet it on the busiest day, because the day that produces the most traffic is also the day most likely to produce the biggest single item. So the failure arrives with no recent change to blame, and the setting that caused it may predate everyone in the room. There is a second reason it surfaces late: the ceiling is enforced per hop, and a record that clears the writer and the node can still meet a smaller ceiling further along the path, so the first sign may not be a failed write at all. ## What to do about it 1. **Confirm the reason is size** by reading the error, rather than inferring it from timing. A refusal that coincides with a traffic peak is easy to misfile as overload. 2. **Measure the real record-size distribution**, and look at its tail. The decision is about the largest records you will genuinely produce, not the average one. 3. **Choose between shrinking the record and raising the ceiling.** Sending less in one record costs the writing application some work; raising the ceiling is paid for by everything else on the node. 4. **If you raise it, raise every hop** the record has to clear, since the smallest ceiling on the path is the one that binds. 5. **Decide what the writing application does with a record it cannot send.** Failing the caller loudly is almost always better than discarding it where nobody will notice. ## Where designs differ - Some platforms count only the payload against the ceiling; others include the key and the attached metadata, so the effective ceiling the application experiences is lower than the number configured. - On some designs, one oversize record inside a grouped request causes the **entire request** to be refused, so well-behaved records travelling with it fail too; on others only the offending record is rejected and the rest are stored. - Where the writer library enforces its own ceiling, the check can fire before any network call; where it does not, only the node knows.
- Using only what the writer sees, how would you separate a size refusal from a refusal caused by a rate ceiling?Repeat the write and vary nothing. A size refusal reproduces exactly, immediately, for that record alone, while smaller records on the same connection succeed. A rate-ceiling refusal is time-dependent: the identical call succeeds once the measured rate falls, it affects records of any size from that principal, and on some platforms it arrives as a slower answer rather than a refusal at all.
- The writer reports a size failure but the node's logs show no such request. Where did the check happen?In the writer itself. Many client libraries hold their own maximum record size and refuse the call locally, so nothing is put on the network and the node has nothing to log. The fix is the same in kind, but it lives in the writing application's configuration rather than on the cluster, which usually means a redeploy instead of an operator change.
saying these in an interview costs you the question
- Thinks retry with backoff will eventually get the same record through.
- Reads a size refusal as the node being overloaded.
- Believes the node shortens an oversize record so it fits.
- Assumes one oversize record drops the connection or stops the client.
- Raises the accepting node's ceiling only and calls the problem fixed.