Quotas, Throttling & Fairness

A saturated node has four responses: queue the work, delay the answer, block the writer, or refuse the request. Each shows up differently at the call site, and the quietest discards records without failing the send.

A byte-rate ceiling caps how many bytes per second the cluster will handle for one named principal - a client identity, user, application or tenant - not per connection. Over it, that principal is either answered more slowly or refused.

A connection cap counts connections held open, not traffic. Holding one consumes memory and bookkeeping on the node whether or not data flows, so a fleet of near-idle clients can exhaust the cap while the byte rate stays tiny.

The node refuses that one write with a size error and keeps serving everything else on the same connection. Retrying the identical record never succeeds: it is a deterministic check, so the record must shrink or a ceiling must rise.

It buys fuller batches, so fewer requests and less fixed per-request cost at the node. It costs up to fifty milliseconds of added delay on the record that opens each batch — and at a low record rate it is nearly all cost, because the batch never fills anyway.

The cluster is accepting records more slowly than the application produces them, so unsent records have filled the writer send buffer. With the buffer full the client blocks the calling thread instead of failing, and broker saturation becomes an application outage.

A throttle delay still serves the request but holds the answer back, so the symptom is latency; a throttle refusal rejects the call and the client must repeat it, so the symptom is errors. Platforms in this class choose opposite defaults.

Break usage down per owner on the affected nodes, over the exact interval of the complaint, and on several axes at once — bytes, request rate, request-time share, connections. Cluster-wide totals cannot separate owners, and the tenant reporting the incident is rarely the cause.

The smallest ceiling on the path decides. A record must clear the writer's own limit, the accepting node's, the copy hop's where one exists, and the reader's; raising one and leaving the others alone moves the failure rather than removing it.

Shared nodes mean shared finite resources. A tenant does not get its own slice of the machine, it gets a turn, so one owner's burst consumes the network, disk, memory and request-handling threads that every other tenant on those nodes is waiting for.

The sending application's own processor time pays, per batch, on its own hosts. The cluster and the network collect the benefit as fewer bytes moved and, where the node keeps the batch as framed, fewer bytes held. Readers pay a smaller decompression cost.

Because the ceiling is judged against traffic accumulated over the interval, not against a single instant. A long interval forgives a short burst and then holds the client back afterwards; a short one clips the burst while it is happening.

Roughly four hundred connections, because a per-instance setting is multiplied by the instance count. The pool size is a per-process number; the cluster only ever sees the product, plus whatever a rolling deploy adds while old and new instances overlap.

Three different things in three different units: connections held open at once, accepted requests allowed to wait before being processed, and requests being worked on simultaneously. They sit in series, and none of the three is measured in bytes or in requests per second.

It may have stopped passing batches through untouched and started decoding and re-encoding them. Anything that forces a node to look inside a compressed batch — a conversion, a stamp, a per-record check — turns a free transfer into processor work that scales with records, not bytes.

Availability of the calling path has been bought with data, and with the evidence that the data is gone. Sends keep succeeding, error rates stay flat, and the loss is detectable only by reconciling records offered against records the cluster accepted.

Check whether the slowdown is confined to one principal. A node deliberately holding one identity's answers leaves every other client normal and the cluster's own saturation unremarkable, while a struggling cluster slows everybody at once.

Clients are opening a connection per unit of work instead of reusing a pooled, long-lived one. A count that rises and falls with traffic, with short connection lifetimes, is connection churn; a count that only ever rises is a leak.

While the hardware stays shared, every available lever is a fairness lever, not an isolation one. Bounding an owner limits how much of the pool it can take and moves most of the cost onto it, but the resources are still shared, so interference is bounded rather than eliminated.

A ceiling is a reservation, not a permission. The node must be able to hold a worst-case record for every request it is willing to handle at once, so memory headroom scales with the ceiling multiplied by requests in flight — paid by every client, for the benefit of the rare large record.

The record is accepted, stored and perfectly healthy from the cluster's point of view, but that one reader cannot take it. Depending on the design it stalls on that record or fails to complete it repeatedly, while every other reader is unaffected.

Treat it as a negotiation, not a setting. Measure records per request and compressibility per caller, price the change for the team that would make it, and be ready to buy capacity instead when the ask would break that writer's own latency commitments.

Overload should surface in the writers, as bounded waits and explicit failures, rather than in ever-deeper queues on the cluster. Require every writer to bound its unsent buffer, cap how long a send may wait, declare whether its records tolerate loss, and have a plan for records it cannot send.

Set defaults in more than one unit - bytes and requests per second, plus a request-handling share where it exists - and review consumption against each ceiling routinely, so no team first meets its ceiling mid-incident.