A paid vendor caps your account at 100 requests per second. How do you set your client's rate, burst and coalescing policy?

## Why this is a judgment call and not a constant A contractual limit looks like a number you can paste into a constructor. It is really an allowance that several parties are spending simultaneously, measured by someone else's clock, priced by someone else's contract, and owned by someone who may not be an engineer. The interesting decisions are about allocation, overflow behaviour and cost — and each of them can legitimately be overruled by the person who pays the invoice. ## Decide the target, not the ceiling Run below the contractual number. Two reasons, both practical. First, the two counters disagree: retries you consider one logical call are several to them, their window boundaries are not yours, in-flight calls are counted at different instants, and other consumers of the same account exist that you may not have inventoried. Second, the penalties are asymmetric. Leaving five percent of a paid quota unused costs five percent of a line item; tripping their limiter costs failed user-visible work, and with some vendors a rejected call still bills or still counts. How far below is a real decision — a comfortable margin for a hard contractual cap that triggers penalties, a thin one for a soft limit that merely rejects. ## Allocate the budget across the fleet and across workloads This is the part teams get wrong most often. A per-process limiter bounds one process. N replicas each holding the full allowance send N times the allowance, and the failure appears exactly when you scale up to handle load — the worst possible moment. The options each have a cost: - **Divide by replica count.** Simple and local, but wrong the moment the replica count changes, and wasteful when replicas are unevenly loaded: a quiet replica's share is unusable by a busy one. - **Funnel the traffic through one component** that owns the vendor relationship. Accurate and easy to reason about, at the price of a new dependency in the path and a new thing to make highly available. - **Coordinate through shared state.** Precise but adds latency and a failure mode to every call; you then have to decide what happens when the coordination store is unreachable — fail open and risk the contract, or fail closed and take an outage for a dependency the vendor never sees. Separately, allocate across *workloads*. Interactive requests and an overnight backfill drawing on one quota is a policy question: does the backfill get the leftovers, a fixed slice, or the whole thing outside business hours? Writing that split down, per caller, is the artefact that makes the decision reviewable later. ## Burst is a latency knob with a contractual edge A larger burst improves tail latency for spiky interactive traffic by letting queued work leave at once. It does not change the average. The constraint is the vendor's measurement window: if they enforce per-second and you burst a hundred in one instant, you are non-compliant on their meter even though your minute is well under. Absent documentation, a burst near their stated window's allowance is the defensible choice, and the smallest burst that meets your latency goal is the safest. ## Coalescing is the cost lever, and coupling is its price When the same answer is requested by many callers at once, collapsing those into one billable call is often the single largest saving available, and it costs nothing in correctness *if* the key is exact. The price is coupling: every waiter now shares one execution's latency and one execution's failure. That is a tradeoff the product owner should see stated — fewer vendor calls and lower spend, against a failure that lands on many callers at once instead of one. Whether the savings justify it is measurable: the share of calls that were coalesced tells you directly what you are buying. ## Choose the overflow behaviour deliberately Above the limit, work either waits or is rejected. Waiting preserves the work and spends latency and memory; rejecting preserves latency and loses work. Different callers deserve different answers, and "whatever the library did by default" is not one of them. For interactive traffic, a bounded wait then a clear failure is usually right; for a backfill, waiting indefinitely is fine because nobody is watching. ## Instrument so the policy can be argued about Three signals make the whole thing legible: allowed versus denied counts at your limiter, the vendor's own rejection rate, and the coalesced share of calls. The combinations are diagnostic. Denials at your limiter while the vendor never rejects means you are self-limiting below the capacity you already pay for — raise the target or ask why. Vendor rejections while your limiter is happy means your model of their accounting is wrong, most often because a caller you did not know about shares the account. Both are cheaper to discover on a graph than in an invoice. ## Name the owner The engineering choices — where the limiter lives, how the fleet divides the allowance, which calls coalesce — are yours. The target as a fraction of the paid ceiling, the split between paying and batch workloads, and whether the right move is to buy a larger tier rather than engineer around the limit belong to the person who owns the vendor budget. Present the arithmetic and the options; expect to be overruled on the number, and design so that being overruled is a configuration change rather than a redeployment.