Should per-customer LLM spend caps live in Helicone's gateway or your own service?

## Two different questions wearing the same clothes "Stop this customer spending more than their plan allows" and "stop anything spending catastrophically" look like one requirement and are not. The first is a product entitlement: it has a price attached, a grace policy, an upgrade path, and someone in the company who owns the number. The second is an operational guard: it exists so a retry loop, a bad deploy or a scraper cannot produce a five-figure bill overnight. Conflating them is the usual mistake, and the tooling encourages it because Helicone's `Helicone-RateLimit-Policy` header can plausibly serve either. ## What the gateway gives you cheaply A policy such as `1000;w=3600;u=cents;s=user` is a single header. It requires no schema, no counter store, no distributed rate-limiter, and it meters in cents using Helicone's own cost calculation, which is the unit that actually matters for an LLM workload — request counts are a poor proxy when one call can be a thousand times more expensive than another. Segmenting by user (or by a custom property standing in for a tenant or a feature) needs only that the identifying header already be on the request. For a small team that has not built quota infrastructure, this is genuinely the fastest path to a ceiling, and having a ceiling beats not having one. ## Where it stops being sufficient **Coverage.** The quota governs requests that go through the proxy. Any code path that calls the provider directly — a nightly batch, an evaluation harness, a second service, a data-science notebook with its own key — is outside it. That makes the gateway number an underestimate of organizational spend and, worse, an underestimate that looks authoritative on a dashboard. **Timing.** A cent-denominated quota is evaluated from a call's usage, which exists only after the model has responded. Concurrent requests can all be admitted before any of them is counted, and one very large-context call can cross the line by itself. It constrains a sustained rate; it does not pre-authorize an individual spend. **Expressiveness.** The only outcome is a rejection. Real entitlements need more vocabulary: a soft threshold that warns at eighty percent, a grace window so a paying customer is not cut off mid-workflow, overage that bills rather than blocks, different budgets for different features under one account, an enterprise exception approved by a human. None of that fits in the header, and simulating it by rewriting the policy per request means your application already knows the answer — at which point it may as well enforce it. **Auditability.** Finance and support will eventually ask why a customer was throttled at a particular moment. A decision made in your own service can be logged with the plan, the counter and the input; a 429 from a third-party gateway is much harder to reconstruct after the fact. **Availability coupling.** Enforcement inside the request path means the enforcement's failure modes are your request path's failure modes. Decide explicitly whether you want the gateway to fail open or fail closed for your traffic, and note that a quota is the one Helicone feature whose misconfiguration rejects your own customers rather than merely losing telemetry. ## The arrangement I would defend Own the entitlement. The service that knows the plan checks the budget before the call, and returns a product-shaped answer — proceed, warn, degrade to a cheaper model, or refuse with an upgrade path. That check is testable, auditable and versioned with your pricing. Keep the gateway policy as the outer guard, set well above any legitimate usage, segmented by user or tenant so one bad actor cannot exhaust a global bucket. Its firing is a page, not a routine outcome. This is defence in depth: the guard is there for the case where the entitlement logic itself is what broke. And measure spend outside both. Neither mechanism is a budgeting system; a provider-side cost view or your own aggregation across all credentials is what tells you the true number, and it should be the thing finance looks at. ## The organizational angle One more consideration for a lead: the gateway policy is configuration, so it changes without a deploy, which is a benefit when you need to raise a limit for a large customer at short notice and a hazard when nobody owns the value. Put the policy in the same configuration system as your other limits, review it like code, and write down which team owns raising it. A quota with no owner is either never raised when it should be or quietly raised until it means nothing.