skip to content

One tenant's policy model costs ten times more accelerator time per request; why does an equal-requests-per-tenant fair-share rule still starve the others?

level: seniorimportance: nice to knowfreq 30%

answer

  1. fairness needs a unit
  2. requests are not capacity
  3. charge accelerator time
  4. estimate first, reconcile after
  5. long requests block short ones

basics

~20 s

Because the scarce resource is accelerator time, not requests. Counting requests charges a 400 ms inference the same as a 20 ms one, so the heavy tenant can hold most of the tier while staying comfortably inside its equal share.

solid answer

~50 s

A fair-share rule is only as good as its accounting unit. Take two tenants: one sends 200 requests per second at 20 ms each and holds about 4 requests in flight; the other sends 20 per second at 400 ms each and holds about 8. The second tenant occupies twice the tier on a tenth of the request rate, and a per-request share sees it as the light user. Denominate the share in the scarce resource instead - accelerator-milliseconds - estimated up front from the model and the input, then reconciled with the measured cost after execution so persistent underestimates get charged back. Deficit round robin does this naturally with deficits in milliseconds rather than request counts. Separate queues per cost class stop one long request head-of-line blocking short ones, and a reserved floor decides who degrades first.

go deeper

for a junior

Recall that tenants can run very different models, so two requests on the same tier are not necessarily the same amount of work.

for a middle

Explain how arrival rate times service time turns traffic into occupancy, and why counting requests misprices a tenant whose model is ten times heavier.

for a senior

Show the working scheduler: cost estimated at admission, charged back after execution, queues split by cost class, and an explicit rule for who is shed first.

for a principal

Own the choice between scheduling a tenant among neighbours and giving it dedicated serving capacity, and state what utilisation and unit cost the organisation accepts for that isolation.

## Pick the accounting unit before the scheduling algorithm Weighted fair queuing and deficit round robin both share *something* evenly between tenants. Which something is a design decision, and it is the decision that determines whether the rule works: | unit | binds on | fails when | |---|---|---| | admitted requests | arrival fairness | per-request cost varies between tenants | | accelerator-milliseconds | actual contention for the device | the cost must be estimated before it is known | | input size (bytes, pixels, tokens) | a cheap proxy for cost | the constant relating size to cost is model-specific | On a shared ad-screening tier, tenants pin different policy models: a small text classifier and a heavy multimodal creative scanner sit side by side. Their per-request costs differ by an order of magnitude, so requests are the one unit guaranteed to misprice them. ## The arithmetic Little's Law - concurrency equals arrival rate times service time - converts each tenant's traffic into occupancy: - Tenant A: 200 requests/s x 20 ms = **4 requests in flight**. - Tenant B: 20 requests/s x 400 ms = **8 requests in flight**. Tenant B uses **twice** the capacity on **one tenth** of the request rate. An equal-request share not only fails to restrain B, it actively misdiagnoses the situation: any dashboard sorted by requests per second puts A at the top and B nowhere near it. ## Scheduling in a cost unit 1. **Estimate** a cost `c` for each admitted request from the pinned model version and the input size. The estimate does not have to be exact; it has to be unbiased across tenants. 2. **Schedule** on deficits measured in accelerator-milliseconds: each round, add a quantum to every backlogged tenant's deficit and serve its requests while the deficit covers `c`. 3. **Reconcile** after execution by charging the *measured* cost. The difference carries into the next round as a negative deficit, so a tenant whose requests are systematically underestimated pays it back rather than profiting from the error. Without step 3, the estimate becomes the contract, and any tenant whose workload drifts away from its estimate silently gets free capacity. ## Head-of-line blocking and the batcher Cost variance does not only break fairness; it breaks tail latency: - One 400 ms request ahead of a queue of 20 ms requests adds 400 ms to every one of them, no matter how fair the long-run share is. - A batcher that mixes cost classes makes every request in a batch wait for the slowest member. - The defences are **separate queues per cost class**, a **cap on batch service time**, and per-tenant concurrency limits low enough that one tenant cannot fill the batcher on its own. Fair-share and tail latency are two different goals: long-run shares can be perfectly fair while p99 is ruined. ## Decide who degrades first, do not discover it Under sustained overload something must give, and the platform should have chosen in advance: - every tenant has a **reserved floor** it is never shed below; - shedding takes first from the tenant **furthest above its share**, not uniformly across tenants; - latency-critical tenants can be given a **priority class** that is served ahead of batch-style screening work, with its own smaller ceiling; - the shed response is **retryable and explicit**, so a throttled tenant backs off rather than amplifying. ## The last resort: change the isolation boundary Some workloads cannot be scheduled beside others at all - a model whose resident footprint crowds out every neighbour, or a tenant whose traffic is batch-shaped and arrives in walls. For those, move the tenant onto its **own replica set of the serving tier**: the blast radius between it and everyone else goes to zero, utilisation falls because its idle capacity can no longer be lent out, and its cost per prediction rises. That is a deliberate trade, made per tenant, and it is worth making only after cost-based scheduling has been tried, because it is the expensive answer to a scheduling problem.

  • When do you stop scheduling a tenant beside others and give it its own replica set?
    When its cost profile cannot be scheduled fairly - a resident footprint that crowds out neighbours, or bursts that arrive as walls of work. A dedicated replica set of the serving tier removes the interference entirely, at the price of lower utilisation and a higher cost per prediction, so it is a per-tenant decision rather than a default.
  • Long-run shares are provably fair, yet the light tenant's p99 is terrible. Why?
    Fair-share rules equalise throughput over a window; they say nothing about what a request waits behind within that window. A single 400 ms execution ahead of short requests adds 400 ms to each of them. Bounding tail latency needs separate queues per cost class and a cap on how long any one execution or batch may run.
  • How do you estimate a request's cost before running it?
    From the pinned model version and the observable shape of the input - payload size, image count, sequence length - calibrated against recent measured executions of that version. Keep a per-version rolling average, for example an exponentially weighted moving average of measured cost, and treat the estimate as an admission input that is corrected afterwards, not a promise.

saying these in an interview costs you the question

  • Equal requests per tenant is by definition fair
  • Fair queuing works regardless of the unit being shared
  • A heavy model only hurts the tenant that chose it
  • Autoscaling removes the need to schedule between tenants
  • Estimate a request's cost once and never reconcile it