If an API gateway has a 1-second end-to-end budget for a request that fans out to three parallel downstream calls plus one sequential call after they return, how should it split that 1-second budget across the calls, and what's the risk of splitting it evenly without regard to the call graph shape?
answer
- parallel = each gets ~full remaining budget
- sequential = slices sum, recompute after each step
- even split ignores call-graph shape
- elapsed-time subtraction, not fixed slices
- mandatory final step can get starved
basics
~20 sCalls that run in parallel can each get most of the budget since they finish together, but calls that run one after another must each get a smaller slice so their total doesn't exceed the time left. Splitting evenly regardless of shape wastes budget or starves later steps.
solid answer
~50 sFor parallel (fan-out) calls, each can be given close to the full remaining budget minus a safety margin, since they run concurrently and the gateway only waits for the slowest one; giving each only a third of the time would unnecessarily fail calls that would've succeeded within the real budget. For sequential calls, the budget must be divided so the sum of each step's slice (plus overhead) fits inside what's left, typically by subtracting elapsed time after each step and giving the next step whatever remains, rather than pre-allocating fixed equal slices that ignore how much time earlier steps actually used. The risk of naive even-splitting is either over-conservative timeouts on parallel calls that fail otherwise-successful fast paths, or sequential steps running out of a fixed slice early while there's still real time left in the overall budget, or the reverse: consuming the whole budget in early sequential steps and leaving nothing for a mandatory final step.
go deeper
Should recognize that a shared time budget across multiple calls needs to be divided somehow, even without a precise strategy for parallel vs sequential.
Should articulate that parallel calls can each get nearly the full remaining budget while sequential calls need their slices summed and recomputed after each step.
Should design the recompute-after-each-step pattern for mixed fan-out/sequential graphs and identify the starvation risk to mandatory late steps.
Should set organizational guidance and tooling (e.g., a shared budget-splitting helper in a common RPC client library) so every team building aggregator or gateway services applies the same correct policy instead of reinventing ad hoc, often-wrong even-splitting logic per service.
## The arithmetic underneath deadline propagation Budget splitting is the practical arithmetic problem that sits underneath deadline propagation: once a service has an overall time budget for a request (say, 1 second, computed as time remaining until the original client-facing deadline), and that request requires calling multiple downstream dependencies, someone has to decide how much of that shared budget each individual call gets. The correct split depends entirely on the shape of the call graph, specifically, whether calls happen in parallel (fan-out) or in sequence, and getting this wrong either wastes budget that was actually available or starves a call that needed more of it. ## Parallel calls race the same clock together For parallel calls, three downstream services called concurrently, where the gateway waits for all three (or the slowest) before proceeding, the naive instinct to divide the budget evenly (1 second divided by 3 calls = 333ms each) is wrong and actively harmful. Since the calls run concurrently, the gateway's true constraint is 'the slowest of the three must finish within the remaining budget,' not 'each one gets an equal fixed slice.' Giving each call only 333ms means any one of the three legitimately taking, say, 600ms, well within the true 1-second budget, gets prematurely killed, converting a request that would have succeeded into a failure. The correct approach is to give each parallel call close to the full remaining budget (minus a small safety margin to allow time for the gateway to process the last response and produce its own output), because they're not competing with each other for the clock, they're racing the same clock together. ## Sequential calls draw down one shared allowance For sequential calls, a chain of steps where step 2 can't start until step 1 finishes, the arithmetic is the opposite problem: their durations sum, so pre-allocating fixed equal slices ignores reality in both directions. - If step 1 finishes fast (say 100ms out of a naive 250ms slice for a 4-step, 1-second budget), that leftover 150ms is wasted if step 2 is still capped at its own fixed 250ms rather than being allowed to use the time step 1 didn't need. - Conversely, if step 1 runs slow and consumes its whole slice plus some, a fixed-slice scheme either lets it overrun (breaking the total budget) or kills it right at 250ms even though the caller might have tolerated a bit more if a later step turns out to be trivial. The robust pattern is dynamic remaining-budget computation: after each step, subtract actual elapsed time from the deadline and pass whatever's left to the next step, exactly the deadline-propagation mechanism, applied recursively within a single service's internal sequence of calls, not just across service boundaries. ## Mixed graphs apply both rules in order Mixed graphs, like the gateway's fan-out-then-sequential-step example, need both rules applied in the right order: the parallel block gets a shared timeout equal to most of the remaining budget (since it's bounded by the slowest of the three, not their sum), and after it returns, whatever time is actually left (deadline minus elapsed, which reflects how long the fan-out truly took) becomes the sequential step's budget, not a pre-computed fixed number decided before the fan-out even ran. This means the sequential step effectively gets less time when the fan-out ran slow and more when it ran fast, which is the behavior you want: the total across the whole graph still respects the original 1-second ceiling regardless of how the internal steps happened to split their time. ## What naive even-splitting costs you The real risk of naive even-splitting, ignoring call-graph shape, is twofold. 1. **First, false failures.** Over-conservative fixed slices on parallel calls reject responses that would have arrived comfortably within the true remaining budget, directly hurting availability for no real gain. 2. **Second, budget starvation of mandatory final steps.** If early sequential steps are allowed to consume their full pre-allocated slice even when the overall deadline is tightening (because, say, an earlier fan-out ran unusually slow), a late, possibly critical step, like writing an audit log or committing a transaction, can be left with too little time or none at all, and depending on how that failure is handled, either the whole request fails despite doing most of the useful work, or the final step is skipped or short-circuited silently, producing a subtly incomplete result. This is a genuine production hazard in gateway and aggregator services, the kind of service that calls a product catalog, a pricing service, and an inventory service in parallel and then writes a combined response, and it's why frameworks like resilience4j's `TimeLimiter` and hand-rolled budget-splitting logic in API gateways explicitly compute remaining-time-after-elapsed rather than dividing the total budget by call count up front.
- Why is it wrong to give each of three parallel downstream calls exactly one-third of the remaining time budget?Because parallel calls race the same clock concurrently rather than consuming the budget one after another, the true constraint is that the slowest of the three must finish within the full remaining time, not within a third of it. Splitting evenly needlessly kills calls that were on track to finish well within the real budget, converting successes into failures for no benefit.
- In a sequential chain of calls, why is recomputing the remaining budget after each step better than pre-allocating a fixed timeout to each step in advance?Fixed pre-allocated slices don't reflect what actually happened during earlier steps: a fast earlier step leaves unused time that a fixed-slice scheme wastes, while a slow earlier step can blow past its slice and either break the total budget or need to be killed even though a later step might have needed less time anyway. Recomputing deadline minus elapsed after each step lets every subsequent step use exactly whatever time is truly left.
- What production risk arises if a mandatory final step in a request chain, like committing a transaction or writing an audit log, only gets whatever time is left after earlier fan-out or sequential steps, with no reserved minimum?If earlier steps run slower than typical, the final step can be starved of enough time to complete, causing the whole request to fail late, after most of the expensive work is already done, or, worse, being silently skipped or half-executed if it's not handled carefully. A common mitigation is reserving a minimum time slice for known-critical final steps rather than letting them get whatever scraps remain.
It's like packing for a trip with a fixed suitcase-weight limit: items you're wearing at the same time (parallel) don't each need their own separate limit, but items packed one after another into the same bag (sequential) all draw down the same shared weight allowance, so you have to track what's left after each item, not split the limit evenly up front.
saying these in an interview costs you the question
- Splits a shared time budget evenly across calls without considering whether they run in parallel or sequence
- Doesn't realize parallel calls can each get close to the full remaining budget since they run concurrently
- Uses fixed pre-computed slices for sequential steps instead of recomputing remaining time after each step
- Has no answer for what happens to a critical final step when earlier steps consume most of the budget