skip to content

A moderation queue classifies each arriving item with an external call: what does concatenating those calls sequentially cost against merging them concurrently?

level: middleimportance: must knowfreq 72%

answer

  1. order or throughput, pick one
  2. one in flight versus several
  3. completion order, not element order
  4. throughput is bound over latency
  5. the bound is downstream load

basics

~20 s

Sequential concatenation keeps one call in flight: results arrive in queue order, throughput is capped at one call per call-latency. Bounded merging runs several at once, trading that ordering for throughput and putting the bound's worth of load on the dependency.

solid answer

~50 s

Sequential concatenation subscribes to the next item's classification call only after the previous one completed. One call is in flight, results reach the subscriber in queue order, and throughput is capped at roughly one element per round trip - so if the classifier takes 200 ms, the queue drains at about five items a second no matter how fast it fills. Bounded concurrent merging keeps up to N inner calls alive at once, which raises throughput to roughly N per round trip but delivers each result the moment it arrives, so a fast later item overtakes a slow earlier one and element order is gone. The bound is not a tuning detail: it is the peak concurrent load this step places on the dependency. The choice is therefore ordering against throughput, with the bound deciding how much of the second you buy.

code

pseudocode · 14 lines
pseudocode
// sequential: subscribe to the next inner source only after the previous finished
for each item in queue:
    result = subscribe_and_wait(classify(item))   // one in flight
    emit(result)                                  // emitted in element order

// bounded merge: keep up to LIMIT inner subscriptions alive at once
in_flight = 0
for each item in queue:
    wait until in_flight < LIMIT
    in_flight = in_flight + 1
    subscribe(classify(item), on_result = function(r) {
        emit(r)                                   // emitted in completion order
        in_flight = in_flight - 1
    })

go deeper

for a junior

Recall the trade in one line: doing the calls one after another keeps results in queue order, doing several at once finishes the queue faster but scrambles the order results come back in.

for a middle

Explain the mechanics behind the trade - one inner subscription versus up to the bound, completion order versus element order, and throughput of one call per round trip versus the bound per round trip.

for a senior

Demonstrate that you size the bound against the dependency rather than inheriting a default, and that you know the ordering defect stays invisible in tests where every stubbed call returns in microseconds.

for a principal

The trade-off worth owning is that ordering is a contract with the consumer, not a property of the pipeline: decide whether total order, per-key order or no order is the real requirement before the throughput conversation starts.

## The two contracts Every flattening strategy makes a promise about **how many inner calls may be alive at once** and therefore about **what order results come out in**. Sequential concatenation and bounded concurrent merging sit at the two ends of that span, and choosing between them is the most asked reactive design question because both answers are defensible and the wrong one is a production defect rather than a style complaint. **Sequential concatenation.** The step subscribes to one item's classification call, waits for it to complete, then subscribes to the next. Consequences: - Exactly **one** inner call is in flight, so the dependency sees a concurrency of one from this step. - Completion order is forced to equal element order, so results reach the subscriber in queue order for free. - Throughput is **one element per inner-call latency**. It does not improve because the queue fills faster; it improves only if the call gets faster. - Elements accumulate upstream while the step waits, which in a demand-driven pipeline shows up as demand not being requested rather than as elements being dropped. **Bounded concurrent merging.** The step subscribes to up to N inner calls at once and re-emits each result as it arrives. Consequences: - Up to **N** inner calls are in flight, so peak load on the dependency is N calls from this step - multiplied by however many instances of the pipeline are running. - Results are delivered in **completion order**. A 50 ms call started second overtakes a 2 s call started first. - Throughput rises to roughly **N per inner-call latency**, until the dependency slows down under the added concurrency, at which point raising N raises latency faster than it raises throughput. - Memory in flight grows with N, since N partly-processed elements and their pending results are alive at once. ## Side by side | property | sequential concatenation | bounded concurrent merge | |---|---|---| | inner calls in flight | one | up to the bound N | | result order | element order | completion order | | throughput ceiling | 1 / latency | about N / latency | | peak load on the dependency | one call | N calls | | worst case from a burst | queue lag grows | dependency sees N simultaneous calls | ## What people get wrong The first mistake is assuming concurrency makes an individual call faster. It does not - each call takes what it takes, and under contention it takes longer. Concurrency raises the *rate* at which the step finishes elements, by overlapping waiting time that was previously serialised. The second is picking merging by default and discovering the ordering requirement later, from a consumer that assumed queue order. That defect hides in tests, because stubbed calls return in a few microseconds with almost identical latency, and the interleaving that reveals it only happens when real latencies differ. The third is not knowing what the bound is. Flattening steps differ here: some impose a modest fixed default, others are effectively unbounded unless you say otherwise. Unbounded means one in-flight call per queued element, so a thousand-item burst becomes a thousand simultaneous calls - which is a memory problem locally and an availability problem for whoever is on the other end. State the bound explicitly rather than inheriting one. ## Between the two ends The choice is not binary. Two intermediate positions matter: 1. **Order-preserving concurrent flattening.** Subscribe eagerly up to the bound, but hold each completed result until its predecessors have been released, so delivery follows subscription order. Calls overlap exactly as in a plain merge; what you pay is memory for held results and head-of-line delay whenever the earliest call is the slowest. 2. **Partitioned ordering.** Concatenate sequentially within a key - all decisions about one item in order - and merge across keys. This is often what the requirement actually was, and it buys back most of the throughput. ## How to answer in an interview Name the two axes first - calls in flight, and order out - then say which the case in front of you cares about. For a moderation queue whose consumer writes independent verdicts, merging with a stated bound is right, and the bound comes from the classifier's capacity. For a queue feeding a consumer that replays decisions in order, sequential concatenation is right unless throughput will not meet arrival rate, in which case you move to order-preserving concurrency or partition by key. What an interviewer listens for is that you priced both sides rather than reciting a preference.

  • What is the concurrency bound when nobody states one?
    There is always a bound; it is just chosen for you, and implementations differ - some impose a modest fixed default, others are effectively unbounded. Unbounded means one in-flight call per queued element, so a burst upstream becomes a matching burst on the dependency. Read the flattening step's contract and state the bound yourself.
  • Can calls run concurrently and still deliver results in element order?
    Yes. An order-preserving concurrent flatten subscribes eagerly up to the bound but holds each completed inner result and releases them in subscription order. The calls overlap exactly as before; what you pay is memory for completed-but-held results and head-of-line delay whenever the earliest call is the slowest.
  • How much throughput does a given bound buy?
    Roughly the bound divided by the average inner-call latency, and only until the dependency itself slows under the added concurrency. Past that knee, raising the bound raises latency faster than throughput, so the useful bound is found by measuring the pair together rather than by arithmetic alone.

saying these in an interview costs you the question

  • Picks concurrent merging by default without asking whether order matters
  • Thinks concurrency makes each individual call return faster
  • Cannot say how many inner calls the merging step keeps in flight
  • Treats an unbounded merge as free because the pipeline looks idle
  • Believes merged results still arrive in the order elements arrived