skip to content

How do you give parallel cases their own callback destinations when the system's callback address is a single deployed setting?

level: seniorimportance: should knowfreq 42%

answer

  1. Routing beats filtering
  2. Give each case its own address
  3. Reads must not remove anything
  4. Late callbacks outlive their case
  5. Old registrations still point somewhere

basics

~20 s

Prefer routing over sharing: register a destination per case, usually one receiver with a per-case path segment, through the product's own configuration surface. Where the address truly is global, make every read filtered and non-destructive so no case drains another's evidence.

solid answer

~40 s

Take the highest level of isolation the product allows. Best: the destination is configurable per subscription or per tenant, so the case registers its own address - normally one shared receiver with a per-case path segment - and routing separates the traffic before any assertion runs. Next best: one receiver per parallel worker, addressed by that worker's index, which removes cross-talk between workers but not between cases that worker runs in sequence. Worst: a single deployed address every case shares, in which case the store must be read **non-destructively** and filtered by each case's own key, because a drain-the-store or pop-the-oldest read makes one case steal another's evidence and turns two good cases into one green and one intermittent failure. Scope retention per run too, so an earlier run's callbacks cannot satisfy today's predicate.

code

pseudocode · 11 lines
pseudocode
path = "/cb/" + run_id + "/" + case_id          # the case owns the address
receiver.claim_path(path)
sub = create_subscription(destination = base_address + path)

# filtered and non-destructive: nothing is removed from the store
mine = receiver.requests_at(path).where(body.reference == case_ref)
assert mine.count == 1

on_finish:
    delete_subscription(sub)                    # stop future deliveries
    receiver.release_path(path)                 # retention clears the requests

go deeper

for a junior

Know that a callback receiver shared by cases running at once holds requests that are not yours, and that reading in a way that removes entries takes evidence away from somebody else.

for a middle

Explain the routing options - a path segment per case, a receiver per parallel worker - and how each destination gets configured. Be ready to say why a shared store must be read in a filtered, non-destructive way.

for a senior

Show that cross-talk survives the end of a case: a late callback lands on whoever owns that address next. Cover registration cleanup and per-run retention as part of the isolation story, not as housekeeping.

for a principal

Own the position that a single global callback address is a testability defect worth raising with the product team, and decide deliberately when serialising a few cases is cheaper than changing the system.

When several cases run at once against one callback receiver, two different things go wrong and they need different fixes. The first is **mismatching**: a case reads a request that another case caused. The second, worse, is **stolen evidence**: a case removes a request another case had not read yet, so the victim reports that nothing was ever sent. Correlation on a case-owned key fixes the first. Only isolation of the destination, or a read discipline that removes nothing, fixes the second. ## Three levels of isolation | Level | How it works | What it isolates | What it needs from the product | | --- | --- | --- | --- | | Per-case destination | The case registers its own address, usually one receiver with a path segment unique to the case | Everything: cases in parallel, and cases in sequence on one worker | Destinations configurable per subscription or per tenant | | Per-worker destination | One address per parallel worker, chosen by the worker's index | Cross-talk between workers running at the same time | A destination that can be set per environment slice or per tenant | | Single shared destination | Every case's callbacks land in one store, separated only at read time | Nothing by itself; separation depends entirely on the read | Nothing - this is the fallback when the address is fixed | Take the highest row available. Routing is stronger than filtering because it happens before anything can be read wrongly, it needs no discipline from whoever writes the next case, and it makes the "who caused this" question a property of the address instead of an assertion each case has to get right. ## Why the destructive read is the dangerous one A receiver that hands out requests by removing them - drain, pop, take-and-clear - feels tidy and is a data race with a friendly interface. The sequence: 1. Case A triggers its action; the system will deliver its callback shortly. 2. Case B triggers its action; its callback is delivered first and sits in the store. 3. Case A reads, drains the store, and finds B's request - which does not match A's key, so A reports nothing arrived. 4. Case B reads a now-empty store and also reports nothing arrived. Two failures, neither reproducible alone, and both pointing at the product. The rule is simple: **reads never mutate the store**. Filter, take a copy, assert; let retention, not readers, remove anything. ## Cross-talk outlives the case A per-worker address contains the loud failure but leaves a quieter one. A callback can arrive after the case that caused it has finished - the system was slow, or the notification was retried on its own schedule - and the next case that worker runs now owns that address. It sees a request it did not cause. If its predicate is loose, it asserts on it; if its predicate is tight, it merely has confusing debris in a failure artefact. Either way, the cure is the same: make the address belong to the case, not to the worker, so a late arrival lands at a path nobody is reading any more. ## Registration hygiene Where destinations are registered per case, the registrations themselves become state that outlives the run: - A case that failed before retiring its subscription leaves the system attempting deliveries to an address nobody reads. - Enough dead destinations and the sender may treat that host as unhealthy, which affects the next case that registers there. - Subscriptions from a run last week can still deliver into a receiver that a new case has just claimed a path on. Give the run a naming convention it owns - a prefix identifying the run - and a sweep that retires any subscription carrying that prefix from an earlier run. Scope the receiver's retention to the run for the same reason: an old request that satisfies today's predicate is worse than no request at all, because it passes. ## When the address really is fixed Some systems read the callback destination from deployment configuration and offer no per-subscription override. Then: - Keep one shared receiver, read filtered and non-destructive, correlate on a case-owned key, and accept that a case can only prove presence cheaply and absence expensively. - Consider **serialising** the handful of cases that genuinely need to own the destination - for example those asserting that exactly one notification was produced - rather than making the whole suite serial. - Say plainly that this is a testability constraint of the product, not a limitation of the suite, and put a per-subscription destination on the list of things worth changing. It is usually a small change that removes a whole class of intermittent failure for everyone integrating with the product, not only for the tests.

  • Why is a per-worker receiver weaker isolation than a per-case destination?
    A worker runs many cases in sequence, and a callback can arrive after the case that caused it has finished, so the next case on that worker inherits it. Per-worker addressing removes cross-talk between workers running simultaneously, which is the loudest failure, but leaves the late-arrival-from-the-previous-case failure entirely in place. A per-case path closes both at once.
  • A case registered a destination and failed before retiring it. What does the next run inherit?
    A live subscription pointing at an address nobody reads, so the system keeps attempting deliveries and may come to treat that host as unhealthy - which then degrades the next case that registers there. Give the run a naming prefix it owns and a sweep that retires subscriptions carrying that prefix from earlier runs, so leaked registrations cannot accumulate.
  • When is it reasonable to serialise a case rather than isolate its destination?
    When the product offers no per-subscription destination and the case's claim genuinely requires exclusive ownership of the shared address - typically an assertion that exactly one notification was produced, which no filter can prove while others are writing. Serialise those few and let the rest run parallel with filtered reads; making the whole suite serial to fix a handful of cases trades minutes of wall clock for nothing.

saying these in an interview costs you the question

  • Sharing one destination and reading whichever callback arrived last
  • Draining the receiver's store during a read while other cases are running
  • Assuming callbacks stop arriving when the case that caused them ends
  • Leaving registrations behind that still point at a claimed address
  • Isolating by timing, by starting cases far enough apart