skip to content

Your platform accepts hand-written stream sources from other teams — what must an automated conformance check assert about each?

level: seniorimportance: nice to knowfreq 30%

answer

  1. a consumer written to be suspicious
  2. record order, count terminals
  3. latch a flag at the ending
  4. entry counter catches overlap
  5. races are sampled, never proven

basics

~20 s

Subscribe with a recording consumer and assert the grammar: the handle arrives before any value, at most one terminal signal, nothing after it, a failure carries a reason, and no two deliveries overlap. The last one can only be sampled under stress.

solid answer

~50 s

Attach a recording consumer that timestamps every signal, latches a `terminated` flag, and wraps each callback in an entry counter. From one run it can assert **with certainty**: the handle preceded the first value, terminals number at most one, nothing followed the terminal, and a failure carried a reason rather than being an empty completion. The **overlap** rule is different in kind — a race either happens on that schedule or does not, so the counter proves a violation but never proves conformance, and the check has to run repeatedly, with several producing workers and injected delays, to sample schedules. Termination is worse: a source that will never end is indistinguishable from one that stalled, so the harness reports a stall against a time budget rather than claiming a violation. Run each source both bare and behind a stage, since some adapters only misbehave when something sits in front of them.

code

pseudocode · 18 lines
pseudocode
state: terminated = false, inside = 0, seenHandle = false

function attached(h):  seenHandle = true

function value(v):
    enter(); assert seenHandle; assert not terminated; exit()

function completed():
    enter(); assert not terminated; terminated = true; exit()

function failure(reason):
    enter(); assert not terminated; assert reason is present
    terminated = true; exit()

function enter():
    if increment(inside) != 1: fail("overlapping delivery")
function exit():
    decrement(inside)

go deeper

for a junior

Know that the contract is checkable at all: a consumer can record what arrives and compare it against the grammar the source promised.

for a middle

Explain which rules one run settles — ordering, counts, nothing after the ending — and which need a latched flag rather than an after-the-fact scan.

for a senior

Separate proof from sampling out loud: the entry counter proves a race when it trips and proves nothing when it does not, so stress and repetition are part of the method.

for a principal

Decide what the gate actually blocks, what it merely reports, and what the submitting team must declare, so the check stays trustworthy instead of being routinely overridden.

## What a conformance harness is A conformance harness is a consumer written to be suspicious. It attaches to the source under test, records every signal it receives with a timestamp and the identity of the delivering worker, and asserts the contract as the signals arrive rather than afterwards, so a violation names the moment it happened. Nothing about it is specific to the source: any source that claims to obey the signal contract can be pointed at the same harness, which is the practical dividend of having a contract at all. ## Rules the harness can settle from a single run These are structural — they depend only on the order and count of signals, so one observation decides them. 1. **The handle precedes the first value.** Record arrival order; a value seen while no handle has been recorded is a violation. 2. **At most one terminal signal.** Count completions and failures together; two of anything is a violation. 3. **Nothing follows the terminal.** Latch a flag when the first terminal arrives; any later signal fails immediately. 4. **A failure carries a reason.** A failure signal with nothing in it tells the consumer only that something ended, which is what a completion already says. 5. **Repeat subscription behaves.** Attach twice and assert the grammar independently for each run, since terminals are counted per run. ## The rule the harness can only sample Overlapping delivery is a timing property, and this is where an honest engineer separates what a green test proves from what it suggests. - Wrap every callback in an **entry counter**: increment on entry, and fail if the value after the increment is anything but one. Decrement on exit. - A trip is **proof of a violation**. No trip across a run is **not proof of conformance** — that schedule simply did not overlap. - So the check has to *manufacture* schedules: many repetitions, a source configured with several producing workers, delays injected inside the consumer callback to widen the window, and a consumer slow enough to be caught inside. - Report the result honestly as *no overlap observed across N runs*, not as *serial delivery verified*. | rule | how it is observed | what a green result means | |---|---|---| | handle before first value | recorded arrival order | settled for the run | | at most one terminal | signal counters | settled for the run | | nothing after the terminal | latched flag, asserted per signal | settled for the run | | failure carries a reason | inspect the signal | settled for the run | | no overlapping delivery | entry counter under stress | sampled, never proven | | the source eventually ends | time budget | undecidable; reported as a stall | ## Termination is not a conformance question The contract caps terminal signals at one; it does not require one. So a source that runs for ten minutes without ending may be an endless source behaving perfectly, or an adapter that quietly stopped. The harness cannot tell them apart, and any check that reports the second as a contract violation will eventually reject a legitimate source. The workable arrangement is to have the submitting team **declare** whether the source is expected to end, and have the harness assert a stall only against that declaration and a stated time budget. ## Making the harness catch what bare testing misses Two refinements repay their cost: - **Run the source both bare and behind a stage.** Some hand-written adapters deliver concurrently and get away with it against a trivially thread-safe consumer, then fail the moment a stateful stage sits in front. Testing both shapes catches that class. - **Exercise the cancellation path as well as the clean path.** Sources whose emission is gated correctly on the happy path frequently have a second, ungated path for shutdown, and that path is where a signal after the terminal usually comes from. Note that a short tail of signals after a consumer asks to stop is a separate subject with its own rules; the harness should assert the grammar on whatever does arrive, not assume delivery halts instantly. ## What to do with a failing source Treat the classes differently, because their cost differs. A structural violation is deterministic, cheap to fix, and should block acceptance outright. An overlap trip is worse than it looks — it is a proven race that light testing will never reproduce again — and should also block. A stall against the declared budget is a report rather than a verdict, and goes back to the submitting team with the observation and the budget, because only they know whether the source was supposed to end.

  • Why can the harness never certify that a source delivers serially?
    Because overlap is a property of the schedules that actually occurred. The entry counter proves a violation when it trips, but a run with no trip only shows that those schedules did not overlap. Repetition, extra producing workers and injected delays raise the sampling rate; they do not turn it into proof.
  • How should the harness treat a source that has not ended after the time budget?
    As a stall report, not a violation. The contract permits a source never to end, so the harness needs the submitting team to declare whether this one is expected to, and asserts only against that declaration plus the stated budget.
  • Why test each source behind a stage as well as bare?
    Because an adapter that delivers concurrently can pass against a thread-safe recording consumer and still corrupt a stateful stage placed in front of it. Running both shapes catches sources that were built against one particular consumer rather than against the contract.

saying these in an interview costs you the question

  • Reports a green run as proof that delivery is serial
  • Treats a source that never ends as a contract violation
  • Checks only the happy path and skips the shutdown path
  • Asserts after the run instead of at each signal
  • Accepts a failure signal that carries no reason
  • Tests one subscription only, so per-run rules go unchecked