Why is team-draft interleaving unavailable as a cheap online read for a deterioration worklist shared by a ward team?
answer
- credit has to land on one item
- impressions must be independent
- the unit here is a ward-shift
- acting on a patient changes the patient
- overlap kills the discriminating rows
basics
~20 sTeam-draft interleaving needs many independent impressions and an action attributable to one ranker's item. A ward worklist is one shared list per team per shift, worked top to bottom, and each action changes the patient - so neither condition holds.
solid answer
~50 sTeam-draft interleaving compares two rankers by merging their lists - the two take turns picking their top remaining item - and crediting each subsequent action to whichever ranker contributed that item. It is cheap because it removes the variance between units, so a comparison lands on far less traffic than a split test. A deterioration worklist supplies none of what it needs: the independent unit is a ward-shift, not a row, and there are only a handful per day; clinicians work the whole list rather than choosing among items, so a single action carries no preference signal; and an escalation alters the patient, which destroys the counterfactual the other ranker's rows would have had. The framing consequence is that your online read has to attach to a coarser unit than a request, and you should say so before you promise a fast one.
go deeper
Know what interleaving is for - comparing two rankings inside the same shown list - and that it depends on many independent views with actions traceable to single items.
Explain the merge and the credit rule, and why the sensitivity comes from the rows the two rankers disagree on rather than from the shared ones.
Check the product shape before reaching for the technique: name the independent unit, whether an action is item-local, and what the exposure costs in a clinical path.
Decide what evidence the organisation will accept when the cheap online read is structurally unavailable, and price the slower alternative into the delivery plan up front.
## What team-draft interleaving is Given two candidate rankings of the same items, team-draft interleaving builds a single merged list: a coin flip decides who picks first in each round, then each ranker takes its highest-ranked remaining item. The merged list is shown once. Every subsequent action is credited to the ranker that contributed the acted-on item, and the ranker with more credited actions wins that impression. Aggregated over many impressions, this is a comparison within the same unit rather than between two groups of units, which is why it is famously cheap: the noise between units cancels. ## The three things it needs 1. **Many independent impressions.** The comparison is a tally over units. A handful of units per day cannot produce one. 2. **An action attributable to exactly one item.** Credit must be assignable, and an item's action must not be caused by the other ranker's items. 3. **Rankers that disagree.** Items both rankers would have picked carry credit that is essentially random with respect to quality; the discriminating signal comes from rows only one ranker chose. ## What a shift worklist supplies instead | interleaving assumes | a shared deterioration worklist gives | |---|---| | one impression per viewer, many per hour | one list per ward per shift, shared by a whole team | | an action selecting among presented items | a team working the list top to bottom, plus patients escalated from outside the list entirely | | an action that leaves the item unchanged | an escalation that alters the patient's trajectory, and the label with it | | a low-stakes preference signal | a care decision inside a safety-critical pathway | | rankers whose disagreement is safe to show | half the merged list produced by a ranker that has not been validated on a ward | ## The overlap problem Even if the unit problem were solved, sensitivity depends on disagreement. Two deterioration rankers trained on the same features usually agree heavily at the top - the sickest patients are obvious to both. On a twenty-row list with heavy overlap, only a few rows per shift are uniquely one ranker's, so each shift contributes a couple of discriminating observations at best. Combine that with a handful of shifts per day and the cheap read stops being cheap. ## What is left, and what framing owes the design - **The online read has to attach to a coarser unit** than a row or a request - a ward-shift, or a unit - because that is where independence actually lives. Saying which unit the read attaches to is a framing decision; the experiment design that follows it is a separate conversation with its own owners. - **A comparison that does not put an unvalidated ranker in front of clinicians** is worth more here than one that is merely fast. Scoring both systems on the same live population and comparing their worklists without changing what the team sees gives an early read with no clinical exposure. - **Cheap non-outcome reads exist and should be named in framing**: overlap between the two systems' top-twenty lists, the share of surfaced patients the team had already flagged, and the alert load per nurse-shift. None of them is the launch metric. All of them move in days rather than a quarter, and they tell you whether the candidate is doing anything different at all. ## Why this belongs in framing The attractive property of interleaving is that it makes an online read affordable, and a plan that quietly assumes one is available will promise a launch decision it cannot deliver. Deciding, in the first ten minutes, that this product shape has no per-request unit is what forces the honest sequence: state the slow online metric, state the unit it attaches to, and name the fast reads that are explicitly not it. A candidate who reaches for interleaving here without checking the unit of attribution is applying a ranking-product reflex to a shared-worklist product, and the interviewer is asking precisely to see whether that check happens.
- What unit does the online read attach to once no per-request read is available?A coarser one where independence genuinely holds - the ward-shift, or the ward. That choice belongs in framing, because it decides how fast the launch read can possibly accumulate and therefore how much weight the offline number will have to carry. How the comparison is then designed and analysed at that unit is a separate discipline with its own owners.
- Would a personal worklist per clinician make interleaving available?It restores independent impressions and per-viewer attribution, so the unit problem softens. The action problem stays: escalating a patient changes that patient's trajectory, and two clinicians can hold the same patient on different lists, so credit still leaks between rankers. You would need the acted-on outcome to be local to one list before the tally means anything.
saying these in an interview costs you the question
- Thinks interleaving needs only two rankings, not attributable per-item actions.
- Counts each worklist row as an independent unit of observation.
- Treats a clinician choosing whom to see as a preference click.
- Believes heavy overlap between two rankers makes interleaving more sensitive.
- Claims interleaving is unusable for any list ranked by risk.