Why is interleaving invalid for a sponsored-strip change that adds a fourth slot and widens the candidate pool?
answer
- one pool, one container, one merged list
- no faithful merge across two layouts
- unopposed listings win by existing
- inventory change and ordering change conflated
- page-level effects sit outside the list
basics
~20 sInterleaving compares two orderings of one shared pool inside one unchanged container. A fourth slot means no single merged strip represents both policies, and a widened pool lets the candidate win on listings the incumbent never had rather than on better ordering.
solid answer
~50 sTeam-draft interleaving rests on three preconditions: one merged list can legitimately stand for both policies, the layout and slot count are identical under both, and every listing could in principle have been drafted by either team. A fourth slot breaks the first two - you would have to render either three slots or four, so the merged strip is testing a layout change, and the extra slot's clicks have no counterpart in the incumbent world. A widened candidate pool breaks the third: the candidate's exclusive listings are unopposed, so a preference win can mean nothing more than new inventory. Both changes also move page-level outcomes such as abandonment and organic engagement, which a within-list comparison cannot see at all. This change belongs on a traffic split with a long-term holdout behind it, not on an interleaved strip.
go deeper
Remember the shape of the constraint: interleaving mixes two orderings of the same listings into one strip. If the two worlds do not agree on how many slots exist or which listings are available, there is nothing fair to mix.
Be able to name the three preconditions - one merged list valid for both, identical layout and slot count, one shared candidate pool - and say which one a given change breaks.
Show the split: interleave the ordering change, take the layout and inventory change to a traffic split, and say plainly what each instrument did and did not establish.
The judgment is about evidence discipline. A cheap instrument applied outside its preconditions produces a number that reads like proof, and that number is what ends up in the launch review.
## What interleaving assumes before it can answer anything Team-draft interleaving is a narrow instrument. It answers one question - **of these two orderings of the same listings, which do people reach for** - and it can only answer it when three things are true at once. | precondition | why it exists | what breaks without it | |---|---|---| | one merged list is a legitimate rendering of both policies | the user must see something either policy could have produced | the merged strip is its own third thing, and the verdict is about it | | identical layout, slot count and placement | position drives attention; both teams must compete for the same real estate | the change under test is the container, not the ranking | | one shared candidate pool | every listing must be draftable by either team | unopposed listings win slots by existing, not by ranking | A change that satisfies all three - a new scoring function over the same retrieved listings, in the same three slots - is exactly what interleaving is for. The change in this question satisfies none. ## Break one: the slot count moved With three slots under the incumbent and four under the candidate, there is no merged strip that is faithful to both: - Render three and the candidate is being judged with its fourth slot removed - a policy it never proposed. - Render four and every user in the test is receiving the candidate's layout change, so the incumbent is being judged inside the candidate's world. - The fourth slot's impressions and clicks have no counterpart in the incumbent world at all, so there is nothing to credit them against. Even if you pick one layout and pretend, the verdict conflates two effects: more sponsored inventory and a different ordering of it. That conflation is not a measurement problem you can subtract out afterwards, because a single number came back. ## Break two: the pool widened Suppose the candidate adds a retrieval source and can surface listings the incumbent never retrieves. Now: 1. On any request where the candidate drafts one of its exclusive listings, the incumbent could not have contested that slot with the same item. 2. If that listing is clicked, the candidate wins the request. 3. The run therefore measures **new inventory** and **better ordering** added together, with no way to tell how much came from each. This is a genuine trap, because a pool change is often the most valuable part of a candidate and the one a team most wants credit for. Interleaving simply is not the instrument that can grant it. ## What a within-list comparison structurally cannot see Both breaks share a deeper limit. Interleaving asks about preference **among the listings that are on the page**, so effects that live above or around the strip are outside its field of view: - whether a bigger strip pushes organic results down and shortens the session; - whether more sponsored density makes users abandon the page; - whether spend moved between sellers because a new source got exposure; - whether the click a user gave the candidate was a click they would otherwise have given to an organic result. None of these are within-list preferences, and none can be recovered from a per-slot credit rule. ## What is still testable this way The useful move in an interview is to split the candidate into the part interleaving can judge and the part it cannot. Hold the layout and the pool fixed, and interleave the **ordering** change alone; that answer is clean and cheap, and it tells you whether the new scoring function is worth carrying further. Then take the slot count and the retrieval source - the parts that change the page and the inventory - to a between-user split where page-level and session-level outcomes are visible, and let the long-term holdout account for what those changes do over months. The red line to hold in a design round: **interleaving compares rankings, not pages.** The moment the container, the density or the pool differs between the two worlds, the merged list stops being a fair rendering of both and the instrument no longer has an opinion worth trusting.
- Could you salvage it by capping the candidate to three slots for the run?You can, and it is often the right move, but be explicit about what you learned: the ranking change over three slots. The fourth slot is untested, and it is usually the part with the largest page-level effect. Shipping it on the back of that verdict is claiming evidence you did not collect.
- The candidate keeps the same pool but applies a seller-diversity rule that suppresses repeats. Is interleaving valid?Usually yes, because it is still an ordering and filtering decision over the same retrieved listings in the same slots. Watch one edge: if the rule suppresses a listing the other team wanted to draft, the draft must still respect the merged strip's constraints consistently, or the suppression will look like a ranking preference.
saying these in an interview costs you the question
- Thinks interleaving can judge a change in slot count
- Believes a bigger candidate pool only makes the test stronger
- Treats a within-list preference win as a page-level result
- Assumes any difference between two rankers is interleavable
- Says the extra slot's clicks can be compared against nothing