In an ad auction where treatment and control bid against each other, when is a budget-split test worth its cost?

## Why auctions are the hardest interference case An ad auction is a zero-sum allocation over a fixed inventory: for each impression, exactly one bidder wins, and the price the winner pays is determined by the other bids. That makes every advertiser's outcome a direct function of every other advertiser's behaviour. Randomly splitting advertisers into treatment and control does not create two independent experiments — it creates one auction in which some participants were handed a change and the rest have to live with the consequences. The bias is structural. If the treatment raises effective bids or improves ranking for treated advertisers, then in the same auction: - treated bidders **win more** impressions, - control bidders **win fewer** and, on the impressions they still win, often **pay more**, because the treated bids sit above them in the price ladder, - so the treated-minus-control contrast picks up both the gain and the induced loss. The measured effect is the effect of *having the change while your rivals do not*, which is not the effect of launching it to everyone. At full rollout, every bidder has it, the relative advantage disappears, and what remains is only whatever the change does to total efficiency. ## What a budget split actually isolates A budget-split design creates two parallel auction universes. Traffic is partitioned — each impression is routed to exactly one universe — and each advertiser's budget is partitioned alongside it, so the same advertiser participates in both universes with separate, non-fungible spend. The change is applied in one universe only. Because a bidder in universe A never competes for an impression with a bidder in universe B, the cross-arm distortion is gone and the within-universe comparison is a legitimate contrast between two complete auction environments. Splitting the traffic alone is not sufficient and is a classic implementation failure: if the budgets remain shared, spend exhausted in one universe throttles the same advertiser in the other, and the arms are coupled through the pacing system even though they never bid against each other. ## What it costs **Liquidity.** Each universe runs with a fraction of the bidders and a fraction of the impressions. Auction outcomes are highly non-linear in participation: with fewer competing bids per impression, competition is softer, clearing prices are lower, and the marginal bidder is different. The design therefore removes interference between the arms while changing the market being measured. The estimate is unbiased *for the half-scale world* and only approximately right for the full one — a subtler failure than cannibalization, and one that no amount of running time repairs. **Pacing behaviour.** Budget pacing algorithms behave differently against smaller budgets over the same time window. Effects that appear or vanish because pacing is operating in a different regime will not reproduce at launch. **Engineering.** Routing traffic deterministically, splitting budgets in the accounting system, and keeping the universes leak-proof is substantial work, usually a platform investment rather than an experiment configuration. That is why the decision is a policy decision, not a per-test one. ## The decision rule The question to ask about any candidate change is: **does it reallocate impressions between advertisers?** - Changes to bidding, pacing, reserve prices, ranking or eligibility do reallocate. They need isolation, or the read is inflated. - Changes to reporting, creative rendering, or an advertiser-facing tool that does not alter what gets bid do not reallocate. An ordinary split answers them well, with far more power and no platform investment. When the classification is unclear, a cheap probe helps: run the change at a small treated share and watch the **control side**. If control-arm win rates fall or control clearing prices rise as treatment exposure ramps, the arms are coupled and the ordinary split is compromised. ## The cheaper alternative when isolation is not feasible Ramp the treatment across several exposure shares and study how the estimated per-advertiser effect moves with the treated share. If the effect decays as more advertisers are treated, most of what the small-share test measured was redistribution among bidders. A stable estimate across shares is evidence of genuine incrementality. This is weaker than isolation — the ramp stages differ in more than exposure, and it takes longer — but it costs one experiment configuration rather than a partitioned marketplace, and it often settles the question well enough to make the launch call. ## Translating a half-scale answer into a forecast Even a well-run budget split leaves a translation problem: the business wants a launch number, and the design produced a number from a thinner market. The defensible practice is to treat the budget-split result as evidence about **direction and mechanism**, confirm the magnitude with a staged ramp on live traffic, and state the residual uncertainty explicitly rather than presenting the half-scale point estimate as a forecast. Owning that distinction — unbiased for what, approximate for what — is what separates a principal answer from a competent one.