In an A/B test, what does it mean for a user to trigger into the experiment?
answer
- assignment is not exposure
- a condition inside the product flow
- 97% never see the change
- the set the change can actually move
basics
~20 sTriggering means the user actually reached the condition where the tested change can act on them, such as opening the settings page a new feature lives on. Assigned users who never reach it are never exposed.
solid answer
~40 sAssignment and exposure are two different events. A user is assigned the moment the experiment platform puts their identifier into control or treatment, which usually happens before they have done anything relevant. A user triggers later, when they reach the point in the product where the two arms would actually behave differently, for example opening the settings page that hosts the new feature. If that page is opened by roughly 3% of assigned users, the other 97% behave identically in both arms no matter how good the change is. So the triggered set is the population the change can possibly move, while the assigned set is everyone the platform touched. Knowing which of the two a number was computed on is the first thing to establish about any experiment readout.
go deeper
Be ready to state plainly that assignment and exposure are different events, and to give one example of a change most assigned users never encounter.
Explain what makes a trigger condition usable: defined before launch, evaluated identically in both arms, timestamped, and never a treatment-only action.
Show that you check trigger instrumentation before trusting any readout, including whether outcomes are counted from exposure rather than from assignment.
Own the convention: every experiment declares its trigger condition and outcome window up front, so nobody negotiates the exposed population after results land.
## Two different moments An online experiment records at least two events per user, and confusing them is the single most common source of misread results. **Assignment** is the platform decision: this identifier belongs to control, that one to treatment. It is typically made early and cheaply, often on the first request of a session, and it happens whether or not the user ever goes anywhere near the thing being tested. **Triggering** (also called exposure, or entering the experiment) is the product moment where the assignment can finally make a difference: the user reaches the screen, the code path, or the state in which treatment and control would render or behave differently. Assignment is about bookkeeping. Triggering is about the physics of the change. ## A concrete shape Suppose the change under test is a new control that only appears on the settings page, and roughly 3% of assigned users open the settings page during the experiment window. Every one of those 3% could in principle behave differently depending on their arm. The remaining 97% see byte-identical product in control and treatment. Their sessions, purchases, retention and churn are drawn from exactly the same distribution in both arms. Those untriggered users are not neutral, though. They contribute outcome values to whichever average you compute, and those values carry variance. They add noise to the comparison without being able to add signal. That asymmetry is the whole reason triggering matters to an analyst rather than only to an engineer. ## What makes a good trigger condition A usable trigger definition has four properties. 1. **It is a condition, not an outcome.** Opening the settings page is a trigger. Clicking the new button is not, because only treatment has a new button, so that set does not exist in control at all. 2. **It is evaluated the same way in both arms.** The control code path has to check the same condition at the same point in the flow and record that the user would have qualified, even though nothing visible happens. Without that record you simply have no control group for the triggered set. 3. **It is fixed before launch.** A trigger chosen after the results are in is a knob, and knobs get turned until the number looks good. 4. **It is timestamped.** Knowing when a user triggered lets you count outcomes only from that moment forward, rather than folding in behaviour from before the change could possibly have mattered. ## Why the timestamp matters If a user triggers on day 6 of a 14-day experiment and you count their entire 14 days of activity, the first 6 days are pure pre-exposure noise attached to a treated unit. Restricting the outcome window to post-trigger activity keeps the measured behaviour to the period in which the treatment could have acted. On metrics with heavy per-user variance, that alone can be a large sensitivity gain. ## Triggering is not a licence to delete users A junior instinct on hearing all of this is: fine, throw away everyone who did not trigger. That is one legitimate readout, but it is a different question from the one the business usually asks, and it is only trustworthy when the control-side trigger record is real and the change does not itself alter who qualifies. The safe default readout keeps every assigned user exactly as assigned; the triggered readout is the sharper instrument that has to be earned with correct logging. ## What to take away When someone quotes an experiment result, the first question is: over which users? Everyone the platform assigned, or only the ones who reached the change? The same underlying effect can be quoted as a big number or a tiny one depending purely on that answer, and neither number is wrong as long as it is labelled.
- If someone is assigned to treatment but never triggers, do you drop them from the analysis?Not by default. The safe readout keeps every assigned user in the arm they were assigned to, because that is the comparison randomisation guarantees. Dropping untriggered users is a separate, sharper readout that requires the control arm to carry a matching record of who would have triggered. If you only have that record on the treatment side, dropping is not an option.
- Does the trigger have to be a single moment in the product?It has to be a single, precisely specified condition, but it can be any well-defined product state: a page load, an eligibility check passing, a first impression of a surface, an API call reaching a branch. What matters is that the condition is defined before launch, evaluated at the same point in the flow in both arms, and timestamped so outcomes can be counted from exposure onward.
Handing out coupons at the door is assignment. Walking up to the till where the coupon can be scanned is triggering, and only the people who reach the till can be affected by it.
saying these in an interview costs you the question
- Assumes everyone assigned to treatment actually saw the change
- Treats assignment and exposure as the same event
- Defines the trigger condition after looking at the results
- Uses a treatment-only action, like clicking the new button, as the trigger
- Counts outcomes from assignment time rather than from trigger time