Why can a candidate send-time model ramped to 10% of users deliver two notifications to one user in a day?

## Why a duplicate send is the failure, not a noisy number In a notification service, the model's output is acted on: the predicted hour becomes a real send. That makes the ramp's assignment an **operational** invariant, not only a measurement one. If a user can be resolved to the candidate side at one moment and the incumbent side at another, two different code paths each believe they are responsible for that user's notification, and the user receives two of them. The user does not see a slightly different send hour; the user sees spam, and the guardrail you are watching — opt-out or mute rate — moves for a reason that has nothing to do with whether the candidate picks good hours. ## The two properties the assignment needs **1. It must be a function, not a draw.** The side is computed from inputs that do not change during the ramp: - a **stable user identifier**, not a session, device or request id; - a **fixed ramp identifier** mixed in, so this ramp's split is independent of any other split running at the same time; - a bucket space large enough to express the percentages you intend to use. Because the same inputs always produce the same bucket, no state has to be stored, no lookup can go stale, and a planner restarting mid-run resumes with the same answers. Crucially, the comparison is `bucket < threshold`, and the threshold only *rises* as the ramp advances — so the only movement is **incumbent to candidate**. A user never returns to the incumbent side because the percentage changed, which is what would otherwise make a user's send hour oscillate day to day. **2. The decision must be resolved once and carried.** The planning job resolves the side, then writes it onto the queued send along with an **idempotency key** — user id, send date, campaign — that is claimed before anything is enqueued. Anything downstream reads the stamped version rather than recomputing it. This is what makes the assignment survive: - a planner retry after a partial failure; - a ramp percentage change made while a day's sends are already planned; - two planning jobs running concurrently during a deploy overlap. ## How the duplicate actually arises | Mistake | What the user experiences | |---|---| | Side drawn randomly per scoring pass | Two planners both claim the user; two notifications | | Hash includes a run id or timestamp | The bucket changes between runs, same outcome | | Percentage re-applied to already-queued sends | A send planned by the incumbent is re-planned by the candidate | | Two planners with separate queues and no shared claim | Each is internally consistent, and together they double-send | | Assignment keyed on device rather than user | A multi-device user is on both sides at once | The last row is the subtle one: the unit of assignment has to be the entity the **action** is aimed at. A notification is aimed at a person, so the person is the unit. ## The single-writer discipline Even with a perfect hash, two processes can both decide to send if neither is authoritative. The durable fix is that exactly one writer claims the right to send to a user for a given day. A claim on the idempotency key is that writer's ticket: the first claim wins, the second planner sees the key taken and skips the user entirely. This costs one conditional write per user per day and removes the whole class. ## What this does *not* solve Stickiness keeps a user on one side; it does not make the side's behaviour good. A candidate that predicts a 5am hour will send at 5am, once, to exactly the 10% assigned to it — and that is precisely the point. A correct ramp makes the harm *proportional and attributable*: one send per user per day, from a known version, to a known share of the population. Everything else in the rollout — guardrail halts, the bake window, the revert — depends on that attribution being true.