How do you keep Thompson sampling responsive when an arm's true rate changes over time?

## Why a mature sampler stops adapting Thompson sampling's convergence is also its trap. The posterior width for an arm shrinks roughly like one over the square root of its observation count. An arm with 100,000 observations has a posterior so tight that its plausible range is a fraction of a percentage point wide. That is exactly what you wanted while the world was stationary — the sampler stopped wasting traffic and settled on the leader. Now suppose the world moves. A seasonal promotion whose true click rate genuinely rises in December is a canonical case: yesterday the arm was worth 2%, today it is worth 6%. Two things now conspire: 1. The arm's posterior is concentrated near 2% and puts essentially no mass near 6%, so its draws still lose to the leader. 2. Because its draws lose, it is served almost never — so the new, higher-converting observations that would move the posterior arrive at a trickle. Even when they do arrive, each new observation is one voice against tens of thousands of old ones. The update is swamped. This is lock-in, and it is not a bug in the implementation; it is what an estimator that treats all history as equally relevant is supposed to do. ## Deliberate forgetting The cure is to stop treating a year-old observation as evidence about today. **Discounting.** Each round, before folding in new data, multiply each arm's accumulated counts by a discount factor `gamma` slightly below one, pulling them back toward the prior. Old observations then decay geometrically. In steady state the effective number of observations an arm retains is about `1 / (1 - gamma)`: with `gamma = 0.99` an arm remembers roughly the last 100 observations, with `gamma = 0.999` roughly the last 1,000. The posterior width stops shrinking past the width implied by that effective count, so the arm can always be re-explored. Applying the discount to *all* arms each round, not only the played one, is what lets an unserved arm drift back toward the prior and start winning draws again — which is precisely the re-exploration you need. **Sliding window.** Simpler and easier to explain: keep only the last W observations per arm and rebuild the posterior from those. Sharper forgetting than geometric discounting, at the cost of a hard boundary and more storage. **Periodic reset.** Blunt but sometimes right: at a known boundary — a seasonal changeover, a catalogue refresh — reset every arm's posterior toward the prior. Use it when you *know* the world changed rather than trying to detect it. **Change detection plus reset.** Monitor recent reward rates per arm against what the posterior predicts, and reset only the arm that breaks. This adapts fastest to abrupt shifts but adds a detector with its own false-alarm rate. ## Choosing how much to forget The discount factor is a bet on how fast the world changes, expressed as an effective memory in observations. Pick it from the timescale over which reward rates are actually stable in your system, converted into per-arm observation counts: if rates hold roughly constant for a week and a typical arm sees 5,000 impressions a week, an effective memory of a few thousand observations is defensible. The honest way to set it is to replay historical logs at several values and compare accumulated reward, rather than picking a round number. ## The cost when nothing changes Forgetting is not free. In a genuinely stationary environment, discounting throws away information you paid for. Posteriors stay permanently wider than the data justifies, the sampler keeps sending exploratory traffic to arms it has already resolved, and regret accumulates without bound instead of flattening. Aggressive discounting also makes the allocation noisy: served shares swing around on short-run luck, which looks alarming on a dashboard and can trigger unnecessary intervention. So the choice is a variance-versus-staleness trade: forget quickly and you track change fast but chase noise; forget slowly and you are stable but late. There is no setting that avoids both. ## Operational notes worth raising - **A wide prior at launch does not help.** It only affects the first few observations; months later the posterior is tight regardless. Non-stationarity has to be handled continuously, not at initialisation. - **Distinguish a real shift from a measurement break.** A click-tracking regression looks exactly like a rate change to the sampler, and discounting will happily adapt to the broken numbers. Instrument health checks belong alongside the sampler. - **Bound the allocation velocity.** If the sampler can swing 80% of traffic overnight in response to a discounted posterior, a transient glitch becomes an outage. Rate-limiting how fast served shares can move is a cheap guardrail. ## The one-line answer A stationary Thompson sampler converges and then cannot un-converge; add geometric discounting or a sliding window so posteriors keep a bounded effective sample size, sized to the timescale on which your rates genuinely move.