How do you decide whether in-app upsell placement is a contextual bandit or full reinforcement learning?

## The one question that decides it Both formulations involve choosing actions and learning from the rewards those actions produce, and both face the tension between taking the action that currently looks best and taking one that would teach you more. That shared tension is what makes people reach for the reinforcement-learning label. It is the wrong discriminator. The discriminator is: **does the action I take now change the situation I will face next?** If it does not, the problem is a contextual bandit. If it does, the problem is sequential and the machinery for attributing reward across a sequence of actions becomes necessary. ## The anatomy of a contextual bandit round A single round has four parts and no fifth: 1. **Context.** A feature vector describing the situation — plan tier, tenure, screen, device, session depth, recent activity. It arrives from the world and the system does not choose it. 2. **Arm.** One action selected from a fixed, usually small set — here, which upsell placement or which of four send-hours for an email. 3. **Reward.** A number observed for the chosen arm only: the upsell was accepted or it was not. 4. **End of round.** The next context is drawn from the world independently of the arm just played. What is absent is a state transition. Nothing the system did carries into the next decision except through what it learned. That is the whole difference, and it is why a contextual bandit sits between supervised learning and reinforcement learning: it has supervised learning's one-shot decisions and reinforcement learning's partial feedback, without reinforcement learning's dynamics. ## Applying it to upsell placement In the simple version of the product, an upsell impression pays its own reward — accepted or dismissed — and then the user is back where they were: same plan, same screen, same intent. The next impression's context is essentially independent of the placement chosen last time. That is a bandit, and modelling it as anything more is paying for machinery you cannot feed. The interesting part of the interview is the honest wrinkle. Real upsells do have carry-over. Show the same prompt five times in one session and the user is more likely to dismiss it, more likely to be annoyed, marginally more likely to churn. Strictly, that is a state transition. So does it become reinforcement learning? ## The pragmatic middle move Usually not, and the reason is worth being able to argue. Two cheap changes absorb mild carry-over inside the bandit formulation: - **Put the state into the context.** Impressions in the last hour, days since the last upsell, dismissals this session, whether the user has ever accepted. If the thing that changes between rounds is observable, it can simply be a feature, and the decision stays one-shot conditional on it. - **Price the harm into the reward.** If a dismissal has a cost, subtract it. If churn risk matters, define the reward on a horizon that includes it rather than on the immediate accept. Those two moves cover a large share of what looks like sequentiality. What they do not cover is a problem where the *optimal* action now is one with a negative immediate reward, taken specifically to reach a state where a much larger reward becomes available later. When the strategy genuinely requires sacrificing now to set something up later, no amount of feature engineering rescues the bandit framing, and the sequential formulation earns its cost. ## Why the simpler formulation is the default The bandit formulation is not merely easier to implement; it is cheaper on every axis that matters at scale. - **Data.** Each round is an independent training example of the form (context, arm, reward). Learning a reward model per arm is close to a supervised problem. Sequential methods have to estimate long-run consequences, which requires coverage of whole sequences and lets errors compound along them. - **Evaluation.** Judging a candidate bandit policy from logs is hard, but tractable, because the unit of evaluation is a single decision. Judging a candidate sequential policy means reasoning about a whole trajectory that the logged policy never produced. - **Failure mode.** A bad bandit decision costs one impression. A bad sequential policy can steer users down a path over many steps before anyone notices. - **Debuggability.** A stakeholder can be shown why a given user saw a given placement. Explaining a decision that was taken for its effect three steps ahead is much harder to defend in a product review. ## What it costs to get this wrong in each direction Model a genuinely sequential problem as a bandit and the policy turns greedy about immediate reward: it shows the upsell on every impression because each one individually looked positive, while the slow damage to retention never enters the reward being optimised. The signature is short-term metric gains alongside a decline in a metric that is not measured per round. Model a one-shot problem as reinforcement learning and you pay for a simulator or an enormous volume of interaction, adopt an evaluation story nobody can validate, and take on a system whose failures are hard to attribute — all to solve a problem that a well-featured bandit would have handled with a fraction of the traffic.