What does the discount factor gamma control in a reinforcement learning agent's return?

## The return, not the reward An agent in an MDP does not maximise the next reward; it maximises the **return**, the accumulated reward over the rest of the trajectory. The standard form is the **discounted return**: ``` G_t = r_{t+1} + gamma*r_{t+2} + gamma^2*r_{t+3} + ... = sum over k >= 0 of gamma^k * r_{t+k+1} ``` with `gamma` in `[0, 1]`. A reward `k` steps into the future is multiplied by `gamma^k`. That single number does three jobs at once. ## Job one: it sets the planning horizon Because the weights decay geometrically, gamma decides how far ahead the agent effectively looks. A useful rule of thumb is an **effective horizon of about `1/(1 - gamma)` steps**: `gamma = 0.9` gives roughly 10 steps, `gamma = 0.99` roughly 100, `gamma = 0.999` roughly 1000. Rewards well beyond that horizon contribute almost nothing to the return, so the agent behaves as if they did not exist. Concretely, take a warehouse parcel-sorting robot that receives +1 per parcel delivered, and suppose reaching a pallet takes one step per aisle: | gamma | +1 two aisles away | +1 twenty aisles away | |---|---|---| | 0.5 | 0.25 | about 0.000001 | | 0.9 | 0.81 | about 0.12 | | 0.99 | 0.98 | about 0.82 | At `gamma = 0.5` the far pallet is invisible and the robot works only its local aisle. At `gamma = 0.99` the two pallets are nearly equally attractive and the robot will cross the warehouse. **Nothing about the environment changed — only the objective did.** This is why gamma is a modelling decision with real behavioural consequences, not a nuisance hyperparameter. ## Job two: it keeps the return finite Tasks come in two shapes. - **Episodic**: there is a terminal state and every trajectory ends — a board game reaching checkmate, a delivery run completing. Returns are finite sums, and `gamma = 1` is perfectly well defined. - **Continuing**: there is no terminal state and the interaction goes on indefinitely — a data-centre cooling controller that runs forever. Here the undiscounted sum of rewards can diverge, and two policies that both accumulate infinite reward cannot be compared. With `gamma < 1` and bounded rewards (`|r| <= R_max`), the return is bounded by `R_max / (1 - gamma)`, so it always converges and policies stay comparable. That is the formal reason discounting exists. The alternative for continuing tasks is an average-reward formulation, which optimises reward per step instead — a valid but less common choice. ## Job three: it expresses genuine preference Sometimes discounting is not a mathematical convenience but the truth about the problem. Money now really is worth more than money next year; a customer served today really is worth more than one served next month, because they may churn in between. When the domain has a genuine interest rate or survival hazard, gamma should reflect it rather than be tuned. ## Choosing gamma - **Match it to the horizon that matters.** If a decision's consequences play out over about 50 steps, a gamma with an effective horizon of 5 steps will systematically choose wrong. - **Mind the time step.** Gamma is per *step*, so halving the control interval doubles the number of steps in the same wall-clock horizon and the old gamma is now far too impatient. If you change the sampling rate, you must re-derive gamma. - **Higher is not free.** As gamma approaches 1, return estimates have higher variance, credit takes longer to propagate backwards over many steps, and learning slows down markedly. Practitioners often start lower and raise gamma as learning stabilises. - **Never set gamma = 1 for a continuing task.** The objective is then undefined and the algorithm's estimates can grow without bound. ## What gamma is not It is not a learning rate — that scales an update; gamma changes what the agent is trying to achieve at all. It is not a way to make the agent safer or more cautious; a myopic agent is not a conservative agent, it is just one that ignores consequences. And it does not change the environment's dynamics: two agents with different gammas face the same transitions and will still learn different policies, because they are solving different objectives on the same world. ## The diagnostic use When an agent stubbornly refuses to make an investment that pays off later — the robot never crossing the warehouse, the controller never spinning up cooling in advance of a heat spike — the first suspect is a gamma whose effective horizon is shorter than the payoff delay. Compute `1/(1 - gamma)`, compare it to the number of steps between the action and its consequence, and you often have the diagnosis before you look at anything else.