In tabular Q-learning, when would you prefer a constant learning rate over a decaying one?

## What the learning rate actually does Every update in tabular Q-learning has the form ``` Q(s,a) <- Q(s,a) + alpha * ( target - Q(s,a) ) ``` which rearranges to `Q <- (1 - alpha) * Q + alpha * target`. So `alpha` in `(0,1]` is a blend weight: at `alpha = 1` the stored value is thrown away and replaced by a single noisy sample; at very small `alpha` the table barely moves and learning crawls. Unroll the recursion and a constant `alpha` gives an **exponentially recency-weighted average** of all the targets ever seen for that entry, with older ones decaying by `(1 - alpha)` per visit. A decaying `alpha_n = 1/n` gives instead the plain running average of every target seen. That is the whole tradeoff in one line: exponential recency weighting **tracks**, a running average **converges**. ## The convergence conditions Tabular Q-learning converges with probability one to the optimal action values `Q*` under a small set of assumptions: 1. **Coverage** — every state-action pair is visited infinitely often. This is what the behaviour policy must supply; it need not be a good policy, only a sufficiently curious one. 2. **Robbins-Monro step sizes** — for each state-action pair, `sum of alpha_n = infinity` and `sum of alpha_n^2 < infinity`. 3. Bounded rewards and a discount factor `gamma < 1` (or a properly terminating episodic task). The two step-size conditions have plain meanings. The first says the steps must remain large enough, in total, to move the estimate arbitrarily far — otherwise a bad initialisation is never escaped. The second says they must shrink fast enough that the accumulated sampling noise is finite — otherwise the estimate keeps being knocked around forever. A schedule `alpha_n = n^-p` satisfies both exactly when `0.5 < p <= 1`; `1/n` is the boundary case, and anything decaying faster than `n^-1` fails the first condition. Crucially, `n` here should be the **visit count for that state-action pair**, not the global step counter. Decaying on wall-clock steps starves rarely visited pairs: by the time they are first sampled, their step size is already tiny and they never learn anything. ## What a constant rate costs you A constant `alpha` fails the second condition, and the failure is visible rather than theoretical. On a stochastic environment — say a windy grid where the same action from the same cell sometimes lands somewhere else — the target is a random variable, and with a fixed step size the estimate never stops chasing the newest draw. The values settle into a band around the truth rather than onto it, and the width of that band grows roughly with `alpha`. Set `alpha` high enough and the symptoms are unmistakable: values for frequently visited pairs swing between evaluations, the greedy policy read off the table flips back and forth, and the average return plateaus at a noisy level well below what the problem allows. The fix is boring — halve the rate and see whether the band narrows and the policy stops churning. Note what does *not* happen: tabular Q-learning with a too-large `alpha` thrashes, it does not blow up the way an over-large step can in a parametric model, because each table entry is bounded by the returns it is averaging. "It diverged" is usually the wrong diagnosis here. ## When you deliberately give up the guarantee The convergence proof assumes a **stationary** MDP: fixed transition probabilities, fixed rewards. Plenty of real problems are not. Consider a tabular agent over discretised battery charge levels deciding each hour whether to charge, hold or sell. Tariff structures change, the household's consumption pattern changes with the season, the battery itself ages. The optimal action values genuinely move. A decayed step size in that setting is actively harmful: after a few months the effective learning rate is near zero, the agent is frozen onto a world that no longer exists, and no amount of new evidence can shift it. A small constant rate keeps a rolling window of relevance and lets the table follow the drift, at the price of permanent jitter. So the decision is not "which is correct" but **what the system is being optimised for over its lifetime**: - One-off training against a simulator, a fixed problem, a result you want to claim is optimal → decay per state-action visit count, with the exponent in the valid range. - A long-lived agent in a moving world → small constant rate, accept the noise, and blunt it downstream by acting on smoothed values or by requiring a margin before the greedy action is allowed to switch. - A middle path that is often the pragmatic answer: decay to a floor rather than to zero, which keeps a residual ability to adapt while spending most of the run in the low-noise regime. Whichever you pick, make the choice explicit and monitored. Track policy churn (how often the greedy action changes for high-traffic states) and the spread of a few representative values; those two series tell you within an evaluation cycle whether your step size is too hot or already frozen.