How do optimistic initial Q-values make an agent explore without any random actions?

### The idea Exploration usually comes from randomness. Optimistic initialisation gets it from a **bias in the starting estimates** instead, and the resulting agent can be completely deterministic. Set every entry of the value table to a number higher than any return the task can actually deliver. A greedy agent then behaves like this: it takes the action with the highest estimate; the reward that comes back is smaller than the optimistic promise; the update revises that entry downwards; and the action that is now highest is one it has not tried yet, because untried entries still hold the optimistic value. Repeat, and the agent works through the untried actions systematically, state by state. It stops exploring only when every entry has been pulled down to something realistic — at which point the largest one is genuinely the best it has found. A useful way to say it: **the agent is not curious, it is disappointed**, and disappointment is what moves it on. ### Worked feel for it Take a crop-irrigation agent choosing an irrigation level for each soil-moisture state, where the best achievable per-episode value is around 100. Initialise every `Q(s, a)` to 150. On a dry-soil state it picks whichever action the tie-break favours, say heavy watering, and collects a return worth 60. That entry drops toward 60. Next time the same state is reached, the low-water action still sits at 150 and is now the argmax, so it gets tried — even though nothing in the agent is random and nothing told it to be curious. Within a few visits per state, every action has been sampled at least once, which is precisely the coverage guarantee you want and precisely what a greedy agent with neutral initialisation never provides. ### Why it works better in sequential problems than you might expect In a problem where actions also move the state, optimism does more than sweep the actions of the current state. Value updates bootstrap: the estimate for a state-action pair is pulled towards the reward plus the discounted value of the *successor* state. If the successor has never been visited, its value is still optimistic, so the pair leading to it inherits some of that optimism. The effect is that **unvisited regions of the state space look valuable**, and the greedy policy walks towards them. This is a genuinely directed form of exploration: it heads for the unknown rather than jittering in place, which is exactly what per-step randomness fails to do. ### Setting the initial value You need an upper bound on the achievable value. For a discounted task with maximum single-step reward `r_max` and discount `gamma`, the bound is `r_max / (1 - gamma)`; for an episodic task it is roughly `r_max` times the episode length. Two ways to get it wrong: - **Not optimistic enough.** If the initial value is below what some action really earns, that action's estimate rises rather than falls, the agent keeps re-selecting it, and the mechanism never fires. You get a greedy agent with no exploration at all. - **Absurdly optimistic.** Set it at 10,000 when returns are around 100 and the agent spends a very long time methodically disproving every action in every state before it exploits anything. The cost of that thoroughness is real reward, and in a large state space it may never finish. This scale sensitivity is the main practical objection: `epsilon` is a probability and means the same thing in every problem, while an optimistic initial value has to be re-derived whenever the reward scale changes. ### The core limitation: it wears off Optimism is a **finite quantity of exploration pressure, spent once**. After the initial values have been dragged down to realistic ones, a purely greedy agent has no remaining mechanism for retrying anything. If the environment then changes — a soil type behaves differently after a wet season, an action that used to be poor becomes the best — the agent will never discover it, because discovering it requires taking an action whose estimate says not to. That makes it a **stationary-environment technique**. In a drifting environment you pair it with a small permanent epsilon floor, or periodically reset entries towards optimism, or decay the estimates back up over time. A second limitation: with function approximation, values generalise across states, so pushing one entry down drags related ones with it and the tidy "each action tried once per state" behaviour degrades. It is at its cleanest in tabular problems. ### Combining it with epsilon-greedy They are complementary rather than redundant. Optimism gives fast, systematic coverage at the start — every action tried in every reachable state, with no reward wasted on re-testing options already disproven. A small epsilon floor afterwards keeps a trickle of exploration alive for the rest of the run. Their failure modes are different, so using both is common and sensible.