skip to content

Why does an epsilon-greedy reinforcement learning agent deliberately take non-greedy actions?

level: juniorimportance: must knowfreq 78%

answer

  1. the values are estimates, not truths
  2. one unlucky sample can be permanent
  3. the policy chooses its own training data
  4. epsilon is the random-action probability
  5. the random draw can re-pick the greedy action

basics

~20 s

An agent's action values are only estimates. With probability epsilon it acts randomly so it keeps sampling actions its estimates currently undervalue, which corrects an unlucky early estimate instead of locking onto the first action that looked good.

solid answer

~40 s

Epsilon-greedy takes the action with the highest current estimated value with probability `1 - epsilon`, and a uniformly random action with probability `epsilon`. The random branch exists because those values are estimates built from a few noisy samples: if an action's first returns happen to be poor, a purely greedy agent stops selecting it, and the estimate is then frozen wrong forever because the only evidence that would fix it is evidence the agent will never collect. Exploration keeps every action under revision. The price is real: epsilon of the time you knowingly take an action you believe is worse. One detail worth stating — the greedy action's total probability is `1 - epsilon + epsilon/|A|`, since the uniform branch can select it too.

code

python · 19 lines
python
import random

q = {"left": 1.2, "right": 1.5, "stay": 0.0}   # current value estimates
epsilon = 0.1

def select(values, eps):
    if random.random() < eps:
        return random.choice(list(values))            # explore: uniform over all actions
    best = max(values.values())
    return random.choice([a for a, v in values.items() if v == best])  # exploit

random.seed(0)
counts = {a: 0 for a in q}
for _ in range(10000):
    counts[select(q, epsilon)] += 1

for a in q:
    print(a, counts[a] / 10000)
# left 0.0345 / right 0.9327 / stay 0.0328

go deeper

for a junior

Be ready to state the rule in one line: with probability epsilon act uniformly at random, otherwise take the highest-valued action. Then say why, in one more line: the values are estimates, and a bad early sample would otherwise never be corrected.

for a middle

An interviewer expects the mechanics. Know that the greedy action's total probability is 1 - epsilon + epsilon over the number of actions, and be able to explain why uniform exploration wastes budget on options already ruled out.

for a senior

Show you have actually tuned this. Talk about where you started epsilon, how you knew exploration was still buying information, and why you evaluate a greedy rollout separately from the behaviour the agent shows while training.

for a principal

Own the framing that exploration is a spend, not a hyperparameter detail. Every exploratory step is paid in real reward, and the team should be able to say what that budget buys and when it stops being worth paying.

### What the agent actually knows A reinforcement-learning agent picks actions from an *estimate* of how good each action is. Write it `Q(s, a)`: the agent's current guess at the total discounted reward it will collect if it takes action `a` in state `s` and behaves sensibly afterwards. The word that matters is *guess*. Before training, every entry is whatever you initialised it to. After a few episodes, each entry summarises a handful of noisy samples. The **greedy** action in a state is `argmax_a Q(s, a)` — the one with the highest current estimate. A purely greedy agent always takes it. ### Why pure greed fails Two actions are available in some state. A has a true expected value of 5, B a true expected value of 8, and the rewards are noisy. The agent happens to try B first and gets 2. Its estimate of B drops toward 2. It then tries A and gets 5. Now `Q(A) = 5` beats `Q(B) = 2`, so a greedy agent takes A — and taking A only ever updates `Q(A)`. `Q(B)` stays at 2 for the rest of training. The agent has committed to the worse action on the strength of one unlucky sample, and no amount of extra training rescues it. The deep point to make in an interview is not that greedy is impatient. It is that **the agent's policy decides what data the agent will ever see**. In supervised learning the dataset is handed to you; here the choice of action is also the choice of training example, so a wrong estimate can protect itself indefinitely. Exploration is how you break that loop. ### The rule itself At each decision point, draw a uniform random number. If it is below `epsilon`, pick an action uniformly at random from the available set; otherwise take the greedy action. Two consequences follow. First, **every action in a visited state has selection probability at least `epsilon/|A|`**, where `|A|` is the number of actions. That positive floor is the whole mechanism: an action that keeps being selected keeps being updated, so its estimate can recover from a bad start. Second, the greedy action's total probability is `1 - epsilon + epsilon/|A|`, not `1 - epsilon`, because the random branch can land on it as well. With `epsilon = 0.1` and four actions that is 0.925. ### What exploration costs Every exploratory step is a step where you knowingly take an action you believe is worse. The expected immediate cost per step is roughly `epsilon` times the average gap between the greedy action's value and a random action's value. That is a real, ongoing charge against the reward the agent collects, which is why the parameter is usually reduced as the estimates firm up rather than left fixed. ### Strengths and weaknesses of the uniform scheme It is trivially simple: one knob, no assumptions about the reward distribution, and it works with any way of representing values. Its weakness is that it is **undirected**. An action that has been sampled a thousand times and is clearly bad receives exactly the same `epsilon/|A|` share as one that has never been tried, so in a large action set most exploratory steps buy almost no information. Schemes that weight exploration by how uncertain or how novel an option is are strictly better on that axis, at the cost of extra bookkeeping. The second weakness only appears in sequential problems: because the coin is re-flipped every step, exploratory behaviour is a random walk rather than a purposeful trip to an unexplored region of the state space. ### How it is used in practice `epsilon` typically starts high — often at 1.0, since the value table is meaningless at that point — and is annealed downward as estimates stabilise, usually to a small floor rather than exactly zero so that a change in the environment can still be noticed. Behaviour during training and behaviour at evaluation are separated: you evaluate the greedy policy, because the exploratory steps depress the measured return and would make a good agent look mediocre. ### What an interviewer is listening for The phrase "the values are estimates", the self-confirming-data argument, the fact that exploration is paid for in real reward, and awareness that `epsilon` is normally a schedule rather than a constant.

  • How do you compute the total probability that an epsilon-greedy agent takes the greedy action?
    `1 - epsilon + epsilon/|A|`, where `|A|` is the number of available actions. The greedy branch fires with probability `1 - epsilon`, and the uniform branch, which fires with probability `epsilon`, still picks the greedy action one time in `|A|`. With `epsilon = 0.1` and four actions that is 0.925, not 0.9.
  • What is the main weakness of uniform random exploration?
    It is undirected: it spends the same budget on an action it has strong evidence against as on one it has never tried. In a large action set most exploratory steps therefore buy almost no information. Exploration weighted by uncertainty or novelty targets the same budget at the options where the estimate is actually in doubt.
  • Does exploration still make sense once an agent's values have converged?
    Only if the environment can change. In a stationary problem, exploring after convergence is pure cost and epsilon should go towards zero. In a drifting environment a small permanent floor is what lets the agent notice that a previously poor action has become good — you accept a permanent trickle of regret in exchange for staying current.

A commuter who tries one route, hits a crash on day one, and never drives it again has priced that road on a single bad morning. Occasionally taking it anyway is the only way to find out it is normally the fastest.

saying these in an interview costs you the question

  • Says exploration just adds noise, with no purpose given
  • Claims a purely greedy agent still reaches the optimal policy
  • Thinks epsilon is the probability of taking the best action
  • Believes exploration is free because rewards get averaged
  • Cannot say what exploration costs in reward terms

context

resolving links…