What do a GRU's update and reset gates do, and how does that differ from an LSTM's gating?
answer
- two gates, not three
- one knob ties keeping to writing
- reset acts before the recurrent matrix
- three weight blocks against four
basics
~20 sA GRU has two gates. The update gate blends the old hidden state with a new candidate; the reset gate controls how much past state feeds that candidate. An LSTM uses three gates plus a separate cell state.
solid answer
~50 sA GRU cell computes two sigmoid gates from the input and previous hidden state: an update gate `z_t` and a reset gate `r_t`. The candidate is `cand_t = tanh(W_h x_t + U_h (r_t * h_(t-1)) + b_h)` — the reset gate multiplies the previous state *before* it enters the candidate, so it can suppress carried history. The new state is a convex blend: `h_t = (1 - z_t) * h_(t-1) + z_t * cand_t`. That single interpolation does what an LSTM splits across a forget gate and an input gate: keeping and writing are tied, not independent. A GRU also has no separate cell state and no output gate, so the hidden state it exposes is the whole state. Structurally that is three weight blocks (`z`, `r`, candidate) against an LSTM's four, roughly 25% fewer parameters at the same input and hidden size.
code
python · 24 linesimport math
def sig(v): return [1.0 / (1.0 + math.exp(-a)) for a in v]
def tnh(v): return [math.tanh(a) for a in v]
def affine(W, x, U, h, b):
return [sum(W[i][j] * x[j] for j in range(len(x)))
+ sum(U[i][j] * h[j] for j in range(len(h))) + b[i]
for i in range(len(b))]
x, h = [1.0, -2.0], [0.5, -0.5]
Wz, Uz, bz = [[0.3, -0.1], [0.0, 0.2]], [[0.1, 0.0], [-0.2, 0.1]], [0.0, 0.0]
Wr, Ur, br = [[-0.4, 0.2], [0.1, 0.3]], [[0.0, 0.1], [0.2, -0.1]], [0.0, 0.0]
Wh, Uh, bh = [[0.5, 0.1], [-0.3, 0.4]], [[0.2, -0.2], [0.1, 0.3]], [0.0, 0.0]
z = sig(affine(Wz, x, Uz, h, bz)) # update gate
r = sig(affine(Wr, x, Ur, h, br)) # reset gate
reset_h = [r[i] * h[i] for i in range(2)] # history is gated first
cand = tnh(affine(Wh, x, Uh, reset_h, bh))
h_new = [(1 - z[i]) * h[i] + z[i] * cand[i] for i in range(2)]
print('z', z)
print('r', r)
print('h_new', h_new)go deeper
Be able to name the two gates and say what each is for: the update gate decides how much of the old state survives, the reset gate decides how much past state the new proposal may use.
Expect to write the three equations from memory, including the fact that the reset gate multiplies the previous state before the recurrent matrix, and to derive the three-blocks-against-four parameter ratio.
Show you know what the tied keep-and-write knob costs you in expressiveness and where the 25% weight cut actually pays off — footprint and per-step memory traffic on small on-device cells.
Own the framing that a same-hidden-size comparison is not parameter-matched, and that picking a cell type is a budget and maintenance decision rather than an architectural truth claim.
## The problem every recurrent cell has to solve A recurrent cell reads one element of a sequence at a time and keeps a hidden state — a fixed-size vector of numbers — that summarises everything seen so far. A plain recurrent cell overwrites that state at every step with `h_t = tanh(W x_t + U h_(t-1) + b)`. Because the state passes through a multiplication and a squashing nonlinearity at every step, information from far back is repeatedly rescaled and gradients shrink toward zero as they are propagated backwards through many steps. Gated cells attack this by learning, per unit and per step, *how much* to keep and *how much* to overwrite. ## The two GRU gates A gated recurrent unit (GRU) uses two gates, each a vector of numbers between 0 and 1 produced by a sigmoid: - **Update gate**: `z_t = sigmoid(W_z x_t + U_z h_(t-1) + b_z)` - **Reset gate**: `r_t = sigmoid(W_r x_t + U_r h_(t-1) + b_r)` The candidate state — the proposal for what the new state could be — is `cand_t = tanh(W_h x_t + U_h (r_t * h_(t-1)) + b_h)` where `*` is elementwise multiplication. Note *where* the reset gate acts: it multiplies the previous hidden state **before** the recurrent matrix `U_h` is applied. With `r_t` near 0 for a unit, that unit's candidate is computed almost entirely from the current input, as if the sequence had just started. That is the mechanism behind restarting a cell mid-stream at a sentence or session boundary: the carried history stops feeding the proposal. The state update is a convex interpolation: `h_t = (1 - z_t) * h_(t-1) + z_t * cand_t` With `z_t` near 0 a unit copies its previous value forward almost unchanged; the local Jacobian along that unit is close to the identity, which is what lets gradients survive many steps. With `z_t` near 1 the unit is overwritten by the candidate. (Sign conventions differ between write-ups — some place `z_t` on the old state — but it is always the same one-knob interpolation.) ## What is structurally missing compared to an LSTM An LSTM keeps two vectors, a cell state and a hidden state, and uses three gates — forget, input and output — plus a candidate block. Two consequences follow. First, **keeping and writing are tied in a GRU**. An LSTM's forget gate and input gate are computed separately, so it can keep the old contents *and* add new contents at the same step, or discard without writing. A GRU's single `z_t` cannot: whatever fraction it writes is exactly the fraction it stops keeping. That is a real loss of expressiveness, and it is also why a GRU has fewer knobs to mis-set. Second, **a GRU has no output gate and no separate cell state**. An LSTM can hold something in its cell state while hiding it from the rest of the network; a GRU's hidden state is simultaneously its memory and its output, so anything it retains is visible to the next layer. ## The parameter arithmetic With input size `D` and hidden size `H`, each weight block costs `H*D + H*H + H` parameters (an input matrix, a recurrent matrix, a bias). A GRU has three such blocks — update, reset, candidate — so `3 * (H*D + H*H + H)`. An LSTM has four — forget, input, output, candidate — so `4 * (H*D + H*H + H)`. The ratio is exactly 3/4: a GRU carries about 25% fewer weights at the same sizes, and reads about 25% fewer bytes of weights per step. On a small on-device model — say a 64-unit cell for keyword spotting — that cut lands on both the model footprint and the per-step memory traffic, which is why GRUs show up disproportionately in embedded audio front-ends. The corollary matters when you benchmark: comparing a GRU and an LSTM *at the same hidden size* is not a parameter-matched comparison. If you want capacity matched, widen the GRU by roughly 2/sqrt(3) in the hidden-size term, or say plainly which quantity you held fixed. ## Two details that are commonly got wrong **The reset gate does not clear the state.** It only removes history from the *candidate*. Even with `r_t = 0`, the update `h_t = (1 - z_t) * h_(t-1) + z_t * cand_t` still carries `(1 - z_t) * h_(t-1)` forward. A genuine mid-stream restart needs `r_t` near 0 **and** `z_t` near 1 for those units. **A GRU is not a cheap approximation that always loses.** On many mid-sized sequence-tagging and acoustic tasks the two cells land within seed-to-seed noise of each other, and the GRU trains slightly faster per step. The honest summary is that the gap is task- and budget-dependent, not that one architecture dominates.
- Does driving the reset gate to zero wipe a GRU unit's memory at a session boundary?Not by itself. The reset gate only removes carried history from the candidate; the update still adds `(1 - z_t) * h_(t-1)`, so the old value keeps flowing. A real mid-stream restart needs the reset gate near 0 and the update gate near 1 together, so the unit both proposes an input-only candidate and writes it over the old state.
- How many parameters does a GRU layer have relative to an LSTM layer at the same input and hidden size?About three quarters. Each weight block costs `H*D + H*H + H`; the GRU has three blocks (update, reset, candidate) and the LSTM has four (forget, input, output, candidate), so the ratio is 3/4 regardless of the sizes. That is also roughly the cut in bytes of weights read per step.
- Why does a GRU need no output gate?Because it has only one state vector. An LSTM's output gate exists to decide how much of the internal cell state is exposed as the hidden state; a GRU's hidden state is both the memory and the exposed output, so there is nothing to filter between them. The cost is that a GRU cannot retain something while hiding it from the next layer.
An LSTM has separate taps for draining the tank and filling it; a GRU has one mixing valve, so it can only refill exactly as fast as it drains.
saying these in an interview costs you the question
- Says a GRU keeps a separate cell state like an LSTM
- Claims the reset gate decides what is written to the new state
- Thinks the update gate has independent keep and write knobs
- Says a GRU and an LSTM have the same parameter count
- Places the reset gate after the recurrent matrix instead of before it