How does an attention-based seq2seq decoder build its context vector at each output step?
answer
- the decoder looks back every step
- one score per source position
- normalise across source, not target
- weights sum to one
- weighted average of encoder states
basics
~20 sAt each output step the decoder scores its current state against every encoder state, softmaxes those scores over source positions into weights summing to one, then averages the encoder states with those weights — that average is the context vector.
solid answer
~50 sThe encoder runs once and leaves one hidden state per source position, `h_1 ... h_S`. At output step `t` the decoder holds a state `s_t`, and a score function produces one scalar `e_tj = score(s_t, h_j)` for every source position. Those `S` scores go through a softmax **across source positions**, `a_tj = exp(e_tj) / sum_k exp(e_tk)`, so they are non-negative and sum to one. The context vector is the weighted average `c_t = sum_j a_tj * h_j`, a point inside the convex hull of the encoder states, pulled toward whichever positions scored highest. It is then combined with the decoder state to predict the next token. Nothing supervises the weights — they are learned end to end from the training loss, which is why the resulting `T x S` matrix is called a *soft* alignment.
code
python · 15 linesimport math
enc = [[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]] # three encoder states
dec = [0.8, 0.2] # decoder state at this output step
mask = [True, True, False] # third source position is padding
scores = [sum(d * hk for d, hk in zip(dec, h)) for h in enc] # dot score
scores = [s if keep else -1e9 for s, keep in zip(scores, mask)] # pads first
exps = [math.exp(s) for s in scores]
total = sum(exps)
alpha = [e / total for e in exps]
context = [sum(a * h[k] for a, h in zip(alpha, enc)) for k in (0, 1)]
print([round(a, 4) for a in alpha]) # [0.6457, 0.3543, 0.0]
print([round(c, 4) for c in context]) # [0.6457, 0.3543]go deeper
Be ready to recite the three steps in order: score every encoder state against the current decoder state, softmax those scores over source positions, take the weighted average. Say out loud that the weights sum to one.
Explain why the softmax axis is the source axis and what that implies for the T-by-S alignment matrix, and be able to state that the encoder runs once while scoring and mixing repeat each output step.
Expect to talk about cost and correctness in practice: score count growing as source length times target length, padded positions masked before the softmax rather than zeroed after, and what a plotted alignment map tells you about a run.
Own the framing that soft attention is a differentiable relaxation of hard selection, trained with no alignment supervision, and be ready to argue when that interpretability is worth relying on for debugging versus when it is merely suggestive.
## The setup A sequence-to-sequence model has two halves. The **encoder** reads the source sequence (say a 50-token sentence, or a few hundred audio frames) and produces one hidden vector per input position: `h_1, h_2, ..., h_S`. The **decoder** emits the output sequence one token at a time, carrying its own recurrent state `s_1, s_2, ..., s_T`. Attention is the mechanism that lets the decoder, at every one of its `T` steps, look back at all `S` encoder states and decide how much of each to use. ## The three operations, in order **1. Score.** A scoring function assigns one real number per source position: ``` e_tj = score(s_t, h_j) for j = 1..S ``` The score measures how relevant source position `j` is to the token about to be emitted at step `t`. It is a scalar, not a vector — one number per (output step, source position) pair. **2. Normalise.** The `S` raw scores are pushed through a softmax **over the source axis**: ``` a_tj = exp(e_tj) / sum_{k=1..S} exp(e_tk) ``` This is the step candidates most often get backwards. The normaliser runs over *source positions for a fixed output step*, not over output steps. The consequence: for each output step, `a_t1 + a_t2 + ... + a_tS = 1`, every weight is in `[0, 1]`, and the vector `a_t` is a probability distribution over where to look. If you stack all steps you get a `T x S` alignment matrix whose **rows** sum to one; its columns need not sum to anything in particular, which is exactly why a source word can be attended to twice (over-translation) or never (under-translation). **3. Mix.** The context vector is the weighted average of the encoder states: ``` c_t = sum_{j=1..S} a_tj * h_j ``` Because the weights are non-negative and sum to one, `c_t` lives in the convex hull of the encoder states — it is a soft, differentiable selection. A one-hot `a_t` would be hard selection (pick exactly one source position); softmax gives the smooth relaxation that gradients can flow through. That differentiability is the whole trick: no alignment labels are needed, and the score function's parameters are trained by the same next-token loss that trains everything else. `c_t` is then combined with `s_t` — concatenated and passed through a small transform, or fed into the recurrent step — before the output distribution over the vocabulary is produced. ## What is recomputed and what is not The encoder runs **once per example**. Its states `h_j` are computed and then reused at every output step. What is redone at each of the `T` steps is only the cheap part: `S` scores, one softmax of length `S`, and one weighted sum. So the cost is `S x T` score evaluations — for a 50-token source and a 60-token target, 3000 of them — plus `S x T` scalar-times-vector accumulations. Each is trivial; it is the product of the lengths that grows, and that is what makes very long sources expensive. ## Padding Real batches pad short sequences up to the longest in the batch. Those padded positions still have encoder states, and the softmax will happily assign them weight. The fix belongs **before** the softmax: set `e_tj` at padded positions to a large negative value, so `exp(e_tj)` is effectively zero and the surviving weights still sum to one. Zeroing the weights *after* the softmax is the common bug — the pads have already eaten part of the normaliser, so the remaining weights sum to less than one. ## Why the weights are interpretable Plotting the `T x S` matrix as a heatmap gives a picture of which source positions drove which output tokens. For translation between similarly ordered languages it looks like a bright near-diagonal band. That interpretability is a by-product, not a guarantee: the weights show where the model read from, which is correlated with, but not the same as, a linguist's alignment. ## Common mistakes - Believing the context is the encoder's last hidden state. That is the no-attention baseline; the point of attention is that `c_t` changes with `t`. - Softmaxing over the wrong axis, which would make the weights compete across output steps. - Thinking the encoder is re-run per output step. It is not; only the scoring and mixing repeat. - Treating alignment weights as learned constants. They are computed from the current states, so they differ per example and per step.
- What length does the alignment weight vector have at one output step, and which axis does the softmax run over?It has one entry per source position, so its length is the source length `S`. The softmax runs over that source axis for a fixed output step. Stack all `T` steps and you get a `T x S` matrix whose rows each sum to one; columns are unconstrained, which is how a source position can be attended to repeatedly or skipped entirely.
- Does the encoder have to run again for each output step?No. The encoder runs once per example and its `S` hidden states are cached. Each output step only recomputes `S` scores, one softmax, and one weighted sum over those cached states. That is why attention adds cost proportional to source length times target length rather than re-encoding cost.
- How many alignment scores does this cost for a 50-token source and a 60-token target?3000 — one score per (source position, output step) pair, so `50 x 60`. Each score is cheap, but the count is the product of the two lengths, so doubling the source doubles the scoring work for every output token. This product is the reason long-source attention gets expensive well before the encoder itself does.
saying these in an interview costs you the question
- Says the softmax is taken over target positions instead of source
- Claims the context vector is the encoder's final hidden state
- Thinks the encoder is re-run at every output step
- Describes alignment weights as fixed learned parameters per position
- Says the context vector concatenates all encoder states