How do Bahdanau's additive attention score and Luong's multiplicative score differ?

## Same slot, two functions Attention needs a function `score(s, h)` that turns a decoder state `s` and an encoder state `h_j` into one number. Everything downstream — softmax over source positions, weighted average — is identical. The two named families differ only in that function, and the naming comes from the two papers that introduced attention for translation. ## Bahdanau: additive (also called concat) scoring ``` e_j = v^T tanh(W s + U h_j) ``` Read it as a one-hidden-layer network over the pair. `W` maps the decoder state (size `d_dec`) into an attention space of size `d_a`; `U` maps the encoder state (size `d_enc`) into the same space; the two projections are **added**, hence *additive*; `tanh` makes the combination nonlinear; and `v`, a vector of size `d_a`, collapses it to a scalar. Parameter count: `d_a * d_dec` for `W`, `d_a * d_enc` for `U`, `d_a` for `v`. Two consequences follow immediately. First, `d_dec` and `d_enc` may be anything — the projections reconcile them, which mattered in the original design where the encoder was bidirectional and each `h_j` was the concatenation of a forward and a backward state, so twice the decoder's width. Second, the score is not linear in the pair; the tanh lets it express relevance patterns a plain inner product cannot. The cost is the shape of the computation. `W s` is computed once per output step, but `U h_j` is per source position (and can be precomputed once for the whole source, a common optimisation), and the addition, tanh and read-out are per pair — an elementwise nonlinearity over an `S x d_a` buffer at every output step. ## Luong: multiplicative scoring Three variants were proposed; two are the multiplicative ones. ``` dot: e_j = s^T h_j general: e_j = s^T W h_j ``` **dot** has no parameters at all. Relevance is cosine-like alignment in the existing state space, scaled by the two magnitudes. Its constraint is rigid: `d_dec` must equal `d_enc`, and the two spaces must already be comparable — the encoder and decoder must have learned to place related content in the same directions, which they can, since both are trained jointly. **general** inserts one matrix of size `d_dec x d_enc`. That restores flexibility on both counts: sizes may differ, and the model learns a bilinear form saying which decoder directions should match which encoder directions rather than assuming the identity. The reason multiplicative scoring is preferred on real hardware is that the whole set of `S` scores is one matrix-vector (or, batched over output steps, one matrix-matrix) product. There is no elementwise nonlinearity and no `S x d_a` intermediate. One caveat worth naming: the magnitude of an inner product grows with the dimension of the states, so with wide states raw dot scores can become large, and a softmax over large-magnitude scores is very peaked. The `general` variant's matrix can absorb some of that; designers otherwise control it explicitly. (Luong's third variant, *concat*, is the additive form; the family split is really additive versus multiplicative, not one paper versus the other.) ## The wiring difference candidates forget The scoring function is not the only difference between the two designs. - **Bahdanau** computes the alignment from the decoder state *before* the current recurrent update — the previous state `s_{t-1}` — and feeds the resulting context vector into that recurrent step as extra input. Context influences the state. - **Luong** computes the alignment from the decoder's *current* top-layer state `s_t`, then combines context and state (concatenate, then a linear map and tanh) to form the vector that feeds the output distribution. Context influences the prediction; an *input-feeding* variant also carries that combined vector into the next step's input. Both are trained end to end and neither ordering is universally better; interviewers ask about it to see whether you have read the mechanism rather than a one-line summary. ## Choosing - Same-width encoder and decoder, want the cheapest thing that works: `dot`. - Widths differ, or you want a learned notion of which directions match: `general`. - Small model, plenty of compute per score, or you want the extra expressiveness of a nonlinear score: additive. The difference in parameter count is small next to the rest of the model, so the decision is usually driven by shape compatibility and by how much per-pair work you are willing to do.