Why is an embedding lookup equivalent to multiplying a one-hot vector by a weight matrix?
answer
- one row per id in a matrix
- picking a row is a matrix product
- the zeros do no useful work
- backward pass is an outer product
basics
~20 sAn embedding table is a weight matrix with one row per id. A one-hot vector times that matrix selects exactly that row and zeroes the rest, so the lookup is the same linear layer computed by indexing instead of multiplying.
solid answer
~40 sAn embedding table is a weight matrix `W` of shape (vocabulary size, width), and a one-hot vector for id `i` times `W` returns row `i` and nothing else -- every other term is multiplied by zero. So an embedding layer is a fully-connected layer whose input happens to be one-hot; the row gather is an implementation shortcut, not a different operation. For a 50,000-symbol vocabulary the matmul does 50,000 multiply-adds per output number, almost all against zeros, and forces you to materialise the one-hot block; the gather copies one row. The equivalence also settles the backward pass: the gradient with respect to `W` is the outer product of the one-hot vector with the upstream gradient, which is zero on every row except `i`. Only the rows a batch actually used receive gradient.
code
python · 19 linesW = [[0.1, 0.2, 0.3], # row 0
[0.4, 0.5, 0.6], # row 1
[0.7, 0.8, 0.9]] # row 2
idx = 1
one_hot = [1.0 if r == idx else 0.0 for r in range(len(W))]
via_matmul = [sum(one_hot[r] * W[r][c] for r in range(len(W)))
for c in range(len(W[0]))]
via_gather = W[idx]
print(via_matmul) # [0.4, 0.5, 0.6]
print(via_gather) # [0.4, 0.5, 0.6]
# The gradient wrt W is the outer product of the one-hot vector with the
# upstream gradient, so every row except `idx` is exactly zero.
upstream = [0.01, -0.02, 0.03]
grad_W = [[one_hot[r] * upstream[c] for c in range(3)] for r in range(3)]
for row in grad_W:
print(row) # [0.0, -0.0, 0.0] / [0.01, -0.02, 0.03] / [0.0, -0.0, 0.0]go deeper
Be ready to say what the layer stores and returns: a table with one row per id, and one dense vector out per id. Know that the rows are learned parameters, not something looked up from a fixed dictionary.
This is your tier. Derive the equivalence out loud -- one-hot times the matrix cancels every row but one -- and explain why the gather is used anyway: no zero multiplications, no one-hot block in memory.
Show that you draw consequences from the equivalence: gradient reaches only the rows a batch touched, so rows train at a rate set by id frequency, and dense weight decay or momentum can still move rows the batch never used.
Own the framing that an embedding layer buys no expressive power on its own -- it is a linear reparameterisation of the first layer. Argue where the modelling effort should go instead, and when a per-id row is the wrong way to spend parameters at all.
## What the layer actually is An **embedding lookup layer** owns exactly one thing: a matrix, call it `W`, with **one row per distinct id** in a fixed vocabulary and **one column per embedding dimension**. If a categorical field can take `V` different values and you choose a width `d`, then `W` has shape `(V, d)` and holds `V * d` learned parameters. The layer's input is an integer id (or a batch, or a batch of sequences of them); its output is *the row of `W` at that integer*, a vector of length `d`. There is no bias term and no activation function inside the layer. The rows start as random numbers. They are parameters like any others, and they are trained by the same loss that trains the rest of the network. Whatever structure ends up in the table -- similar ids landing near each other -- is a by-product of the task, not something the layer is told to do. ## The one-hot equivalence, step by step Write id `i` as a **one-hot vector** `e_i` of length `V`: all zeros, with a single `1` at position `i`. Now compute the ordinary matrix product `e_i @ W`. For output column `c`: ``` out[c] = sum over r of e_i[r] * W[r][c] ``` Every term with `r != i` is multiplied by `0` and vanishes; the single surviving term is `1 * W[i][c]`. So `out = W[i]`, exactly the row the lookup returns. A fully-connected layer with weight matrix `W`, no bias, fed a one-hot vector, **is** the embedding lookup. They are not two similar operations; they are the same function written two ways. That is why an embedding layer is a *linear* layer. It adds no expressive power of its own beyond "each id gets its own free vector". All the nonlinearity in the model lives in the layers after it. ## Why nobody computes it as a matmul Take a 50,000-symbol vocabulary and a width of 128. The one-hot form asks for a `(batch, 50000)` matrix multiplied by a `(50000, 128)` one: 50,000 multiply-adds per output number, of which 49,999 are multiplications by zero. It also asks you to *materialise* the one-hot block -- for a batch of 256 items that is 12.8 million numbers whose only content is 256 ones. The gather form copies one row of 128 numbers per id. Same output, a few hundred times less arithmetic and orders of magnitude less memory. The equivalence is the justification; the gather is the implementation. ## What the equivalence tells you about the backward pass Because the forward operation is `e_i @ W`, the gradient of the loss with respect to `W` is the **outer product** of `e_i` with the upstream gradient `dL/dout`: ``` dL/dW[r][c] = e_i[r] * dL/dout[c] ``` which is `dL/dout[c]` on row `i` and **exactly zero on every other row**. Over a batch, the gradient of a row is the sum of the upstream gradients of every position in the batch that used that id. Three occurrences of an id in one batch means three contributions added into one row. Three practical consequences follow directly: 1. **Rows train at a rate set by their frequency.** A head id that appears in most batches gets thousands of updates per epoch; a tail id that appears twice in the whole dataset gets two. The table's quality is wildly uneven across rows even though every row has the same shape and the same learning rate. 2. **Most of the table is idle on any given step.** Only the touched rows have gradient. Optimizers that keep per-parameter state or apply weight decay to *all* parameters can still move untouched rows -- momentum carries a row forward, decoupled weight decay shrinks it -- so "untouched" and "unchanged" are not the same thing. 3. **Sparse gradient is not a sparse weight matrix.** The table is fully dense; it is the *gradient* that is mostly zeros on a given step. ## Common confusions worth naming - *"An embedding layer is a nonlinear transformation."* It is a linear map from a one-hot space. It cannot even represent a nonlinear function of the id, because the id has no numeric meaning to be nonlinear in. - *"The output is a one-hot vector."* The input side is conceptually one-hot; the output is a dense width-`d` vector. - *"It compresses the input."* It is better described as re-parameterising: instead of one weight per (id, output-unit) pair in the first dense layer, you learn `d` numbers per id and let the following layers share the mapping from those `d` numbers onward. - *"Feed the integer instead."* A single weight times the integer forces every id onto one shared direction, scaled by an id number that carries no magnitude or ordering. Ids are nominal labels; the embedding gives each one an unconstrained vector, which is precisely the freedom the integer denies. ## How to say it in an interview "An embedding table is a weight matrix with one row per id, so multiplying a one-hot vector by it selects a row -- the lookup and the matmul are the same linear layer. We index instead of multiplying because the matmul is almost entirely multiplications by zero and forces you to build the one-hot block. The equivalence also fixes the backward pass: the gradient is an outer product with the one-hot vector, so only the rows a batch touched get gradient, and rare ids therefore get very few updates."
- Why not feed the raw integer id into the first dense layer instead?A single weight times the integer forces every id onto one shared direction, scaled by the id number: id 900 becomes 300 times id 3, an ordering and a magnitude that nominal ids do not have. An embedding gives each id its own unconstrained vector, so unrelated ids can sit anywhere and related ones can end up close together because the loss put them there.
- If the lookup is a linear layer, does it need a bias or an activation?Neither, normally. A bias would add the same constant to every id's vector, which the next layer's own bias already absorbs. An activation on the rows is just a reparameterisation of free parameters -- and a saturating one only restricts their range. Nonlinearity belongs in the first layer after the embeddings are combined.
- After one training step on a batch of 64 ids, which rows of the table have changed?Only the rows those 64 ids indexed have nonzero gradient; a row used three times in the batch accumulates three contributions. But an optimizer carrying momentum, or applying weight decay to all parameters, can still move untouched rows, so a rare id's vector drifts or decays between the steps that actually train it.
A one-hot vector is a shopping list with exactly one item ticked. The matmul walks every shelf in the store and takes only the ticked item; the lookup walks straight to that shelf. Same basket, very different walk.
saying these in an interview costs you the question
- Calls the embedding layer a nonlinear transformation
- Says the lookup learns a different function than the matmul
- Thinks every row updates on every training step
- Confuses a sparse gradient with a sparse weight matrix
- Says the layer outputs a one-hot vector