How do you count the parameters of a 784-128-10 fully-connected neural network?

## What counts as a parameter A **parameter** is a number that training updates: the weights and the biases stored inside the layers. It is not a hyperparameter (the layer widths, the depth, the learning rate — you choose those), and it has nothing to do with the data (the number of training rows, the batch size, the number of epochs). When someone says "this model has 101,770 parameters", they mean: this is how many numbers the optimizer is allowed to move. ## One fully-connected layer A fully-connected (dense) layer takes `n_in` numbers in and produces `n_out` numbers out. Every output unit computes a weighted sum over **all** of the inputs and then adds its own bias: ``` z_j = b_j + sum over i of ( w_ij * x_i ) ``` So one output unit owns `n_in` weights plus `1` bias — `n_in + 1` numbers. There are `n_out` such units, giving ``` params = n_in * n_out + n_out = n_out * (n_in + 1) ``` The first term is the weight matrix: one entry per (input unit, output unit) pair. The second term is the bias vector: one entry per output unit. Whether you store the matrix as `n_in` by `n_out` or transposed as `n_out` by `n_in` is a notational choice and does not change the count — only the two widths matter. ## Walking a whole MLP Write the network as a list of widths and step through **adjacent pairs**. For the classic handwritten-digit MLP with a 784-pixel input, one 128-unit hidden layer and 10 output classes, the widths are `[784, 128, 10]`, which gives two pairs: - `784 -> 128`: `784 * 128 = 100,352` weights `+ 128` biases `= 100,480` - `128 -> 10`: `128 * 10 = 1,280` weights `+ 10` biases `= 1,290` Total: **101,770 parameters**. Two things trip people up here. First, the input "layer" contributes nothing of its own — it is just the data arriving, and the 784 only appears because it is the fan-in of the first weight matrix. Second, each hidden width is used **twice**: 128 is the output width of the first layer and the input width of the second. It is counted once in each product, never twice in the same product. The pattern generalises to any depth. With widths `[w0, w1, ..., wL]` the total is the sum over consecutive pairs of `w_(k) * w_(k+1) + w_(k+1)`. ## Biases: always count them, never worry about them In the digit MLP the biases are `128 + 10 = 138` out of 101,770 — about **0.14 percent**. The gap widens as layers get wider, because weights grow with the product of two widths while biases grow with just one. An eight-layer trunk of 2048-by-2048 layers carries `8 * 2048 * 2048 = 33,554,432` weights against `8 * 2048 = 16,384` biases — under **0.05 percent**. That is the whole story of biases in a parameter count: they are the term candidates forget, and the term that never changes the answer. Forgetting them in an interview reads as carelessness rather than as a rounding decision, so say them out loud even while noting they are negligible. Dropping them is a real design choice in some layers, but it is a modelling decision, not a way to make a model smaller. ## What does not change the count - **Batch size.** The same weight matrix is applied to every row in the batch; 1 row or 1,024 rows, the parameters are identical. - **Dataset size and epochs.** Training longer or on more data moves the parameters; it does not create more of them. - **Most activation functions.** A ReLU, a sigmoid or a tanh has no learnable numbers at all, so it adds zero. Only a *parametric* activation, which learns a coefficient of its own, contributes anything — and then only a handful of values. ## Doing it fast, out loud Interviewers often want the count on a whiteboard in under a minute, so practise the shortcut: multiply the adjacent widths, keep the largest product first, then add the small change. For `[784, 128, 10]`: "about 100k in the first layer, about 1.3k in the second, so roughly 101.7k — 101,770 exactly with biases." Leading with the dominant term shows you know where the mass sits, and it makes an arithmetic slip in the small term harmless. ## Where hand counts go wrong Counting the input units as parameters; adding the widths instead of multiplying them; skipping the bias vector; counting a hidden layer's parameters once for the layer below and again for the layer above; and quoting a number that silently includes or excludes something the interviewer did not ask about. Say your convention as you go — "weights plus biases, all layers, nothing else" — and the count becomes checkable rather than a magic number.