Neural Network Basics

Input (32, 12288), then (32, 512), then (32, 128), and finally (32, 10). The batch size stays on the leading axis at every layer; only the trailing feature width changes, once per weight matrix.

Each fully-connected layer holds inputs times outputs weights plus one bias per output unit. For 784-128-10 that is 784 x 128 + 128 = 100,480 and 128 x 10 + 10 = 1,290, totalling 101,770 parameters.

Per-label sigmoids fit tasks where one example can carry several labels at once: each output is its own independent probability and the outputs need not sum to one. A softmax makes the classes share a single probability budget, which only fits mutually exclusive classes.

A softmax head emits one score per class and normalizes those scores into probabilities that sum to one. K classes need K units: with fewer, some class can never be predicted; extra units create classes no label ever selects.

An epoch is one full pass over the data. At batch size 128, 50,000 examples give 390 full batches plus a ragged batch of 80, so 391 iterations per epoch, one update each, and 1,173 updates over 3 epochs.

Composing affine maps yields another affine map: W2(W1x + b1) + b2 equals (W2W1)x + (W2b1 + b2). Depth with no nonlinearity between the layers buys more parameters and a different optimisation path, never a larger family of functions.

Depth composes. Each layer reuses every feature the layer below it built, so complexity compounds with depth, while a shallow net must build each feature independently from the raw input. For some functions that costs exponentially more units.

An embedding table is a weight matrix with one row per id. A one-hot vector times that matrix selects exactly that row and zeroes the rest, so the lookup is the same linear layer computed by indexing instead of multiplying.

The forward pass turns the 64 inputs into activations and one loss number, the backward pass fills a gradient for every parameter, and the update writes new parameter values. Only parameters, optimizer state and gradient buffers change; the batch, targets and architecture do not.

It is an existence result: for any continuous target on a closed bounded region and any error tolerance, some one-hidden-layer network with a non-polynomial activation stays inside that tolerance. The width needed is unbounded and unspecified.

There is no row for it, so the lookup fails on an out-of-range index unless the vocabulary reserves an unknown row. That row only helps if rare ids were routed to it during training.

Each hidden unit thresholds the same two flags at a different level: one fires when at least one flag is on, the other only when both are. The output unit fires when the first fires and the second does not.

Broadcasting aligns from the right, so a (256,) bias adds one value per unit to every row, while an (8, 1) bias adds one value per row across all units. Both run; only the first is a real bias.

Cut the second hidden width. The 1024-to-1024 matrix holds about 1.05M of the 1.35M parameters, so taking that layer to 512 units drops the total to roughly 695,000. The 40-input layer holds almost nothing.

A plain linear output head is unbounded by construction, so negative predictions are expected, not a bug. Fix it in the head: predict in log space and exponentiate, pass the output through a positive-valued transform, or accept the negatives and clip only at serving.

Yes, for a top-1 or top-k label. Softmax exponentiates each score and divides them all by the same positive total, so it never changes their order. You need the normalized values only when something downstream consumes the number itself.

Parameter count bounds what a network could fit, not what training selects. Gradient descent picks a biased, simple subset of the functions that fit, and the architecture's priors narrow it further. Only held-out data settles it.

Tune one threshold per tag on a held-out set rather than applying 0.5 everywhere. Tags differ in base rate, score distribution and the cost of a mistake, so 'live recording' may fire best at 0.2 while 'acoustic guitar' needs 0.7.

The head gains one weight row and one bias for the new class; the trunk keeps its shape. All 21 classes then share one probability budget, so old outputs shift and old thresholds no longer transfer.

Dropout stays active, so every validation number is a noisy sample of a randomly thinned network, and normalization layers use the validation batch's own statistics and refresh their stored ones. The metric becomes irreproducible, batch-order dependent, and validation data leaks into the model.

The guarantee is stated on a closed bounded region chosen before the weights are picked, and closeness is claimed only on that region. Outside it the network is unconstrained and simply follows whatever its architecture does asymptotically.

Size it from observations per id and a stated memory budget, not from cardinality. Two million rows at width 64 is 128 million parameters, so width is the model's main size decision.

The theorem proves suitable weights exist; it never says any procedure finds them, bounds how many units they need, or says how much data pins them down. Density is a weak property that recommends no architecture on its own.

Treat a name for a hidden unit as a hypothesis and test it: check the unit against labelled held-out conditions, and measure what ablating it costs. Hidden features are learned and distributed, not guaranteed to match human concepts.

A product only checks the feature axis, so a transposed batch can still conform and run. Confirm the layout instead: vary the batch size and see which axis moves, and perturb one example to check only its own output changes.

Let the binding constraint decide. Depth buys parameter efficiency but adds sequential latency and harder optimization; width parallelizes well but costs full fan-in and fan-out per unit. Settle it with small matched-budget runs.

Share a trunk when the two tasks need the same perception features and you want one forward pass; keep them separate when they need independent release cadences. Sharing buys compute and transfer, and costs you a coupled artefact where retraining for one task can regress the other.

Count the run in optimizer steps and tokens consumed, not epochs — an epoch that happens once carries no progress information. Fix a step budget up front, hang evaluation and checkpointing on step intervals, and compare runs at equal tokens seen.