Deep Learning

An artificial neuron multiplies each input by a learned weight, sums the products, adds one learned bias, and passes that single number through a fixed nonlinear function. Only the weights and the bias are learned.

Input (32, 12288), then (32, 512), then (32, 128), and finally (32, 10). The batch size stays on the leading axis at every layer; only the trailing feature width changes, once per weight matrix.

Each fully-connected layer holds inputs times outputs weights plus one bias per output unit. For 784-128-10 that is 784 x 128 + 128 = 100,480 and 128 x 10 + 10 = 1,290, totalling 101,770 parameters.

Per-label sigmoids fit tasks where one example can carry several labels at once: each output is its own independent probability and the outputs need not sum to one. A softmax makes the classes share a single probability budget, which only fits mutually exclusive classes.

A softmax head emits one score per class and normalizes those scores into probabilities that sum to one. K classes need K units: with fewer, some class can never be predicted; extra units create classes no label ever selects.

Use binary cross-entropy when each output is an independent yes/no decision, so several can be true at once. Use categorical cross-entropy when exactly one of the classes is correct and the outputs must compete for a single unit of probability.

A logit is the raw, unbounded score a network's final layer emits before any squashing function turns it into a probability. Losses take logits so the squashing and the logarithm are computed together, which avoids overflow and log-of-zero.

Accuracy counts correct predictions, so it is a step function of the model's scores. Nudging a weight usually changes nothing at all, leaving a gradient of zero almost everywhere and no slope for gradient descent to follow.

ReLU computes max(0, z): positive pre-activations pass through unchanged and negative ones become zero. Its derivative is exactly 1 wherever the unit is active, so gradients flow back unshrunk, and evaluating it costs a single comparison.

Per-class weighting multiplies each example's loss by a factor set by its class, so errors on rare intents contribute more gradient. It rebalances the effective class prior the model fits. It adds no new information about rare classes.

Training runs a backward pass, and each op's local gradient is a function of the tensors that op actually saw. Inference never runs backward, so an intermediate can be released the moment the next op has consumed it.

A max passes the incoming gradient straight to whichever input won and gives exactly zero to the loser. For ReLU that is a 0/1 mask: positive inputs pass the upstream gradient through unchanged, non-positive ones cut it to zero.

Max pooling sends a window's entire upstream gradient to the one input that was the maximum and exactly zero to the others. Average pooling spreads it evenly, so each input in a k-by-k window receives one k-squared-th of it.

A reused tensor appears in several terms of the chain rule, so its gradient is the sum of one contribution per consumer. The reverse sweep adds each contribution into a single buffer as it reaches that consumer.

The forward pass records a directed acyclic graph whose nodes are ops and whose edges are the tensors between them. The backward sweep replays it in reverse topological order: a node is visited only after every op that consumed its output.

A large step keeps stochastic gradient descent bouncing around a minimum instead of settling in it. Shrinking the step later in training lets the noisy gradient estimates average out, so the parameters settle and the loss stops oscillating.

Momentum keeps a running average of past gradients, the velocity, and steps along it rather than along the raw gradient. Consistent directions build up and move faster; components that flip sign each step cancel, so the path stops zig-zagging.

AdaGrad divides the global learning rate by the square root of each parameter's own accumulated squared gradients. Consistently large gradients get small steps, quiet parameters keep large ones, and because the sum only grows, every step shrinks over time.

Both moving averages start at zero, so early on they read too small — the squared-gradient average far more so. Dividing each by one minus its decay rate to the step count removes that startup bias.

Adam keeps two exponential moving averages per parameter: one of the gradient, one of the squared gradient. The update is the first divided by the square root of the second, so each parameter gets its own step size.

In training, batch normalization standardizes each channel with the current mini-batch's mean and variance. In evaluation it uses running averages collected during training, so predictions become deterministic and stop depending on which other samples share the batch.

Random label-preserving transforms show the network a different version of each image every epoch, so it cannot memorise exact pixels. This effectively enlarges the training set and bakes in invariances such as small shifts and rotations, cutting variance.

Dropout zeroes each hidden unit at random on every training step, so no unit can rely on a particular other unit being present. Training therefore fits many thinned, weight-sharing subnetworks whose behaviour is averaged at evaluation.

LayerNorm computes a mean and variance across the feature dimensions of each sample on its own, then rescales with a learned per-feature gain and bias. No other sample enters the statistic, so batch size and batch composition cannot change the output.

Weight decay pulls every decayed weight slightly toward zero on each update, on top of the gradient step. The result is a smaller-norm solution: the network fits with less extreme weights, which usually improves held-out performance.

A 1x1 convolution mixes channels at one spatial position: each output channel is a learned linear combination of all input channels at that pixel. It changes channel count cheaply, leaving height and width untouched.

Semantic segmentation labels each pixel with a class but no identity, so touching objects merge. Instance segmentation returns one mask per countable object. Panoptic gives every pixel exactly one class and, for countable classes, an instance id.

IoU is the overlap area of a predicted and a ground-truth box divided by the area they jointly cover. A prediction counts as correct when its IoU with an unclaimed ground-truth box of the same class clears a threshold, commonly 0.5.

A 3x3 filter over a 4-channel input is a 3x3x4 weight block plus one scalar bias. The kernel always spans every input channel, sums across them, and produces a single output channel.

A convolution reuses one small kernel at every position, so it learns a feature detector once instead of relearning it at each pixel. That cuts the parameter count enormously and lets the same feature be found anywhere in the input.

At each output step the decoder scores its current state against every encoder state, softmaxes those scores over source positions into weights summing to one, then averages the encoder states with those weights — that average is the context vector.

Backpropagation through time unrolls the recurrent loop into a chain with one copy of the cell per time step, then runs ordinary backpropagation over that finite graph. All copies share one set of weights.

Only its final hidden state: one fixed-size vector, plus the cell state if the encoder is an LSTM. That single tensor initialises the decoder, and nothing else crosses between the two stacks.

A batch must be one rectangular array, so shorter sequences are padded out to the longest length. The mask marks which steps are real, keeping pad steps out of the loss, out of pooling, and out of the state you read.

A vanilla RNN computes h_t = tanh(W_xh x_t + W_hh h_(t-1) + b). Since h_(t-1) was built the same way from h_(t-2), the state at step t is a fixed-size summary of every input so far.

The discriminator maximizes the log-probability of labelling real data real and generated data fake. The generator minimizes that same quantity, pushing the discriminator toward calling its samples real. One shared value function, optimized in opposite directions.

Mode collapse is when a GAN generator maps many different noise vectors to a few nearly identical outputs, covering only part of the real data. You spot it by decoding a fixed batch of noise vectors and seeing the samples repeat.

An autoencoder is trained to copy its input, so a wide enough code can simply learn the identity map: near-zero loss, nothing learned. A code narrower than the input makes exact copying impossible and forces the encoder to keep only rebuildable structure.

Scoring against the clean original means copying the input no longer wins. The network has to use structure shared across the data to remove the corruption, so the code ends up describing that structure instead of the input itself.

Only the discriminator can create pressure to obey the label. If it judges images alone, any realistic image passes, so the generator's cheapest strategy is to ignore its label input and reproduce the overall data distribution.

The batches are being read in the stored file order without shuffling. If that order is sorted by class, every batch holds one class, the loss swings as the class changes, and the same swings repeat each epoch.

A learning rate that is too high drops the loss fast, then leaves it oscillating inside a band or diverging; one that is too low gives a smooth, near-linear descent still falling at the last epoch.

All units in a layer then compute the same output and receive the same gradient, so they update identically and stay duplicates forever. The layer has the power of a single unit. Random asymmetric values break that tie.

Grad-CAM averages the gradients of one class score over each feature map of a chosen convolutional layer to get per-channel weights, sums the feature maps with those weights, and applies ReLU. The coarse result is upsampled onto the image.

Per layer, log the fraction of units that output zero for every example in a fixed probe batch, plus the fraction of tanh or sigmoid outputs past 0.99 in magnitude. Both are per-unit across the batch, not per example.

Catastrophic forgetting is the sharp drop in a network's performance on its original task after it is trained on a new one. Fine-tuning minimises only the new task's loss, so the shared weights drift away from the old solution.

Covariate shift means the input distribution moves from source to target while the rule mapping input to label is unchanged. The network was fitted where source data lived, so target inputs land where its decision boundary was never pinned down.

Pretraining already puts the weights in a good region, so fine-tuning only needs small corrections. A large step moves every weight far enough to destroy the learned features, and a small target dataset cannot rebuild them.

Discard the old output layer and attach a new, randomly initialised one with seven outputs, keeping the layers beneath it. The old head maps to the wrong label set, so none of its weights are reusable.

A classifier can only recognise identities it was trained on, so every new person means retraining. An embedding model learns a distance where same-person pairs land close together, so a new identity is enrolled by storing one vector.

Activation checkpointing trades compute for memory. Instead of keeping every layer's forward outputs alive until the backward pass, it keeps only a few segment boundaries and recomputes the rest on demand, costing roughly one extra forward pass per step.

The input pipeline, not the model. The GPU idles waiting for batches while the host decodes and resizes images. Fixes are overlapping data preparation with compute, adding host-side parallelism, and pre-processing tiles into a cheaper stored form.

A 16-bit float's 5 exponent bits bottom out near 6e-5 for normal values and near 6e-8 once subnormals run out. A gradient smaller than that has no representation, so it stores as exactly zero and that weight stops moving.

Mixed-precision training runs the forward and backward passes in a 16-bit float while keeping a single-precision copy of the weights. Wins come from moving half the bytes and from matrix units that multiply 16-bit inputs at much higher throughput.

Weights, gradients and optimizer state are sized by the parameter count, not by batch size. Activations, the per-layer outputs held for the backward pass, exist once per sample, so halving the batch roughly halves that share.

With no non-zero reward, every return and every value target is zero, so the policy gradient and the TD error are exactly zero. Nothing is learned slowly; there is no signal at all until the agent stumbles onto a reward.

Consecutive transitions are almost identical, so learning from them in order gives correlated, unstable updates. A replay buffer stores past transitions and samples shuffled batches from them, breaking that correlation and letting each transition be reused many times.

The critic learns a state-value estimate V(s). The actor is updated with the advantage - reward plus gamma times the next state's value, minus the current state's value - a signal centred on zero that reinforces only actions better than average.

PPO maximises min(r*A, clip(r, 1-eps, 1+eps)*A), where r is the new-over-old action probability ratio and A the advantage. Once r passes the bound in the direction the advantage favours, the objective flattens and that sample stops pushing.

DDPG's critic scores a state-action pair and is differentiable in the action, so the actor is trained by pushing its output uphill on that score: the gradient is dQ/da evaluated at the actor's own action, chained with the actor's parameter gradient.

The teacher's wrong-class probabilities encode which classes resemble each other, structure a one-hot label throws away. Each example then supplies a whole similarity-ranked distribution rather than a single index, giving the student a richer and lower-variance training signal.

FLOPs counts arithmetic only. Latency also pays for moving data, for per-layer fixed costs, and for how well each operation uses the device's parallel units, so a low-FLOP network built from poorly-supported operations can lose on the stopwatch.

One 4096x4096 matrix holds about 16.8M weights; replacing it with a 4096x256 matrix times a 256x4096 matrix holds about 2.1M, an 8x cut. The saving exists only while the rank stays below half the layer width.

Prune one layer at a time to a fixed ratio, leave every other layer dense, and record the held-out drop. Repeating over several ratios gives a per-layer sensitivity curve that ranks which layers absorb cuts.

Unstructured magnitude pruning zeroes the individual weights with the smallest absolute values, wherever they sit in the tensor, using |w| as a cheap proxy for importance. Retraining with those weights held at zero lets the survivors re-fit what was lost.

A raw neighbour sum scales with how many neighbours a node has, so hubs produce huge activations and ordinary nodes tiny ones. Dividing each edge's contribution by sqrt(d_i * d_j) keeps every node's aggregate on a comparable scale.

A graph reaches the model as a node feature matrix — one row of features per node — plus an edge list of connected node id pairs, and optionally an edge feature matrix. Node ids are addresses, not values.

Transductive training assumes every node you will ever score was already in the graph when the model was fitted. Inductive training learns a function of node features and neighbourhoods, so a node added later can be scored without refitting.

One round has three steps: each neighbour builds a message from its state and the connecting edge; the node pools those messages with an order-free aggregator; a learned update then combines the pooled vector with the node's own previous state.

Permutation invariance means relabelling the nodes leaves the output unchanged, which is what a single graph-level prediction requires. Equivariance means the output moves with the relabelling: per-node predictions keep their values but come back in the new node order.