Activations and Loss Functions

A logit is the raw, unbounded score a network's final layer emits before any squashing function turns it into a probability. Losses take logits so the squashing and the logarithm are computed together, which avoids overflow and log-of-zero.

Accuracy counts correct predictions, so it is a step function of the model's scores. Nudging a weight usually changes nothing at all, leaving a gradient of zero almost everywhere and no slope for gradient descent to follow.

ReLU computes max(0, z): positive pre-activations pass through unchanged and negative ones become zero. Its derivative is exactly 1 wherever the unit is active, so gradients flow back unshrunk, and evaluating it costs a single comparison.

Per-class weighting multiplies each example's loss by a factor set by its class, so errors on rare intents contribute more gradient. It rebalances the effective class prior the model fits. It adds no new information about rare classes.

A sigmoid unit's gradient collapses toward zero. The curve flattens at both extremes, so its derivative - at most 0.25, at the origin - becomes almost nothing in the tails, and the layers below stop learning.

ReLU hard-zeroes every negative input. GELU and SiLU multiply the input by a smooth gate between 0 and 1 - a Gaussian CDF for GELU, a sigmoid for SiLU - so small negatives survive and the derivative stays continuous.

The softmax derivative cancels against the derivative of the log instead of multiplying into it. Differentiating -log of the correct class's probability through the softmax leaves predicted probability minus one-hot target, one subtraction per logit, bounded between -1 and 1.

Subtracting the row maximum caps every exponent at zero, so the exponential cannot overflow. Adding it back outside the logarithm leaves the value provably identical: factoring it out of the sum turns it into an added constant.

Set weights so each term contributes comparable gradient, not so the raw numbers match. A depth error in metres dwarfs a cross-entropy in nats, so it dominates by accident of units. Tune the weights on per-task validation metrics, never on total loss.

Aleatoric uncertainty is noise inherent in the data — a noisy sensor, identical inputs with different labels — and more data will not remove it. Epistemic uncertainty is the model's ignorance of regions it barely saw, and more data shrinks it.

Softmax normalizes scores across the trained classes, so its output ranks them and must sum to one — never evidence that the input belongs to any. Nothing in training penalised confident nonsense, and far from the data the gaps can grow.

Squared error's per-example gradient grows linearly with the residual, so an example whose error is 50 times larger contributes about 50 times the gradient. Batch averaging divides everyone equally, so it never dilutes that imbalance.

A ReLU unit dies when its pre-activation is negative for every input in the data. It then outputs zero, its local derivative is zero, and so the loss sends exactly zero gradient to its weights and bias — nothing ever moves them again.

Focal loss multiplies cross-entropy by (1 - p_t)^gamma. At gamma = 2 and p_t = 0.99 that factor is 0.0001, so the example contributes one ten-thousandth of its cross-entropy and stops steering training.

Tanh is zero-centred and steeper. Its outputs span (-1, 1) rather than (0, 1), so the next layer's weight gradients are not all forced to share a sign, and its peak derivative is 1.0 against sigmoid's 0.25.

Squared error keeps the softmax Jacobian's probability factor instead of cancelling it, so an example whose correct class sits at probability 0.01 sends almost no signal. The composed objective is also non-convex, unlike cross-entropy.

Hinge loss demands a cushion, not just a correct sign: with max(0, 1 - y*s), an example scored 0.9 toward the right class still pays 0.1, and only past 1.0 does its gradient hit zero. Cross-entropy never stops pushing.

An auxiliary head attached partway up gives the early layers a short gradient path and their own training signal, which eases optimisation of a deep stack. Its coefficient is annealed toward zero so the real head governs the final model.

That the target is Gaussian around the network's output with a constant, input-independent variance. Minimising squared error is exactly maximising that Gaussian likelihood, which is why the trained output estimates the conditional mean of the target.

Monotonicity is not required for training or for function approximation. SiLU dips to about -0.278 near x = -1.278 and then rises; its derivative is continuous and its output is bounded below, so gradient descent handles it fine.

The chain materializes a probability first, and that intermediate destroys information: a tail value rounds to exactly zero, so its logarithm is negative infinity. Fusing computes the log-probability as a subtraction that never forms the probability at all.

Tune the threshold first: train with cross-entropy, then pick the operating point on a held-out split. A differentiable soft-F1 loss, built from summed probabilities instead of hard counts, is the fallback when no fixed threshold works.

Sample several plausible models and measure their disagreement on the input. Keeping dropout active for thirty stochastic forward passes is the cheap option; five independently seeded networks are the stronger one. The spread, not the average, is the signal.

Leaky ReLU gives the negative branch a small fixed slope so units cannot go permanently silent. PReLU learns that slope, usually one per channel, at the cost of extra parameters. ELU saturates toward a negative bound, centring activations but paying for an exponential.

Both target the same rebalanced objective and give the same expected gradient. They differ in gradient variance, mini-batch composition, what one epoch means, compute per pass, and how badly the rare rows get memorised.

The pre-activations are far too large and positive, because the input features were fed in unscaled. Decibel-scale loudness values in the tens, summed over many channels, push every unit deep into the sigmoid's flat tail.

Only if measurement says the activation is genuinely costing you. GELU evaluates a transcendental per element where ReLU does a comparison, and on a small backbone with large feature maps that can be a real share of frame time.

Negative cosine between two task gradients means an update helping one hurts the other, and no choice of weights removes that. Name the task the product is judged on, fix the degradation you will accept elsewhere, then pick a mechanism to enforce it.

Decide what the spikes are before touching the loss. Corruption belongs in the data pipeline, where the fix is auditable. A robust loss like Huber stabilises training but shifts the statistic you estimate, so it under-predicts spikes that turn out to be real.