Model Efficiency and Compression

FLOPs counts arithmetic only. Latency also pays for moving data, for per-layer fixed costs, and for how well each operation uses the device's parallel units, so a low-FLOP network built from poorly-supported operations can lose on the stopwatch.

One 4096x4096 matrix holds about 16.8M weights; replacing it with a 4096x256 matrix times a 256x4096 matrix holds about 2.1M, an 8x cut. The saving exists only while the rank stays below half the layer width.

Prune one layer at a time to a fixed ratio, leave every other layer dense, and record the held-out drop. Repeating over several ratios gives a per-layer sensitivity curve that ranks which layers absorb cuts.

Unstructured magnitude pruning zeroes the individual weights with the smallest absolute values, wherever they sit in the tensor, using |w| as a cheap proxy for importance. Retraining with those weights held at zero lets the survivors re-fit what was lost.

A fake-quantization node clips a tensor, rounds it onto the low-precision grid, then scales it back to floating point. Values stay float but carry real rounding and clipping error, so the network trains against the error deployment imposes.

Deleting whole filters leaves a smaller dense layer, so every multiply shrinks into a shape the hardware already runs fast, with no special kernel. Fewer operations and fewer bytes moved — but accuracy falls faster per removed parameter.

Because the zeros are scattered. Dense matrix kernels multiply every element regardless of value, and a sparse kernel that skips zeros must load an index for each surviving value, whose irregular memory access usually costs more than the multiplies it saves.

An embedding table is read, not multiplied. A request looks up a handful of rows, so the table dominates model size and memory while contributing almost no arithmetic; the small dense layers on top do nearly all the compute.

Because the block's middle operator is cheap per channel, so extra width there costs little while giving the nonlinearity room to work. The expensive parts and the residual stay on the narrow ends, which keeps parameters and stored activations small.

Compound scaling raises depth, width and input resolution together as fixed powers of one coefficient, so extra compute is spread over all three dimensions instead of poured into one, which saturates on its own.

Output-only distillation supervises just the final layer, giving a deep student little guidance on its internal representation. A loss on intermediate feature maps adds per-stage supervision, which helps thin, deep students that are otherwise hard to optimise.

Dividing logits by a temperature above 1 flattens the teacher's softmax output, lifting near-zero wrong-class probabilities into a range that actually influences the student's gradient. Teacher and student share that temperature during training; the deployed student uses T = 1.

Both sit at the network's boundary, where an error has no later layer to correct it, and the input stem holds a negligible share of the parameters. Exempting them costs almost no budget while protecting real accuracy.

Ramp gradually when the target sparsity is high enough that a single cut would remove more than the network can recover from. One-shot pruning of the finished model plus a fine-tune is cheaper and adequate at modest sparsity.

Rounding is a staircase: its derivative is zero almost everywhere, so exact backpropagation would send zero gradient to every quantized weight. The surrogate pretends the rounding step is the identity and passes the incoming gradient straight back.

Filter norm ranks channels by the L1 or L2 magnitude of their weights: cheap and data-free, but it assumes magnitude means influence. Activation and first-order Taylor scores use a calibration batch to estimate how much the loss moves when a channel goes.

Peak activation memory, not weight memory, is the binding cost. At high input resolution the intermediate feature maps that must be live at once run to hundreds of megabytes each, and an encoder-decoder holds high-resolution skip tensors alive across the whole decoder.

By how fast the singular values decay. A steep decay means a few directions carry the map, so a low-rank replacement loses little; a flat, well-conditioned spectrum means every direction matters and factorizing at any useful rank will cost accuracy.

A neural architecture search needs three pieces: the space of operations and connections a candidate may use, an objective that scores it, and a strategy that explores. For an on-device target, score latency measured on that device.

A very large teacher represents a function the small student cannot fit - often not even on training data. Distilled accuracy tracks the teacher-student gap rather than teacher accuracy, so a mid-sized teacher often transfers better.

The instant drop mixes real information loss with recoverable disturbance such as stale normalization statistics. Layers that crater immediately can return to baseline after a short fine-tune, so only the post-recovery ranking predicts the shipped model.

Every contiguous group of four weights along the reduction dimension may keep at most two nonzeros, fixing sparsity at exactly 50 percent. The cost is local selection: within an important group you must drop two weights even if all four matter.

Deleting a filter also removes its bias, the following normalization layer's scale, shift and running statistics for that channel, and the matching input slice of every consumer. Layers joined by a residual sum must drop identical indices.

Demand a from-scratch run of the same architecture with matched epochs, augmentation, tuning budget and seeds, plus a simple-regulariser control. Distillation adds training compute and a smoothing effect, so an unmatched baseline credits ordinary training gains to the teacher.

When the converted model misses a product threshold on the real task metric, cheaper fixes are exhausted, and the deployment target is stable enough that a second training pipeline is re-run rarely. A weekly refresh makes it a bad trade.

State the conditions, then measure under load. Fix device, input shape, precision and batch size; discard warm-up iterations; run long enough to expose thermal throttling; and report a median and a 95th percentile rather than a mean or a best run.

Nothing is compressed, so the gain cannot come from a smaller model. Training against a trained network's output distribution replaces one-hot labels with smoother, per-example targets that regularise and reweight examples - a training-signal effect.

The hard-label cross-entropy term is the only path by which a wrong label reaches the student, so lower its weight and lean on the teacher — but only if the teacher was not itself trained to convergence on the same corrupted labels.

Whenever a layer's activations have a long outlier tail. A min-max range stretches to the largest value ever seen, so ordinary activations collapse into a few levels; a trainable threshold clips the tail and buys back resolution.