A 90-percent unstructured-sparse model runs no faster than the dense one on CPU — why?
answer
- the tensor never changed shape
- a zero costs what a nonzero costs
- every saved value buys you an index
- gathers waste whole cache lines
- regularity is what hardware can exploit
basics
~20 sBecause the zeros are scattered. Dense matrix kernels multiply every element regardless of value, and a sparse kernel that skips zeros must load an index for each surviving value, whose irregular memory access usually costs more than the multiplies it saves.
solid answer
~50 sZeroing a weight does not remove it from the computation. The tensor keeps its shape, and a dense matrix-multiply kernel walks contiguous tiles multiplying every entry — a zero costs exactly what a nonzero costs. To actually skip work you must switch to a sparse representation, which stores each surviving value alongside an index, and that trade is worse than it looks: you save the multiplies but you now load index metadata, your reads become irregular gathers instead of contiguous vector loads, and a fetched cache line may contain a single useful number. Well-tuned dense kernels start far ahead, so the crossover sparsity at which a general sparse kernel wins is very high and shape-dependent. What 90 percent scattered sparsity does buy is a much smaller checkpoint and, with a sparse format, a smaller memory footprint. The number you cut is the parameter count; latency is a different number, and unstructured zeros do not connect them.
go deeper
Remember the one fact that explains everything downstream: pruning zeroes values but leaves the tensor the same shape, so the arithmetic performed is unchanged unless something else changes too.
Be ready to explain the sparse-format trade in mechanical terms — an index per stored value, irregular gathers instead of contiguous loads, and a dense kernel that is already near peak throughput.
Demonstrate that you benchmark rather than infer: crossover sparsity per layer shape and batch size, confirming a sparse kernel is engaged, and noticing when unconverted layers dominate the remaining time.
Own the requirement upstream of the technique. Decide whether the constraint is latency, memory or distribution size before anyone spends accuracy, and set the acceptance criterion as a measurement on target hardware.
## The mismatch After pruning, the weight tensor has the same shape it always had. Nothing about a zero makes hardware skip it: a fused multiply-accumulate on `0 * x` takes exactly as long as one on `0.37 * x`. A dense matrix-multiply kernel is written to stream contiguous tiles of weights and activations through vector units at a fixed rate, and it has no notion of value-dependent work. So the default outcome of unstructured pruning is: **same latency, same arithmetic, smaller numbers.** To turn zeros into saved time you must change the *representation*, and that is where the second problem starts. ## What a sparse representation costs A compressed sparse format stores, for each nonzero, both the value and its coordinate — typically a column index per nonzero plus a small offset array per row. At 90 percent sparsity you store one tenth of the values, but you also store one index per stored value. If a weight is 4 bytes and an index is 4 bytes, your 10x reduction in values becomes roughly a 5x reduction in bytes. Memory saving is real, just smaller than the sparsity number suggests. The latency picture is worse than the byte picture: - **Irregular access.** Each nonzero's index selects an arbitrary activation. Instead of loading a contiguous vector of inputs you gather scattered ones, which is slower per element and often cannot be vectorised as cleanly. - **Cache lines wasted.** Memory arrives in fixed-size lines. A gather that needs one value out of a line pays for the whole line, so the effective bandwidth you get is a fraction of the peak. - **Control overhead.** Row lengths vary, so loop bounds are data-dependent and the tight, fully unrolled inner loop of a dense kernel is not available. - **A hostile starting line.** Dense matrix multiply is the single most heavily optimised routine in numerical computing. A sparse kernel is not competing with naive code; it is competing with something running near the machine's arithmetic peak. Add it up and the crossover — the sparsity at which a general unstructured sparse kernel beats the dense one — sits very high, commonly quoted in the region where the great majority of weights are gone, and it moves with layer shape, batch size and hardware. For a matrix that is small or a batch that is one, it may not exist at all. This is why 90 percent sparse frequently measures as *slower* than dense, not merely equal. ## What you actually gained Be precise about it, because it is not nothing: - **Checkpoint size.** A tensor that is 90 percent zeros compresses extremely well with ordinary compression, so distribution and storage costs drop sharply. - **Resident memory**, if you commit to a sparse format at load time and accept the index overhead. - **A starting point.** A model that tolerates high sparsity is a model with slack, and that slack can be re-spent on a pattern hardware *can* exploit. What you did not gain is the thing the stakeholder asked for, if the ask was latency. ## How to get speed instead The rule is: **the sparsity pattern must be one the hardware can recognise without data-dependent work.** The scattered-nonzero pattern is maximally free and maximally unhelpful. Constrained patterns give up freedom and get regularity back — the semi-structured N:M family being the standard example, where a fixed number of nonzeros per fixed-size group produces a compressed operand of known size that a sparse matrix-multiply unit can consume directly. Coarser regular patterns exist along the same axis. And whichever you choose, the acceptance test is a measurement, not a percentage: latency on the target hardware, at the real batch and sequence shape, against the dense baseline. Two failure modes show up constantly in that measurement. First, the sparse kernel was never engaged at all — the model is running dense tensors that happen to contain zeros, plus mask overhead, which is strictly slower than not pruning. Second, only some layers converted, and the layers left dense dominate the remaining time, so a large speedup on part of the network barely moves the total. ## The interview answer in one line Sparsity is a property of the *values*; latency is a property of the *memory layout and the kernel*. Unstructured magnitude pruning changes the first and leaves the second alone.
- If not latency, what does 90 percent unstructured sparsity actually buy you?A far smaller checkpoint, because a mostly-zero tensor compresses well, and a smaller resident footprint if you commit to a sparse format and accept the index overhead. It also tells you the model had slack. Those are real wins when distribution size or memory is the binding constraint — but if the requirement was latency, you spent accuracy on the wrong axis and should re-spend the slack on a hardware-friendly pattern.
- Your sparse model benchmarks slower than the dense one. What do you check first?Whether a sparse kernel is running at all. The common outcome is that masks were applied to ordinary dense tensors, so you pay full dense arithmetic plus the mask, which is strictly worse than not pruning. After that, check whether the runtime silently fell back to dense for unsupported shapes, and whether only a few layers converted while the dense remainder dominates the time.
- Why does batch size change whether sparse beats dense for a layer?A larger batch amortises the index loading and the irregular gathers over more arithmetic per weight fetched, which moves the crossover in the sparse kernel's favour. At batch one, the layer is dominated by moving weights and indices rather than by multiplying, so the index overhead lands directly on the critical path. The crossover is a property of the shape, not of the model.
Crossing names off a seating chart does not shrink the room. Until you rearrange the tables, the waiter still walks past every empty chair.
saying these in an interview costs you the question
- Assumes hardware skips zeros automatically
- Quotes parameter reduction as if it were a speedup
- Thinks any sparse kernel beats dense given enough zeros
- Ignores the index metadata a sparse format must store
- Never measures latency on the target hardware