Why is a 64-unit GRU slow at streaming inference despite its tiny parameter count?

## Where the time actually goes Count the arithmetic first. A gated recurrent cell with hidden size `H = 64` and input size `D = 40` has three weight blocks, each holding an input matrix (`H` by `D`), a recurrent matrix (`H` by `H`) and a bias. That is roughly `3 * (64*40 + 64*64 + 64)` weights, on the order of twenty thousand numbers, and one step performs about that many multiply-adds. A modern accelerator will do that in less time than it takes to start the operation. Now count the steps. A one-second utterance at a 10-millisecond frame hop is 100 frames; a ten-second one at a 10-millisecond hop is 1,000. Every one of those steps is *dependent*: step `t` needs `h_(t-1)`, which does not exist until step `t-1` has finished. So the wall-clock cost is 1,000 times the cost of one step, and the cost of one step is not the arithmetic — it is the fixed overhead. ## Why the per-step cost is overhead, not arithmetic At batch size one, each gate block's recurrent part is a matrix times a *vector*. That operation loads every weight from memory and uses each of them exactly once, so its arithmetic intensity — operations performed per byte of memory read — is about as low as it can be. Hardware designed for large matrix-times-matrix work, where each loaded weight is reused across many rows of the batch, sits mostly idle. On top of that sits the fixed per-step cost of dispatching and synchronising the work, which for an operation this small is often larger than the operation itself. Two consequences follow, and both are counter-intuitive. **Shrinking the cell barely helps latency.** Halving the hidden size cuts arithmetic by roughly four times, but if the step was never arithmetic-bound the measured latency moves very little. The saving shows up in footprint and in bytes read per step, not in wall-clock, until the cell is large enough for the matrix work to dominate. **Throughput and latency come apart.** The same model can have excellent throughput — many independent streams processed at once — while each individual stream still takes the same wall-clock time, because the streams are batched into one wider operation per step but the step count per stream is unchanged. ## The levers that do work - **Batch across concurrent streams.** If a server handles many utterances at once, batching them turns each step's matrix-vector products into one matrix-matrix product, raising arithmetic intensity and amortising the per-step overhead across all streams. This is the single largest win on a server, and it does nothing for a single-user device. - **Fuse the gate blocks.** The update gate, the reset gate and the candidate's input term can be computed as one concatenated matrix multiply per step instead of three, cutting the number of dispatched operations threefold. The candidate's recurrent term still needs the reset gate applied first, so it stays separate. - **Precompute the input-side projections.** The terms `W_z x_t`, `W_r x_t` and `W_h x_t` depend only on the input at step `t`, not on the hidden state. If a whole buffer of frames is already available, all of those can be computed at once, in parallel, before the loop begins. Only the recurrent terms involving `h_(t-1)` are strictly ordered. This shrinks the sequential part of each step but does not remove it. - **Reduce the step count.** A larger frame hop, a strided or pooled input, or predicting once per several frames all cut `T` directly, which is the quantity latency is proportional to. This is usually the biggest lever available on a device, and it trades against temporal resolution. - **Keep the state and weights resident.** If weights must be re-fetched from slow memory each step, the memory-bound step gets worse. Keeping them in fast memory close to the compute makes the low arithmetic intensity cheaper to live with. ## How to diagnose it Measure latency against sequence length and against hidden size separately. If latency is close to linear in the number of steps and nearly flat in hidden size, you are overhead- and dependency-bound, and the fixes are step-count and batching fixes, not model-size fixes. If it climbs steeply with hidden size, the matrix work has become the cost and shrinking the cell will genuinely pay. ## The honest summary The parameter count of a small recurrent cell is a very poor predictor of its streaming latency. What predicts latency is the number of strictly ordered steps multiplied by a per-step cost that is dominated by overhead and memory traffic. Any optimisation that does not reduce one of those two numbers will not show up on the clock.