A segmentation network with 60 MB of weights fails on a 4 GB edge device — why?

## The memory a running model needs At inference a model's memory footprint has three parts, and only the first is what people usually quote: 1. **Weights** — parameter count times bytes per parameter. Fixed, easy to compute, and here it is 60 MB. 2. **Activations** — the intermediate feature maps produced by each layer. Their size depends on input resolution, channel counts and batch size, not on the parameter count at all. 3. **Working buffers** — scratch space some operators allocate, plus input and output staging. **Peak** memory is what decides whether the model runs. It is the maximum, over the execution schedule, of the total size of everything that must be live at that moment. A tensor is live from the point it is produced until its last consumer has read it; after that its memory can be reused. So peak is a property of the network's *graph shape and execution order*, not simply the sum of all tensors. ## Why dense prediction at high resolution is the hard case Run the arithmetic. A street-scene image at 1024x2048 has about 2.1 million pixels. A first feature map at full resolution with 64 channels, in 32-bit floats, is `1024 * 2048 * 64 * 4` bytes = **512 MiB** — for one tensor, at batch size one. A convolution needs its input and its output live at the same time, so that stage alone can want a gigabyte. An encoder-decoder with skip connections compounds it. The whole point of a skip is that a high-resolution encoder feature map is handed to the matching decoder stage. That tensor must stay live from the moment the encoder produces it until the decoder consumes it, which is most of the forward pass. With several skip levels, several of the largest tensors in the network are alive simultaneously by design. Concatenation-style merging is worse than addition, because the concatenated result is a new tensor as large as both inputs together, and both inputs must be live to produce it. Meanwhile the weights barely move: convolution kernels are small and reused across every spatial position. That is exactly the regime where **parameter memory and activation memory disagree completely**, and where trusting the 60 MB figure gets a deployment rejected. ## How to diagnose it Walk the graph and compute each tensor's size as `elements * bytes-per-element`, then find the point in the execution order where the live set is largest. That is the number to compare against the device budget — and the budget is not the full 4 GB, since the operating system, other processes and the runtime itself take a share. Confirm on the device: a footprint that fits on paper can still fail because of allocator behaviour or an operator's scratch requirements. ## Levers that actually move peak activation memory - **Lower the input resolution.** Activation memory scales with the spatial product, so halving each dimension cuts it by about four. This is the largest single lever, and it costs accuracy on small structures. - **Tile the input.** Run overlapping patches through the network one at a time and stitch the outputs. Peak memory becomes a function of tile size, which you choose. The price is redundant computation in the overlap and possible seams at tile borders. - **Downsample earlier and narrow the early layers.** The stem, operating at the highest resolution, is usually the most expensive part of the memory profile. Reducing resolution one stage sooner, or holding fewer channels while resolution is high, moves peak substantially. - **Store activations in a lower-precision format.** Halving bytes per element halves activation memory directly, with an accuracy question to validate. - **Restructure so tensors die sooner.** Fewer or shorter skip connections, addition instead of concatenation where the architecture allows, and an execution order that consumes large tensors promptly all shrink the live set. - **Keep batch size at one.** Activation memory scales linearly with batch; on a memory-constrained device, batching is a luxury. Note which levers are absent from that list: quantizing or pruning the *weights* saves at most a fraction of 60 MB, and would not have made a difference to a deficit measured in gigabytes. ## What a strong answer sounds like Separate weight memory from activation memory immediately, define peak as the maximum live set over the execution order, produce one concrete tensor-size calculation to show the scale, explain why encoder-decoder skips keep the largest tensors alive, and then propose levers that act on resolution, tiling, precision and graph shape rather than on the weights. The interviewer is watching for whether you check which memory is binding before optimizing anything.