Compute vs Memory Bound

## Reading the symptom Two numbers together tell the story: the accelerator is busy only 30% of the time, and the host CPU is at its limit. That combination is almost never a model problem. It says the training loop spends most of each step waiting for the next batch to be produced and delivered, while the host burns all of its cores producing it. The device's compute ceiling and its bandwidth ceiling are both irrelevant while it has nothing to work on. ## Where the host time goes A typical image pipeline per sample does: read bytes from storage, decode a compressed image, resize or crop to the training resolution, apply augmentations, convert to the numeric layout the model expects, and collate samples into a batch. On 512 x 512 tiles the decode and the resize dominate, and both are CPU-heavy and done *again on every epoch* even though their result is identical each time. Multiply by the batch size and it is entirely ordinary for one host to be unable to feed one accelerator. The delivery leg matters too. Once a batch exists on the host it must be copied to device memory. If that copy is issued synchronously between steps rather than overlapped with the previous step's compute, it adds dead time even when the producers keep up. ## Confirming rather than guessing The cheap decisive test is to remove the pipeline from the measurement: take one batch, keep it resident in device memory, and run training steps on that same batch in a loop. Compare the resulting step time against the real run's step time. - Step time collapses and utilisation rises: the input pipeline is the limit, exactly as the symptom suggested. - Step time barely changes: the device really is busy for that long, and the bottleneck is inside the model — at which point the compute-versus-bandwidth classification of its layers is the next question, not the loader. The second useful measurement is simply how long batch production takes on the host per batch, compared with the model-only step time from the test above. Whichever is larger sets the pace, because the two run in parallel once you overlap them. ## The fixes, roughly in order of payoff 1. **Overlap production with compute.** Produce batch k+1 while batch k is training, keeping a small queue of ready batches. This alone converts a serial `prepare + compute` step into `max(prepare, compute)`. 2. **Parallelise on the host.** Multiple worker processes decoding and augmenting concurrently, sized to the cores actually available. Note the ceiling: if the host is already saturated, more workers only add contention. 3. **Stop paying the decode repeatedly.** Pre-process the dataset once into a compact stored format at the training resolution — already decoded, already resized, written in large sequential records — and read from that. This is usually the single biggest win for image workloads, because it deletes the expensive step rather than parallelising it. 4. **Move work to the device.** Some augmentation and the numeric conversion can run on the accelerator, which has spare capacity by definition in this scenario. 5. **Overlap the transfer.** Issue the host-to-device copy for the next batch while the current step computes, so delivery is not serialised behind compute. 6. **Check storage.** If the tiles come from a remote or slow filesystem in many small files, the host may be blocked on reads rather than on decoding; large sequential reads and local caching fix that variant. ## What not to do Do not start by shrinking the model, lowering its resolution, or reaching for a device with a higher FLOP rating. Every one of those makes the device finish its share sooner and idle *more*, leaving wall-clock time roughly where it was. Do not assume low utilisation always means a small batch either — a bigger batch on a starved run just makes each wait longer. And do not accept a utilisation figure alone as proof: pair it with the fixed-batch timing test, because a device can also show low utilisation when it is blocked on synchronisation or on many tiny operations. ## When lower utilisation is acceptable If the pre-processing genuinely cannot be cached — heavy randomised augmentation that must differ every epoch, or data that is generated on the fly — then some host cost is irreducible, and the engineering question becomes how many host cores per accelerator the workload needs. That is a capacity answer, not a code answer, and stating it that way is what separates a considered response from a guess.