In tf.data, what does shuffle(buffer_size) actually do, and why does its position matter?

## The algorithm `tf.data.Dataset.shuffle(buffer_size, seed=None, reshuffle_each_iteration=None)` implements a reservoir-style sliding shuffle: 1. Pull `buffer_size` elements from the upstream dataset into a buffer. 2. Pick an index in the buffer uniformly at random, emit that element. 3. Pull the next upstream element into the vacated slot. 4. Repeat; at the end of the source, drain the buffer randomly. Two consequences fall straight out of this. First, memory cost is `buffer_size` elements held live — for decoded 224x224x3 float32 images that is roughly 600 KB each, so a buffer of 10,000 is about 6 GB of host RAM. Second, the shuffle is *local*: an element can only move forward or backward by roughly the buffer size in the output order. `buffer_size=1` is a no-op. ## Why locality bites Realistic datasets are rarely stored in random order. Records arrive sorted by class, by capture date, by user id, or grouped one class per file. Feeding a nearly-sorted stream into a model produces batches with almost no class diversity, and with batch normalization or any per-batch statistic that is actively harmful: the loss oscillates, gradients point in wildly different directions from batch to batch, and the run looks like a bad learning rate rather than a bad shuffle. The standard mitigation is two-level shuffling. Shuffle the *file list* — `Dataset.list_files(pattern)` shuffles by default, and `TFRecordDataset` shards can be interleaved in random order — then apply an element-level `shuffle` on top. Randomizing which shards you read from gives long-range mixing cheaply, and the element buffer handles local mixing. If you also control how the data is written, writing records in random order once, offline, makes every epoch cheaper forever. ## Ordering: shuffle before batch `ds.shuffle(1000).batch(32)` shuffles examples and then groups them, so batch composition is random and changes every epoch. `ds.batch(32).shuffle(1000)` groups first, so the batches are fixed sets of neighbouring examples and shuffling only permutes the order in which those fixed groups arrive. That is strictly weaker regularization, and if the data was sorted the batches stay single-class no matter how large the shuffle buffer is. The one time batch-then-shuffle is deliberate is when elements are already random and you want to cheaply reorder large pre-built batches. ## Ordering: shuffle before repeat `ds.shuffle(n).repeat()` completes one full pass — every element exactly once — before starting the next, giving clean, countable epochs. `ds.repeat().shuffle(n)` shuffles across the epoch boundary: the buffer can hold elements from pass 1 and pass 2 at the same time, so an element may be seen twice before another is seen once. It smooths the transition and can be marginally faster, but it makes 'one epoch' a fiction, which matters when you evaluate or checkpoint per epoch. If you use `steps_per_epoch` with an infinite dataset, be explicit about which you chose. ## reshuffle_each_iteration and reproducibility `reshuffle_each_iteration` defaults to `True`: each time you iterate the dataset again, you get a different permutation, which is what you want across epochs. Setting it to `False` together with a fixed `seed` gives the same order every pass — useful for a validation set or for a bit-reproducible run. Note that reproducibility also depends on the rest of the pipeline: `num_parallel_calls` with `deterministic` left at its default preserves order, but explicitly setting `deterministic=False`, or setting `tf.data.Options().deterministic = False`, trades order stability for throughput. ## Interactions to watch - **cache before shuffle, not after.** `ds.shuffle(n).cache()` caches one particular shuffled order and replays it forever, silently killing the reshuffle. Put `cache()` upstream of `shuffle`. - **Sharding for multi-worker input** should be applied deliberately relative to shuffle; sharding a shuffled stream by element gives each worker a random subset, sharding files gives each worker whole shards. - **Cost.** The buffer fill happens before the first element is emitted, so a huge buffer adds visible startup latency to every epoch and shows up as a slow first step. ## What a good answer sounds like Describe the buffer mechanically, state the memory cost, admit that it is not a global shuffle, and name the two ordering rules — before `batch`, before `repeat` — plus the file-level shuffle that makes a modest buffer sufficient.