skip to content

When you call shuffle() on a streaming IterableDataset in Hugging Face Datasets, what is randomised, and why can batches still come out nearly in file order?

level: middleimportance: nice to knowfreq 8%

answer

  1. a window, not a permutation
  2. buffer_size defaults to 1,000
  3. shard order is shuffled too
  4. one big file is one shard
  5. set_epoch reseeds each pass

basics

~20 s

IterableDataset.shuffle() samples rows from a buffer of buffer_size examples (1,000 by default) and shuffles shard order. A corpus stored as one large file is one shard, so only a sliding 1,000-row window is mixed and file order leaks through.

solid answer

~40 s

A stream cannot permute rows it has not read, so `IterableDataset.shuffle(seed, buffer_size)` keeps a **buffer** (1,000 rows by default), yields a random row from it and refills the slot with the next row. It also **shuffles the order of shards** (files, or Parquet row groups after `reshard()`) and, in Datasets 5.0, fills the buffer from up to `max_buffer_input_shards` shards at once (default 10). With one 300 GB file there is one shard, so all you get is a 1,000-row sliding window and topic runs survive. Fixes: store or split the data into many shards (`reshard()` for Parquet, `to_iterable_dataset(num_shards=...)` for a prepared table), raise `buffer_size` as far as memory allows, call `shuffle` before `take`/`skip`, which lock shard order, and call `set_epoch(epoch)` each pass so the effective seed changes.

go deeper

for a junior

Recall that streaming shuffle uses a buffer, 1,000 rows by default, and that seed makes it reproducible.

for a middle

Explain both mechanisms, the sliding buffer and shard-order shuffling, and why a single-file corpus gets only a local window.

for a senior

Diagnose ordered batches: count shards, reshard or re-export, raise the buffer within memory, shuffle before take or skip, and call set_epoch each pass.

for a principal

Treat shard layout as a dataset design decision: publish many moderately sized shards so streaming shuffle and parallel reading work for every consumer.

## Why a stream cannot shuffle exactly A map-style `Dataset` in the Hugging Face **Datasets** library shuffles exactly: it permutes a list of row indices and reads rows through that **indices mapping**. A streaming **`IterableDataset`** has no row indices to permute, because it has not read the rows yet and does not know how many there are. Its `shuffle()` is therefore an **approximate** shuffle built from two mechanisms, and understanding both explains every surprising result. ## Mechanism one: the shuffle buffer `shuffle(seed=42, buffer_size=10_000)` keeps a buffer of `buffer_size` rows. 1. Fill the buffer with the first `buffer_size` rows. 2. Pick one row from the buffer at random and yield it. 3. Put the next row from the stream into the freed slot. 4. Repeat until the stream ends, then drain the buffer. A row can only move about `buffer_size` positions from where it sits in the stream. The default **`buffer_size` is 1,000**, which is a small window for a corpus of hundreds of millions of rows. A larger buffer mixes better but holds more rows in memory, so for long documents or images the buffer itself becomes the memory cost. ## Mechanism two: shard order A streamed dataset is made of **shards**: usually the data files, so a repo of 1,024 Parquet files has 1,024 shards (`ds.num_shards`). - `shuffle` also **shuffles the order of the shards** with the same seed, so iteration does not walk the files alphabetically. - In Datasets 5.0 it fills the buffer from up to **`max_buffer_input_shards`** shards at a time (default 10), interleaving them, which mixes rows from several files inside one buffer. - If the order was fixed earlier by `take()` or `skip()`, shard shuffling is disallowed and only one shard at a time feeds the buffer. Shuffle first, then take or skip. ## Why batches still look ordered | Situation | What you see | Fix | |---|---|---| | One huge file, default buffer | runs of consecutive, similar rows | more shards and a bigger buffer | | Few files, sorted by date or source | each window dominated by one source | `reshard()` (Parquet row groups) before `shuffle` | | `take`/`skip` before `shuffle` | shard order frozen | call `shuffle` first | | Same order every epoch | identical batches each pass | `set_epoch(epoch)` | | Resumed from a checkpoint | buffer contents lost | expected; the buffer refills | `reshard()` splits shards further where the format allows it; for Parquet it shards by row group instead of by file, which can turn 4 files into thousands of shards. If `num_shards` stays at 1 even after that, split the work with `skip` and `take` or rewrite the data as many files. ## Getting a good shuffle from a stream - **Store data in many shards**, for example by publishing it with `push_to_hub`, which writes Parquet shards of at most 500MB each by default. - For a table you already prepared, use **`to_iterable_dataset(num_shards=1024)`**; shards are slices of the local Arrow data, nothing is rewritten, and the stream reads quickly. - Raise **`buffer_size`** to what memory allows; 10,000 is a common starting point in the documentation's examples. - Call **`set_epoch(epoch)`** at the start of each pass: the effective seed becomes `seed + epoch`, so every epoch has a new shard order and buffer sequence while staying reproducible. - Keep the **`seed`** fixed in configuration so a run can be replayed exactly. - Check **`ds.num_shards`** before training: it is the quickest way to tell whether shard shuffling has anything to work with. - Chain **`filter` before `shuffle`** when you can: the buffer then holds only rows that survive the filter, so its limited window is spent on rows you will train on. Shards also matter beyond shuffling: when several loader worker processes read one stream they divide the shards between them, which is the data-loading side's concern, but it is another reason a single-file corpus is a poor format for streaming. ## When a stream is the wrong tool If the model needs a truly uniform shuffle, for example a small fine-tuning set where order effects matter, prepare a memory-mapped `Dataset` and use its exact `shuffle(seed=...)`. Exact shuffling has its own cost, though: reads through the indices mapping are no longer contiguous and can be several times slower until `flatten_indices()` rewrites the data.

  • How large should buffer_size be for a Hugging Face Datasets stream of long documents?
    As large as memory comfortably allows: the buffer holds that many full rows, so 10,000 long documents can cost gigabytes. Past a point, adding shards helps more than a bigger buffer, because shard shuffling and multi-shard filling mix the whole corpus while the buffer only mixes a local window.
  • Why does calling take(1000) before shuffle() in Hugging Face Datasets weaken the shuffle?
    `take` and `skip` lock in the shard order so the requested rows stay well defined. A later `shuffle` then cannot reorder shards and feeds its buffer from one shard at a time, leaving only the local buffer window. Shuffle first, then split with `take` and `skip`.

  • IterableDataset.shuffle permutes the whole stream like Dataset.shuffle
  • The default buffer_size is large enough for any corpus
  • Shard order stays fixed and only rows inside the buffer move
  • Calling shuffle once gives a new order every epoch automatically
  • Calling take() before shuffle() has no effect on shuffle quality

Assumes Datasets 5.0 (checked at 5.0.1), where shuffle() takes max_buffer_input_shards and IterableDataset has reshard(); verify both before relying on them on an older install.