skip to content

Datasets Library

load_dataset returns an Arrow-backed, memory-mapped table or a streaming IterableDataset, and batched map caches each transform. Interviewers care because data plumbing is where fine-tuning stalls.

on this pageshow

questions

6

In Hugging Face Datasets, what does load_dataset() return for a Hub repo or local files, and how do you map files to splits?

level: juniorimportance: must knowfreq 30%

answer

  1. one function, many sources
  2. DatasetDict unless split is given
  3. builder name plus data_files
  4. a {split: files} mapping
  5. revision pins the repo commit

basics

~20 s

load_dataset() returns a DatasetDict of splits, or one Arrow-backed Dataset when split= is given. It reads a Hub repo, a local folder, or a builder name such as "json" with data_files, whose {split: files} mapping names the splits.

solid answer

~40 s

`load_dataset()` takes a Hub repo id, a local directory, or a **builder name** (`"csv"`, `"json"`, `"parquet"`, `"text"`) plus `data_files`. It detects the file format, downloads what it needs, converts it to Arrow in the Datasets cache and memory-maps it. Without `split` you get a `DatasetDict` keyed by split name; with `split="train"` you get one `Dataset` that supports `len()`, `ds[0]` and slicing, and split strings can slice further (`"train[:10%]"`). Splits come from a `data_files` mapping such as `{"train": "train-*.jsonl", "validation": "val.jsonl"}`; a plain string or list lands in a single `train` split; a repo or folder without a mapping is split by file and directory names or its card's YAML. For a Hub repo, `revision=` pins a tag or commit instead of the moving `main` branch.

code

python · 15 lines
python
from datasets import load_dataset

# A dataset repo on the Hub, pinned to one commit
hub = load_dataset("your-org/support-tickets", revision="<commit-sha>")

# Local JSON Lines files mapped to named splits
local = load_dataset(
    "json",
    data_files={"train": "data/train-*.jsonl", "validation": "data/val.jsonl"},
)
print(local)                    # DatasetDict with 'train' and 'validation'

# One split, first 10% only: a Dataset with random access
small = load_dataset("csv", data_files="tickets.csv", split="train[:10%]")
print(len(small), small[0])

go deeper

for a junior

Recall the two return types, DatasetDict without split and Dataset with it, and show a builder name such as json with data_files for local files.

for a middle

Explain how files become splits: a data_files mapping, a single train split for a plain list, and file-name or card-YAML detection for a repo, plus globs and data_dir for partial loads.

for a senior

Talk about repeatability: pin revision for Hub data, know that the first load downloads and converts to Arrow while later loads reuse the cache, and when force_redownload is needed.

for a principal

Frame load_dataset as the contract between data owners and training code: a repo layout and card YAML that name splits, pinned revisions, and loading code that stays a one-line call.

## One entry point for every source `load_dataset()` is the front door of the Hugging Face **Datasets** library. Its first argument, `path`, says where the data lives, and the library picks a **builder** for you: a builder is the reader for one file format (CSV, JSON Lines, Parquet, plain text, Arrow and several media folders), not a particular dataset. | Source | Typical call | What happens | |---|---|---| | Dataset repo on the Hub | `load_dataset("your-org/support-tickets")` | finds the data files in the repo, picks the builder from the most common format, downloads them | | Local folder | `load_dataset("./data/tickets")` | the same detection, run on files in that directory | | Builder name plus files | `load_dataset("json", data_files="tickets.jsonl")` | uses the named builder on exactly the files you list | | Files in a Hub repo by URL | `load_dataset("parquet", data_files="hf://datasets/your-org/support-tickets/data/train.parquet")` | reads those remote files with that builder | The builder names include `"csv"`, `"json"`, `"parquet"`, `"text"`, `"arrow"`, `"webdataset"` and `"imagefolder"`. A `"text"` builder turns every line into one row with a `text` column, which is the usual way to load a raw corpus. ## What comes back The return type depends on two arguments: - **No `split`** returns a **`DatasetDict`**, a dictionary from split name (`"train"`, `"validation"`, `"test"`) to `Dataset`. - **`split="train"`** returns a single **`Dataset`**. It is an Arrow table stored on disk and **memory-mapped**, so `len(ds)`, `ds[0]` and `ds[100:200]` work without reading the whole table into RAM. - **Split strings slice and combine**: `split="train[:10%]"`, `split="train[10:20]"` and `split="train+test"` all work, which is handy for a quick smoke test on a large dataset. - **`streaming=True`** returns an `IterableDataset` (or an `IterableDatasetDict`) instead: nothing is downloaded up front and rows are read lazily while you iterate. A common slip is to write `ds[0]` on the result of a call with no `split`; that indexes the dictionary, not a row, and fails because `0` is not a split name. ## Mapping files to splits Split assignment follows a short set of rules: 1. If `data_files` is a **mapping** such as `{"train": ["train-0.jsonl", "train-1.jsonl"], "validation": "val.jsonl"}`, every key becomes a split. 2. If `data_files` is a **string or a list**, all of those files go into one split named `train`. 3. If you pass no `data_files` for a Hub repo or a folder, the library looks for split names in file and directory names (`train`, `test`, `validation`) or in the YAML configuration of the dataset card, and otherwise puts everything into `train`. Paths in `data_files` may be **glob patterns** (`"data/train-*.jsonl.gz"`) and are resolved relative to the repo or folder being loaded, so you can pull one slice of a large Hub repo, for example only the English files. `data_dir="en"` is the shorthand for every file under one sub-directory. When the data has no validation split at all, `ds.train_test_split(test_size=0.1, seed=42)` carves one out and returns a new `DatasetDict`. ## What happens on a non-streaming load 1. **Resolve** the source and the builder (format, files, split assignment). 2. **Download** the files into the Hugging Face Hub cache (`~/.cache/huggingface/hub` by default) unless they are local. 3. **Convert** them to typed Arrow files in the Datasets cache (`~/.cache/huggingface/datasets` by default, moved with `HF_DATASETS_CACHE` or `cache_dir=`). 4. **Memory-map** those Arrow files and return the `Dataset` objects. On the next run the default `download_mode` (`"reuse_dataset_if_exists"`) finds the prepared Arrow files and skips steps 2 and 3. `download_mode="force_redownload"` throws them away and starts again. ## Pinning the version you loaded A dataset repo on the Hub is a git repository. Without `revision=`, `load_dataset` follows its **`main` branch**, which moves whenever the owner pushes. Passing `revision=` with a tag or a commit sha makes the same call return the same files later; recording that sha next to a trained model is the start of reproducibility, and the wider snapshot and lineage practice is a separate subject. ## What interviewers listen for - Knowing the return type without running it: `DatasetDict` without `split`, `Dataset` with it. - Reaching for a **builder name plus `data_files`** for local files, not for custom parsing code. - Using a **`data_files` mapping** to name splits, and globs to load part of a repo. - Mentioning that the result is **memory-mapped Arrow on disk**, not a pandas frame held in RAM. - Pinning `revision=` when the run has to be repeatable.

  • Your local CSV has no validation file. How do you get one with Hugging Face Datasets?
    Load it with `split="train"` and call `train_test_split(test_size=0.1, seed=42)`, which returns a `DatasetDict` with `train` and `test` splits. The split shuffles by default; the fixed `seed` makes the same rows land in the same split on every run, and you can rename `test` to `validation` if a trainer expects that name.
  • How do you load only the English part of a large multilingual Hub dataset with Hugging Face Datasets?
    Point `data_files` at a glob relative to the repo, for example `data_files="en/*.json.gz"`, or pass `data_dir="en"`. Only the matching files are downloaded and converted, which saves both bandwidth and the disk space the Arrow copy would take.

  • load_dataset always returns one Dataset you can index with ds[0]
  • Local CSV or JSON files need custom parsing code before load_dataset can read them
  • data_files accepts only a single file path
  • load_dataset reads the whole dataset into RAM as a DataFrame
  • Leaving revision unset keeps you on the version you first downloaded
open as a page

Your Hugging Face Datasets map(batched=True) tokenization re-runs from scratch whenever the script starts: how does its cache decide reuse, and what breaks it?

level: middleimportance: must knowfreq 22%

basics

~20 s

Dataset.map() names its output cache file after a fingerprint: a hash of the previous fingerprint, the function and its arguments. A function that cannot be hashed stably, an unseeded shuffle or split upstream, an in-memory dataset or disabled caching forces recomputation.

open as a page

In Hugging Face Datasets, what does with_format('torch') change about the rows a tokenized Dataset returns, and what does it leave alone?

level: juniorimportance: should knowfreq 15%

basics

~20 s

with_format('torch') converts numeric columns to PyTorch tensors when rows are read, leaving the Arrow data unchanged. Strings stay strings, ragged columns come back as lists of tensors rather than one padded tensor, and columns= limits the output.

open as a page

With Hugging Face Datasets, should a 300 GB text corpus on a 32 GB-RAM machine load memory-mapped or with streaming=True, and what does each give up?

level: seniorimportance: should knowfreq 14%

basics

~20 s

RAM is not the limit: a Dataset is memory-mapped Arrow on disk, needing disk for downloads plus the Arrow copy but giving len, row access and exact shuffle. streaming=True writes nothing, iterating lazily with approximate shuffle and no map cache.

open as a page

What does Hugging Face Datasets' push_to_hub() write to a dataset repo, and how do teammates load exactly that upload later?

level: middleimportance: nice to knowfreq 7%

basics

~10 s

push_to_hub() commits Parquet shards (data/train-00000-of-0000N.parquet, at most 500MB each by default) plus dataset-card YAML naming the splits. It returns a CommitInfo; loading with revision= set to that commit sha reproduces exactly that upload.

open as a page

When you call shuffle() on a streaming IterableDataset in Hugging Face Datasets, what is randomised, and why can batches still come out nearly in file order?

level: middleimportance: nice to knowfreq 8%

basics

~20 s

IterableDataset.shuffle() samples rows from a buffer of buffer_size examples (1,000 by default) and shuffles shard order. A corpus stored as one large file is one shard, so only a sliding 1,000-row window is mixed and file order leaks through.

open as a page