In Hugging Face Datasets, what does load_dataset() return for a Hub repo or local files, and how do you map files to splits?
answer
- one function, many sources
- DatasetDict unless split is given
- builder name plus data_files
- a {split: files} mapping
- revision pins the repo commit
basics
~20 sload_dataset() returns a DatasetDict of splits, or one Arrow-backed Dataset when split= is given. It reads a Hub repo, a local folder, or a builder name such as "json" with data_files, whose {split: files} mapping names the splits.
solid answer
~40 s`load_dataset()` takes a Hub repo id, a local directory, or a **builder name** (`"csv"`, `"json"`, `"parquet"`, `"text"`) plus `data_files`. It detects the file format, downloads what it needs, converts it to Arrow in the Datasets cache and memory-maps it. Without `split` you get a `DatasetDict` keyed by split name; with `split="train"` you get one `Dataset` that supports `len()`, `ds[0]` and slicing, and split strings can slice further (`"train[:10%]"`). Splits come from a `data_files` mapping such as `{"train": "train-*.jsonl", "validation": "val.jsonl"}`; a plain string or list lands in a single `train` split; a repo or folder without a mapping is split by file and directory names or its card's YAML. For a Hub repo, `revision=` pins a tag or commit instead of the moving `main` branch.
code
python · 15 linesfrom datasets import load_dataset
# A dataset repo on the Hub, pinned to one commit
hub = load_dataset("your-org/support-tickets", revision="<commit-sha>")
# Local JSON Lines files mapped to named splits
local = load_dataset(
"json",
data_files={"train": "data/train-*.jsonl", "validation": "data/val.jsonl"},
)
print(local) # DatasetDict with 'train' and 'validation'
# One split, first 10% only: a Dataset with random access
small = load_dataset("csv", data_files="tickets.csv", split="train[:10%]")
print(len(small), small[0])go deeper
Recall the two return types, DatasetDict without split and Dataset with it, and show a builder name such as json with data_files for local files.
Explain how files become splits: a data_files mapping, a single train split for a plain list, and file-name or card-YAML detection for a repo, plus globs and data_dir for partial loads.
Talk about repeatability: pin revision for Hub data, know that the first load downloads and converts to Arrow while later loads reuse the cache, and when force_redownload is needed.
Frame load_dataset as the contract between data owners and training code: a repo layout and card YAML that name splits, pinned revisions, and loading code that stays a one-line call.
## One entry point for every source `load_dataset()` is the front door of the Hugging Face **Datasets** library. Its first argument, `path`, says where the data lives, and the library picks a **builder** for you: a builder is the reader for one file format (CSV, JSON Lines, Parquet, plain text, Arrow and several media folders), not a particular dataset. | Source | Typical call | What happens | |---|---|---| | Dataset repo on the Hub | `load_dataset("your-org/support-tickets")` | finds the data files in the repo, picks the builder from the most common format, downloads them | | Local folder | `load_dataset("./data/tickets")` | the same detection, run on files in that directory | | Builder name plus files | `load_dataset("json", data_files="tickets.jsonl")` | uses the named builder on exactly the files you list | | Files in a Hub repo by URL | `load_dataset("parquet", data_files="hf://datasets/your-org/support-tickets/data/train.parquet")` | reads those remote files with that builder | The builder names include `"csv"`, `"json"`, `"parquet"`, `"text"`, `"arrow"`, `"webdataset"` and `"imagefolder"`. A `"text"` builder turns every line into one row with a `text` column, which is the usual way to load a raw corpus. ## What comes back The return type depends on two arguments: - **No `split`** returns a **`DatasetDict`**, a dictionary from split name (`"train"`, `"validation"`, `"test"`) to `Dataset`. - **`split="train"`** returns a single **`Dataset`**. It is an Arrow table stored on disk and **memory-mapped**, so `len(ds)`, `ds[0]` and `ds[100:200]` work without reading the whole table into RAM. - **Split strings slice and combine**: `split="train[:10%]"`, `split="train[10:20]"` and `split="train+test"` all work, which is handy for a quick smoke test on a large dataset. - **`streaming=True`** returns an `IterableDataset` (or an `IterableDatasetDict`) instead: nothing is downloaded up front and rows are read lazily while you iterate. A common slip is to write `ds[0]` on the result of a call with no `split`; that indexes the dictionary, not a row, and fails because `0` is not a split name. ## Mapping files to splits Split assignment follows a short set of rules: 1. If `data_files` is a **mapping** such as `{"train": ["train-0.jsonl", "train-1.jsonl"], "validation": "val.jsonl"}`, every key becomes a split. 2. If `data_files` is a **string or a list**, all of those files go into one split named `train`. 3. If you pass no `data_files` for a Hub repo or a folder, the library looks for split names in file and directory names (`train`, `test`, `validation`) or in the YAML configuration of the dataset card, and otherwise puts everything into `train`. Paths in `data_files` may be **glob patterns** (`"data/train-*.jsonl.gz"`) and are resolved relative to the repo or folder being loaded, so you can pull one slice of a large Hub repo, for example only the English files. `data_dir="en"` is the shorthand for every file under one sub-directory. When the data has no validation split at all, `ds.train_test_split(test_size=0.1, seed=42)` carves one out and returns a new `DatasetDict`. ## What happens on a non-streaming load 1. **Resolve** the source and the builder (format, files, split assignment). 2. **Download** the files into the Hugging Face Hub cache (`~/.cache/huggingface/hub` by default) unless they are local. 3. **Convert** them to typed Arrow files in the Datasets cache (`~/.cache/huggingface/datasets` by default, moved with `HF_DATASETS_CACHE` or `cache_dir=`). 4. **Memory-map** those Arrow files and return the `Dataset` objects. On the next run the default `download_mode` (`"reuse_dataset_if_exists"`) finds the prepared Arrow files and skips steps 2 and 3. `download_mode="force_redownload"` throws them away and starts again. ## Pinning the version you loaded A dataset repo on the Hub is a git repository. Without `revision=`, `load_dataset` follows its **`main` branch**, which moves whenever the owner pushes. Passing `revision=` with a tag or a commit sha makes the same call return the same files later; recording that sha next to a trained model is the start of reproducibility, and the wider snapshot and lineage practice is a separate subject. ## What interviewers listen for - Knowing the return type without running it: `DatasetDict` without `split`, `Dataset` with it. - Reaching for a **builder name plus `data_files`** for local files, not for custom parsing code. - Using a **`data_files` mapping** to name splits, and globs to load part of a repo. - Mentioning that the result is **memory-mapped Arrow on disk**, not a pandas frame held in RAM. - Pinning `revision=` when the run has to be repeatable.
- Your local CSV has no validation file. How do you get one with Hugging Face Datasets?Load it with `split="train"` and call `train_test_split(test_size=0.1, seed=42)`, which returns a `DatasetDict` with `train` and `test` splits. The split shuffles by default; the fixed `seed` makes the same rows land in the same split on every run, and you can rename `test` to `validation` if a trainer expects that name.
- How do you load only the English part of a large multilingual Hub dataset with Hugging Face Datasets?Point `data_files` at a glob relative to the repo, for example `data_files="en/*.json.gz"`, or pass `data_dir="en"`. Only the matching files are downloaded and converted, which saves both bandwidth and the disk space the Arrow copy would take.
- load_dataset always returns one Dataset you can index with ds[0]
- Local CSV or JSON files need custom parsing code before load_dataset can read them
- data_files accepts only a single file path
- load_dataset reads the whole dataset into RAM as a DataFrame
- Leaving revision unset keeps you on the version you first downloaded