Datasets
Loading data from the Hub or local files, streaming vs regular mode, .map() preprocessing, splits, and publishing your own dataset
Datasets
TL;DR
The datasets library loads, transforms, and serves data at any scale through one API — a 50-row CSV and a multi-terabyte web corpus use the same load_dataset() call. The one real decision is streaming vs regular mode, and .map() is the workhorse for turning raw examples into model-ready ones, multiprocessed and cached so you don't redo work you've already done.
| Property | Value |
|---|---|
| Level | Beginner–Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Tokenizers |
| You will understand | How to load data from anywhere, when to stream, how .map() works, and how to publish a dataset yourself |
Loading Data
The same function handles three different sources.
From the Hub
from datasets import load_dataset
ds = load_dataset("imdb", split="train")From local files
ds = load_dataset("csv", data_files="reviews.csv")
ds = load_dataset("json", data_files="reviews.jsonl")
ds = load_dataset("parquet", data_files="reviews.parquet")From a Python dict or list
from datasets import Dataset
ds = Dataset.from_dict({"text": ["good", "bad"], "label": [1, 0]})
ds = Dataset.from_list([{"text": "good", "label": 1}, {"text": "bad", "label": 0}])All three return the same Dataset object with the same API — .map(), .filter(), .train_test_split(), indexing, and slicing all work identically regardless of where the data came from.
Streaming vs Regular Mode
ds_regular = load_dataset("HuggingFaceFW/finewiki", split="train")
ds_streaming = load_dataset("HuggingFaceFW/finewiki", split="train", streaming=True)Choosing a loading mode
Regular mode
RecommendedDownloads and caches the dataset as typed Arrow tables on disk, memory-mapped — so even a dataset larger than your RAM works without loading it all into memory at once. Supports random access, len(), shuffling with a fixed buffer, and re-running your script skips the download entirely. Use this whenever the dataset fits comfortably on local disk.
Streaming mode
Reads examples as your code consumes them — no download, no local copy, and it scales to datasets far larger than any local disk. The tradeoff used to be steep; it no longer is. Use this for datasets too large to store locally, or when you want training to start immediately instead of waiting on a multi-hour download.
Streaming is fast now, not just memory-cheap. Recent releases made HuggingFace's streaming path up to 100× more efficient than earlier versions, reaching throughput on par with local SSDs when training across many workers. The old advice — "only stream when you have no other choice" — is outdated. Streaming is a legitimate default for large datasets, not just a last resort.
Regular (Dataset) | Streaming (IterableDataset) | |
|---|---|---|
| Storage | Downloaded and cached on disk | None — read on demand |
Random access (ds[i]) | Yes | No — sequential iteration only |
len(ds) | Instant | Requires a full pass, or is unavailable |
| Shuffling | True shuffle across the whole dataset | Buffered shuffle — approximate, over a sliding window |
| Startup time | Waits for the full download first | Starts consuming immediately |
| Best for | Fits on disk, needs random access or exact epoch control | Too large for local disk, or you want to start now |
.map() — Preprocessing at Scale
.map() applies a function to every example, and is how raw text becomes tokenized, model-ready input:
def tokenize(example):
return tokenizer(example["text"], truncation=True, max_length=512)
tokenized = ds.map(tokenize, batched=True, num_proc=4)batched=Truepasses your function a batch of examples instead of one at a time — the tokenizer itself is vectorized, so this is significantly faster than mapping one example per call.num_proc=4splits the work across processes for CPU-bound transforms.- Caching:
.map()writes its result to disk keyed by a hash of the function and its inputs. Re-running the same script doesn't redo the transform — it loads the cached result.
.filter() follows the same pattern for dropping examples:
long_reviews = ds.filter(lambda example: len(example["text"]) > 200)Both .map() and .filter() work on IterableDataset (streaming) too — they build a lazy transformation that runs as data flows through, rather than eagerly computing everything up front.
A cached .map() result can go stale. The cache key is derived from the function's identity and the dataset's fingerprint — but if you edit the function's external dependencies (a module-level constant, a closure variable, a file it reads) without changing the function's own code, the cache key can stay the same while the actual behavior changed. If output looks unexpectedly unchanged after editing a mapping function, pass load_from_cache_file=False once to confirm, rather than assuming the fix didn't take.
Setting the Output Format
A freshly tokenized dataset holds plain Python lists and integers. Most training loops want PyTorch tensors instead, and set_format() handles that conversion without copying the underlying Arrow data:
tokenized.set_format(type="torch", columns=["input_ids", "attention_mask", "label"])From this point, indexing the dataset (tokenized[0]) or passing it to a DataLoader returns tensors instead of raw Python lists. This is also where a common shape mismatch shows up: if columns omits a field your training loop expects, that field is silently dropped rather than erroring at format-set time — the KeyError only surfaces later, inside the training step, which makes it harder to trace back to its actual cause.
Splits
Datasets are conventionally divided into named subsets:
| Split | Purpose |
|---|---|
| train | What the model actually learns from |
| validation | Used during development to tune hyperparameters and catch overfitting early |
| test | Held out entirely until the end — the number you report |
Never tune on the test split. The moment you make a decision based on test-split performance — picking a learning rate, choosing when to stop training — it stops measuring generalization and starts measuring how well you fit that specific split. Use validation for every decision; touch test once, at the end.
If a dataset only ships a train split, create the others yourself:
split = ds.train_test_split(test_size=0.1, seed=42)
train_ds, test_ds = split["train"], split["test"]Passing a fixed seed matters for reproducibility — without one, a rerun produces a different split, and any metric comparison across runs is no longer apples-to-apples.
What Streaming Mode Costs You
Choosing IterableDataset isn't free — you give up a few things Dataset gives you by default:
Capabilities lost in streaming mode
Dataset (regular)
IterableDataset (streaming)
For most large-scale training loops none of this matters — you're doing one sequential pass (or several) over more data than you'd ever meaningfully shuffle to true randomness anyway. It matters when you need reproducible exact epochs, need to index into a specific example, or need to know the dataset's exact size before you start.
Publishing Your Own Dataset
ds.push_to_hub("your-username/your-dataset")This creates (or updates) a Hub repo, uploads the data as Parquet shards, and generates a starter dataset card. A good dataset card, like a good model card, documents:
- Task — what the data is for (classification, QA, generation, ...)
- Splits — how many examples per split, and how they were created
- License — what others are allowed to do with the data
- Source — where the raw data came from, and any preprocessing already applied
- Known limitations — biases, gaps, or quality issues you're aware of
Treat a dataset card with the same seriousness as a model card — Model Hub & Cards goes deep on what a good card looks like and why it matters as a trust signal, and the same principles apply to data as to weights.
Concept Checks
Check yourself
A dataset is 4TB and you want to start training today. What loading mode, and why?
Streaming. Regular mode would require downloading the full 4TB before training could begin, and likely wouldn't fit on local disk at all. Streaming reads examples as your training loop consumes them — no local copy, and with recent efficiency improvements, throughput now rivals local SSDs across many workers.
You changed a constant a mapping function references, but .map()'s output looks unchanged. What's the likely cause?
A stale cache hit. The cache key is built from the function and dataset fingerprint, and a change to something the function reads but isn't part of its own code — a module-level constant, an external file — can leave the cache key unchanged even though behavior changed. Rerun once with load_from_cache_file=False to confirm before assuming something else is broken.
Why is tuning hyperparameters on the test split a real mistake, not just poor form?
Because every decision made based on test-split performance is implicitly fitting to that split, even without directly training on it. The test split's entire purpose is to estimate performance on data you've made zero decisions based on — once you've used it to pick a learning rate or stopping point, that estimate is optimistic and no longer trustworthy.
What do you lose by choosing IterableDataset over Dataset?
Random access by index, an instant len(), and true global shuffling — streaming only supports sequential iteration and a buffered, approximate shuffle over a sliding window. For a single large sequential training pass this rarely matters; it matters when you need exact reproducible epochs or need to index a specific example directly.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
load_dataset() | One function for Hub, local files, or in-memory data |
| Regular mode | Downloads, caches as memory-mapped Arrow tables, supports random access |
| Streaming mode | Reads on demand, no local copy, now near-SSD throughput at scale |
.map() | Multiprocessed, cached preprocessing — works on both regular and streaming datasets |
| Stale cache gotcha | A function's external dependencies can change without invalidating the cache key |
| Splits | train / validation / test — tune only on validation, touch test once |
train_test_split() | Creates splits when a dataset ships only one |
push_to_hub() | Publishes a dataset as Parquet shards plus a starter card |
Next
You can load and run a model, tokenize correctly, and prepare data — now learn to judge whether a specific model on the Hub is trustworthy and reproducible before you build on it: Model Hub & Cards.