Tokenizers
How text becomes integers — BPE, WordPiece, and Unigram from scratch, the tokenizer/model matched-pair rule, and chat templates
Tokenizers
TL;DR
A model never sees text — it sees integers. A tokenizer converts a string into a sequence of integer IDs from a fixed vocabulary, and back again. Every model is trained against one specific vocabulary, built by one specific algorithm (BPE, WordPiece, or Unigram), and using the wrong tokenizer with a model produces IDs the model was never trained on — a failure that looks like a working system, not an error.
| Property | Value |
|---|---|
| Level | Beginner–Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Transformers & Pipelines |
| You will understand | How subword tokenization actually builds a vocabulary, why the tokenizer/model pairing is non-negotiable, and how chat templates work |
Why Tokenization Exists
Neural networks operate on numbers. A vocabulary of whole words breaks on anything it hasn't seen before — a typo, a rare name, a new product name — and there's no way to build a whole-word vocabulary large enough to never encounter an unseen word. A vocabulary of individual characters solves the unseen-word problem but produces extremely long sequences for the model to process, and characters alone carry little meaning individually.
Subword tokenization is the compromise that won: build a vocabulary of frequently occurring word pieces — whole common words, but also frequent prefixes, suffixes, and character combinations — so that any string can be represented, common words stay as one token, and rare words fall back to a handful of smaller pieces instead of one unknown token.
tokenizer("Tokenization is not optional.")
# {'input_ids': [101, 19204, 3989, 2003, 2025, 11887, 1012, 102],
# 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}Three Algorithms, One Goal
All three build a fixed vocabulary from a training corpus before the model ever sees data; the difference is how they choose what goes in the vocabulary.
| Algorithm | Used by | Idea |
|---|---|---|
| BPE (Byte-Pair Encoding) | GPT-family, Llama | Start from characters, repeatedly merge the most frequent adjacent pair |
| WordPiece | BERT-family | Like BPE, but merges are chosen to maximize the training corpus's likelihood, not just frequency |
| Unigram | T5, most SentencePiece models | Start with a large candidate vocabulary, prune down to the pieces that matter most |
Watch BPE Build Itself
The clearest way to understand subword tokenization is to build a tiny vocabulary by hand. Toy corpus, three words, shown as character sequences with an end-of-word marker:
"low low low lower lowest"
→ l o w _, l o w _, l o w _, l o w e r _, l o w e s t _Step 0 — start from characters. The initial vocabulary is just the unique symbols: l, o, w, e, r, s, t, _.
Step 1 — count adjacent pairs. (l, o) appears 5 times, (o, w) appears 5 times, (w, _) appears 3 times, and so on. Merge the most frequent pair: (l, o) → lo.
lo w _, lo w _, lo w _, lo w e r _, lo w e s t _Step 2 — recount and merge again. Now (lo, w) is the most frequent pair (5 occurrences). Merge it: lo w → low.
low _, low _, low _, low e r _, low e s t _Step 3 — keep going. (low, _) merges next (3 occurrences, from the three plain "low"s), giving a low_ token that represents the whole word "low" as one piece — while "lower" and "lowest" still decompose into low + e + r/s + t + _, sharing the common prefix.
Repeat until you hit a target vocabulary size (real tokenizers stop in the tens of thousands, not after four merges). The result: common whole words become single tokens, and everything else decomposes into meaningful, reusable pieces rather than falling back to an unknown-token placeholder.
This is exactly why an unfamiliar word doesn't break a subword tokenizer. "lowering" was never in the training corpus above, but with the vocabulary just built it might tokenize as low + er + ing — three known pieces recombined, not one unknown word.
WordPiece runs the same merge loop but scores candidate merges by how much they'd improve the training corpus's likelihood under the resulting vocabulary, not by raw pair frequency. Unigram works in the opposite direction: start with a huge candidate set of substrings, then iteratively remove the ones that hurt overall corpus likelihood least, until the target vocabulary size is reached.
Special Tokens
Every tokenizer reserves a handful of IDs for structural markers that carry no lexical meaning but tell the model something about the input's shape:
| Token | Family | Purpose |
|---|---|---|
[CLS] | BERT-style | Marks the start of input; its final hidden state is often used as a whole-sequence representation |
[SEP] | BERT-style | Separates two segments — e.g. a question and a passage in question answering |
[PAD] | Most families | Fills shorter sequences up to a batch's common length |
| `< | endoftext | >` |
<s> / </s> | Many families | Beginning/end of sequence markers |
tokenizer("Two sentences.", "A second one.")
# input_ids: [CLS] Two sentences . [SEP] A second one . [SEP]Padding tokens need an accompanying attention mask — a parallel array of 1s and 0s telling the model which positions are real content versus padding, so the padding doesn't influence attention scores.
The Matched-Pair Rule
The tokenizer and the model are trained together against one vocabulary, and that pairing is not optional. Loading bert-base-uncased's tokenizer and feeding its output to a Llama model produces a sequence of integer IDs — the code runs, no exception is raised — but those IDs map to completely different subwords in Llama's vocabulary than the ones BERT's tokenizer intended. The model runs a forward pass on effectively random tokens and produces fluent-looking, confidently wrong output. This is the single most consequential mistake beginners make with this stack, precisely because it fails silently instead of erroring.
The fix is mechanical: always load the tokenizer from the same repo as the model.
repo = "meta-llama/Llama-4-Scout"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)If you ever find yourself hardcoding a tokenizer repo string that's different from the model's repo string, stop and double-check that's actually intended — it almost never is.
Chat Templates
Chat-tuned models are trained on conversations formatted a specific way — role markers, turn boundaries, sometimes a system-prompt convention — and that format differs between model families. Sending a model raw, un-templated text ("User: hi\nAssistant:") when it expects a structured template is a common bug: the model doesn't error, it just performs worse, ignores the system prompt, or drifts out of character.
apply_chat_template() handles this correctly by reading the formatting rules baked into the tokenizer's config:
messages = [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is a tokenizer?"},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)tokenize=False returns the formatted string (useful for inspection or passing to a pipeline); tokenize=True (the default) returns ready-to-generate input IDs directly. add_generation_prompt=True appends whatever marker tells the model "now produce the assistant's turn" — without it, some models will continue the user's turn instead of replying.
Two ways to get this wrong
Sending raw text to a chat-tuned model
"User: What is a tokenizer?\nAssistant:" sent as plain text. The model may still produce something plausible-sounding, because it's a capable language model — but it wasn't trained on this literal format, and the gap shows up as ignored system prompts, inconsistent behavior, or subtly worse answers that are hard to attribute to the real cause.
Using the wrong model's chat template
Applying apply_chat_template() from a different model family's tokenizer produces a well-formed string — but in the wrong format, which the target model wasn't trained on. This is a variant of the matched-pair rule above: template and model need to come from the same repo, just like tokenizer and model do.
Fast vs Slow Tokenizers
The tokenizers library provides fast tokenizers, Rust-backed implementations that are the default for supported models. They're substantially faster than the older pure-Python implementations, and — critically — they're the only ones that support offset mapping: returning, for each token, the exact character span in the original string it came from.
encoding = tokenizer("HuggingFace is great", return_offsets_mapping=True)
encoding.offset_mapping
# [(0, 0), (0, 5), (5, 10), (11, 13), (14, 19), (0, 0)]Offset mapping is what makes span-based tasks like named entity recognition possible: the model predicts a label per token, and offsets let you map that prediction back to the exact substring of the original text, including cases where one word split into multiple tokens.
| Fast (Rust-backed) | Slow (pure Python) | |
|---|---|---|
| Speed | Much faster, especially for batches | Noticeably slower |
| Offset mapping | Supported | Not supported |
| Default | Yes, for models that ship one | Only as a fallback |
Training Your Own Tokenizer
You rarely need this — most projects use the tokenizer that shipped with the base model. It's worth doing when:
- Your domain's vocabulary looks nothing like general text — code, chemical formulas, protein sequences
- You're working in a low-resource language poorly represented in existing tokenizers' training data, where a general-purpose vocabulary wastes tokens splitting common words into many pieces
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]"])
tokenizer.train(files=["corpus.txt"], trainer=trainer)A newly trained tokenizer needs a model trained against it — you can't attach a fresh vocabulary to an existing pretrained model's weights and expect it to work. This is a from-scratch-pretraining decision, not a drop-in swap.
Concept Checks
Check yourself
Why does BPE merge 'lo' before merging 'low'+'e'?
Because BPE always merges the single most frequent adjacent pair at each step, and in the toy corpus (l, o) and (o, w) were the most frequent pairs before any word-level piece existed. Merges compound — lo becomes a candidate for further merging with w only after it exists as a unit, which is why the algorithm proceeds character-pair, then piece-pair, gradually building up to whole recurring words.
You swap BERT's tokenizer for a Llama model's tokenizer 'just to try it.' What happens?
The code runs without error and returns a plausible-looking output — that's what makes this bug dangerous. The token IDs produced by BERT's vocabulary don't correspond to the same subwords in Llama's vocabulary, so the model performs a forward pass on essentially scrambled input and produces fluent but meaningless output.
Why do NER pipelines specifically need fast tokenizers?
Because span-based tasks need to map a per-token prediction back to the exact character range in the original string, and only fast (Rust-backed) tokenizers expose that offset mapping. A slow tokenizer can still tokenize the text, but there's no supported way to recover which characters a given token corresponds to.
A chat model ignores your system prompt. What's the first thing to check?
Whether the input went through apply_chat_template() at all, or was sent as raw concatenated text. Chat-tuned models are trained on a specific structured format per family; raw text that merely resembles a conversation wasn't the format the model actually learned, and the system prompt's special positioning in that format is often exactly what gets lost.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Tokenizer | Converts text to integer IDs from a fixed vocabulary, and back |
| BPE | Repeatedly merges the most frequent adjacent pair, starting from characters |
| WordPiece | BPE-like, but merges chosen to maximize training-corpus likelihood |
| Unigram | Starts from a huge vocabulary, prunes to the most useful pieces |
| Special tokens | Structural markers ([CLS], [SEP], padding, end-of-text) with no lexical meaning |
| Matched-pair rule | Tokenizer and model must come from the same repo — mismatches fail silently |
| Chat template | Formats role/content messages into the exact string a chat-tuned model expects |
| Fast tokenizer | Rust-backed, faster, and the only kind supporting offset mapping |
| Offset mapping | Maps each token back to its character span in the original string |
| Training your own | Rarely needed — reserve it for domain vocabularies general tokenizers handle poorly |
Next
Text becomes tokens; now see how the data those tokens come from gets loaded, streamed, and preprocessed at scale: Datasets.
Transformers & Pipelines
pipeline() versus the Auto* classes, generation parameters, dtype, and device_map — the two ways to run a model and when each earns its cost
Datasets
Loading data from the Hub or local files, streaming vs regular mode, .map() preprocessing, splits, and publishing your own dataset