Regularization & Generalization
Overfitting, the train/validation gap as the central diagnostic, and the mechanics of dropout, weight decay, and normalization
Regularization & Generalization
TL;DR
A network with enough capacity can simply memorize its training data — including its noise — instead of learning patterns that generalize, a failure called overfitting. The train/validation loss gap is the diagnostic that reveals it, and dropout, weight decay, and normalization are the standard tools for preventing it.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | The Transformer Architecture |
| You will understand | What overfitting actually is, how to diagnose it, and the mechanics behind the standard fixes |
Overfitting, Precisely
A model with more effective capacity (parameters) than the training data has genuine information content can drive its training loss arbitrarily close to zero — including by memorizing noise and quirks specific to those exact examples, rather than the underlying pattern that would generalize to new data.
A model that achieves zero training loss has not necessarily learned anything useful. With enough parameters relative to the dataset size, a network can fit training data and its noise perfectly, while performing no better than random on data it hasn't seen. Training loss alone is never sufficient evidence a model is good — it only measures how well it fits the examples it was shown.
The Train/Validation Gap: The Central Diagnostic
Held-out validation data — never used to update parameters — reveals whether learning generalized:
Epoch Train loss Validation loss
1 0.85 0.88 ← both dropping together: healthy
5 0.42 0.46
10 0.21 0.25
15 0.09 0.24 ← train still dropping, val flat/rising: overfitting starting
20 0.03 0.31 ← the gap is now unambiguousReading the gap
This single chart is the most informative diagnostic in a training run. Before checking anything more elaborate, plot train and validation loss on the same axes — the shape of the gap tells you immediately whether to add regularization, get more data, or simply stop training earlier.
Dropout: Randomly Removing Signal, On Purpose
nn.Dropout(p) randomly zeroes each activation with probability p during training only, and scales the remaining activations up to compensate, so the expected total signal stays roughly constant:
dropout = torch.nn.Dropout(p=0.1)
model.train()
out_train = dropout(activations) # ~10% of values randomly zeroed, rest scaled by 1/(1-0.1)
model.eval()
out_eval = dropout(activations) # dropout is a no-op in eval mode — nothing is zeroedForgetting model.eval() before inference is one of the most common real bugs in deep learning code. If dropout (or batch norm, which behaves differently too) is left in training mode during inference, predictions become randomly noisy and inconsistent run-to-run — the model will appear to work, just unreliably, which is a far more confusing bug to chase down than an outright crash.
Why randomly removing signal helps: it prevents any single unit or narrow pathway from becoming solely responsible for a pattern the network could exploit to memorize training data — the network is forced to develop redundant, more robust representations, since it can never rely on any one unit being present.
Weight Decay (L2 Regularization)
Adding a penalty proportional to the sum of squared weights to the loss:
total_loss = task_loss + λ * sum(w^2 for w in all_weights)Larger weights are penalized more, which biases training toward smaller weight values overall — and smaller weights generally correspond to simpler, smoother functions that are less prone to fitting noise in the training data. λ (often called weight_decay in an optimizer) controls how strongly this penalty is weighted against the task loss itself.
As covered in Training Loops & Optimization, AdamW applies this penalty directly to the parameters, decoupled from the adaptive gradient update — a correctness fix over the original Adam+L2 combination, not just an implementation detail.
Normalization, in More Depth
Both batch norm and layer norm apply the same core transformation — normalize to zero mean and unit variance, then apply learnable scale and shift so the network can undo the normalization if that's actually useful for a given layer:
normalized = (x - mean) / sqrt(variance + epsilon)
output = gamma * normalized + beta # gamma, beta: learnable parametersThe difference is what's being averaged over: batch norm computes mean/variance across the batch dimension for each feature; layer norm computes them across the feature dimension for each individual example (see The Transformer Architecture for why transformers specifically prefer layer norm).
Normalization is a regularizer partly by accident, and a training-stability tool primarily on purpose. Its main job is smoothing the loss landscape — keeping activations in a well-behaved range makes larger learning rates viable and training more stable. The mild regularizing effect (batch norm's per-batch statistics inject a small amount of noise into training, similar in spirit to dropout) is a secondary benefit, not the primary reason it's used.
Early Stopping
The simplest regularization technique of all: monitor validation loss during training, and stop (or revert to the best checkpoint) once it stops improving, rather than training for a fixed number of epochs regardless of what the validation curve is doing.
best_val_loss = float("inf")
patience, patience_counter = 5, 0
for epoch in range(max_epochs):
train_one_epoch(model, train_loader)
val_loss = evaluate(model, val_loader)
if val_loss < best_val_loss:
best_val_loss = val_loss
patience_counter = 0
save_checkpoint(model, "best.pt")
else:
patience_counter += 1
if patience_counter >= patience:
break # validation loss hasn't improved in `patience` epochs — stopData Augmentation, Briefly
Rather than constraining the model, augmentation expands the effective diversity of the training data itself — random crops/flips for images, synonym substitution or back-translation for text — so the same underlying examples present the model with more variation to learn from, making pure memorization less achievable in the first place. This is a data-side complement to the model-side techniques above, not a replacement for them.
Concept Checks
Check yourself
A model reaches zero training loss. Does that mean it's a good model?
Not necessarily, and often not at all. With enough capacity relative to the dataset, a network can fit the training examples — including their noise — perfectly while generalizing poorly to unseen data. Training loss alone only measures fit to the examples shown; validation performance on held-out data is what actually indicates whether the model learned something generalizable.
Why does forgetting model.eval() before running inference cause inconsistent, rather than simply wrong, predictions?
Because dropout (left in training mode) continues randomly zeroing different activations on every forward pass, so the same input can produce different outputs from run to run — the randomness that's supposed to be a training-time regularizer is still active at inference. This produces intermittent, hard-to-reproduce inconsistency rather than a single, obviously-wrong, and thus easier to diagnose output.
Why is normalization's primary purpose training stability rather than regularization, even though it does have a mild regularizing effect?
Because its core mechanism — keeping activations in a well-behaved, consistent range — is what smooths the loss landscape and makes larger learning rates viable, which is a stability and optimization benefit. The regularizing effect, from batch norm's per-batch statistics injecting a small amount of noise, is a secondary side effect of that mechanism, not the reason normalization layers were introduced.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Overfitting | Memorizing training data (and its noise) instead of learning generalizable patterns |
| Train/validation gap | The central diagnostic — both dropping is healthy, diverging means overfitting |
| Dropout | Randomly zeroes activations during training only, forcing redundant representations |
model.eval() | Required before inference — forgetting it leaves dropout/batch norm in training mode |
| Weight decay (L2) | Penalizes large weights, biasing toward simpler functions |
| AdamW | Applies weight decay directly to parameters, decoupled from the adaptive update |
| Batch vs layer norm | Differ in what's averaged over — batch dimension vs feature dimension |
| Early stopping | Stop training once validation loss stops improving, rather than at a fixed epoch count |
| Data augmentation | Expands effective training diversity from the data side, complementing model-side regularization |
Next
With overfitting under control, the next page covers how networks learn useful representations of their inputs in the first place: Embeddings & Representation Learning.
The Transformer Architecture
Attention built up from scratch — Query/Key/Value, the scaling factor, multi-head attention, positional encoding, and a full transformer block
Embeddings & Representation Learning
What an embedding actually is mechanically, why training makes embedding spaces semantically structured, and how contrastive learning builds them on purpose