Failure Modes & Debugging
Symptom to cause across the whole deep learning stack — the classic bugs that look like mysteries but have mechanical, checkable causes
Failure Modes & Debugging
TL;DR
Most deep learning bugs that feel mysterious have a small, well-known mechanical cause — a forgotten model.eval(), an accumulating gradient from a missing zero_grad(), a shape mismatch, a frozen layer that should be trainable. Learn to recognize the symptom-to-cause pattern once and most future instances resolve in minutes instead of hours.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~16 minutes |
| Prerequisites | Production & Operations |
| You will understand | The classic deep learning bugs, their symptoms, and how to spot each one quickly |
Triage First
Before debugging a specific symptom, ask what changed. Most training bugs are introduced by a recent code change — a refactor, a new layer, a config tweak — not by something that was always broken. Check what's different from the last known-good run before searching broadly.
Symptom to Cause
| Symptom | Usual cause |
|---|---|
| Inference output is noisy or inconsistent between runs on the same input | Forgot model.eval() — dropout and batch norm behave differently in train mode, and are still active |
| Loss decreases far slower than expected, or not at all, despite a reasonable learning rate | Forgot optimizer.zero_grad() — gradients from previous steps are accumulating on top of new ones instead of being reset |
| Fine-tuning appears to run but the model's behavior doesn't change at all | A layer that should be trainable still has requires_grad=False left over from a previous setup |
RuntimeError: size mismatch or similar shape errors | A classic, extremely common PyTorch error — read the shapes in the traceback carefully; it names the exact two tensors that disagree |
| Training order looks suspiciously non-random, or performance is oddly sensitive to data order | DataLoader(shuffle=False) where shuffling was intended, silently biasing what the model sees in what order |
| A LoRA fine-tune runs without error, but barely changes model behavior | target_modules pointed at layers that don't meaningfully affect the task's output — verify with model.print_trainable_parameters() that the trainable parameter count and its location look right |
| Model performs worse on general tasks after fine-tuning on a narrow one | Catastrophic forgetting — learning rate too high or too many epochs on too narrow a dataset, per Fine-Tuning & Transfer Learning |
| Training works on one GPU, breaks or hangs under DDP/FSDP2 | A parameter that isn't used in every forward pass (common with conditional branches) breaks gradient synchronization — every GPU must run the exact same set of parameters through backward every step |
Reading a Shape-Mismatch Traceback
This deserves its own callout because it's the single most common error message in deep learning code, and it's more informative than it looks at first glance:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x768 and 512x256)The traceback is telling you exactly which two tensors disagree and how. Here, something produced a (32, 768) tensor — batch size 32, feature dimension 768 — and it's being passed into a layer expecting an input dimension of 512. The fix is almost always upstream: either the previous layer's output dimension is wrong, or this layer's declared input dimension doesn't match what's actually being fed to it. Read the two shapes, and trace backward from there to where the mismatched dimension was actually produced.
The model.eval() / zero_grad() Pair
These two are grouped because they're the most common source of "the code runs, the result is just subtly wrong" bugs — no error, no traceback, just a model that trains or infers worse than it should for no obvious reason.
# Training loop — easy to get wrong in either direction
model.train() # dropout/batchnorm active — correct for training
for batch in dataloader:
optimizer.zero_grad() # reset — forgetting this accumulates gradients across steps
loss = loss_fn(model(batch.x), batch.y)
loss.backward()
optimizer.step()
model.eval() # dropout/batchnorm now deterministic — correct for inference
with torch.no_grad():
predictions = model(test_batch)Both of these fail silently. Nothing crashes if you forget model.eval() before inference or optimizer.zero_grad() in a training loop — the code runs to completion and produces output, just worse or noisier output than it should. This is exactly why they're worth checking early when something looks subtly off rather than obviously broken.
Concept Checks
Check yourself
A model gives inconsistent predictions on the exact same input across repeated calls. What's the first thing to check?
Whether model.eval() was called before inference. In train mode, dropout randomly zeroes activations and batch norm uses batch statistics rather than running averages, both of which introduce randomness or batch-dependence into what should be a deterministic forward pass at inference time.
Training loss barely decreases even though the learning rate looks reasonable and the loss function is correct. What common bug produces exactly this symptom?
A missing optimizer.zero_grad(). Without resetting gradients each step, new gradients accumulate on top of previous ones rather than replacing them, so the optimizer step is based on a growing, increasingly incorrect gradient rather than the current batch's actual gradient — training technically runs, but effectively can't converge properly.
Why does a shape-mismatch error message like '(32x768 and 512x256)' actually tell you where to look, rather than just being an opaque crash?
Because it names the exact two tensor shapes that disagree and the dimension that doesn't match — here, a 768-dimensional feature meeting a layer expecting 512. That points directly at either the layer that produced the 768-dimensional output or the layer declared with a 512-dimensional input, letting you trace backward to the actual source of the mismatch instead of searching the whole model.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
model.eval() | Required before inference — otherwise dropout/batchnorm stay in training behavior |
optimizer.zero_grad() | Required each step — otherwise gradients silently accumulate across steps |
| Frozen layer left over | requires_grad=False surviving a setup change silently prevents intended training |
| Shape mismatches | The traceback names the exact disagreeing dimensions — read it, don't guess |
shuffle=False by accident | Silently biases the order the model sees training data in |
Wrong LoRA target_modules | Trains without error but barely changes behavior — verify trainable parameter count and location |
| DDP/FSDP2 hangs | Every GPU must run the identical set of parameters through backward each step |
Next
With the common bugs catalogued, the next page walks through one complete worked design that gets these decisions right from the start: Designing a Training System.
Production & Operations
Checkpointing discipline, honest reproducibility, experiment tracking, and budgeting a training run before committing to it
Designing a Training System
One complete worked example — training a custom document-classification encoder from scratch — walked through every layer of the mechanics, with concrete numbers