Evaluating & Debugging Training
Reading loss curves as a real diagnostic skill, systematically tracing NaN loss to its cause, and the mechanics behind vanishing gradients and dead ReLUs
Evaluating & Debugging Training
TL;DR
A loss curve is a diagnostic instrument, not just a number that should go down — its shape tells you whether the model is overfitting, undertrained, or fighting a learning rate that's wrong in a specific, identifiable direction. Most "mysterious" training failures — NaN loss, vanishing gradients, a model that stops improving — have a small, well-known set of mechanical causes, checkable in order.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Distributed Training Internals |
| You will understand | How to read a loss curve, and how to systematically trace training failures to their real cause |
Reading a Loss Curve
The shape of the training and validation loss curves, plotted together, is the single most informative diagnostic available during training.
What each shape actually tells you
Both curves dropping together
RecommendedHealthy. The model is learning generalizable patterns, not just memorizing the training set.
Training drops, validation plateaus then rises
Overfitting. The model has started memorizing training-specific noise instead of general patterns — this is the point to add regularization, get more data, or stop training.
Both curves still dropping steadily, training not yet complete
Undertrained, not broken — more steps, or a larger model, may still help. Don't mistake "still improving" for "failing."
Loss oscillates or spikes erratically
Learning rate too high relative to the batch size and loss landscape — the optimizer is overshooting the minimum on every step instead of descending toward it.
Loss decreases very slowly, smoothly, barely moving
Learning rate too low — training is technically working but will take far longer than necessary to reach a good result.
A loss curve without a validation curve next to it is only half a diagnostic. Training loss alone cannot distinguish "learning well" from "memorizing well" — those look identical on the training curve and only diverge on held-out data. Always plot both, from the first run.
Diagnosing NaN Loss
NaN loss looks alarming and unspecific, but it has a small set of usual causes, worth checking in a fixed order rather than guessing:
NaN loss triage
Check the learning rate
The single most common cause — an update large enough to push a weight to an extreme value, which then produces an overflow or an invalid operation downstream
Check for an unstable operation
log(0) or a negative number, division by a value that can hit zero, an unclamped exp() that can overflow — audit the loss function and any custom layers for these
Check gradient clipping is actually applied
Without it, a single unusually large batch or a temporarily bad region of the loss landscape can produce an update large enough to destabilize everything downstream
# The standard gradient-clipping line, easy to simply forget to add
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)Vanishing and Exploding Gradients
Backpropagation computes each parameter's gradient by multiplying local derivatives together, layer by layer, all the way back through the network via the chain rule. If those per-layer factors are consistently a bit less than 1, the product shrinks exponentially with depth; if consistently a bit more than 1, it grows exponentially.
gradient at layer 1 ≈ (local factor)^depth × gradient at the output
depth = 50, factor = 0.9 → 0.9^50 ≈ 0.005 (vanished — layer 1 barely learns)
depth = 50, factor = 1.1 → 1.1^50 ≈ 117 (exploded — layer 1's update is huge)| Problem | Practical fixes |
|---|---|
| Vanishing gradients | Residual (skip) connections give the gradient a path that bypasses the multiplicative chain; careful weight initialization; normalization layers keep activations in a well-behaved range |
| Exploding gradients | Gradient clipping caps the update magnitude directly; the same initialization and normalization fixes help prevent it in the first place |
This is a large part of why residual connections were such a significant architectural idea. By adding the layer's input directly to its output (output = layer(x) + x), the gradient has an unobstructed additive path straight back through the network, alongside the multiplicative path through the layer itself — so even if the multiplicative path's contribution shrinks toward zero, the gradient signal doesn't vanish entirely.
Monitoring Gradient Norms as a Practical Habit
Logging the global gradient norm (the combined magnitude of the gradient across all parameters) every step turns "training feels unstable" into something you can actually look at:
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
log(step=step, grad_norm=total_norm.item())| Gradient norm pattern | What it usually means |
|---|---|
| Stays in a stable, moderate range | Healthy training |
| Sudden spike | A bad batch, a numerical instability, or the early sign of an exploding-gradient episode |
| Trends toward near-zero over training | Vanishing gradients, or the model has genuinely converged — check which by watching the loss too |
A Dead ReLU
A ReLU unit outputs max(0, x). If a unit's pre-activation input ends up consistently negative — often from an unlucky initialization or too large an update early in training — its output is always zero, its gradient with respect to its input is always zero, and it never receives a learning signal again. It's not broken code; it's a unit that's permanently stopped participating in the network.
A model can train "successfully" with a meaningful fraction of its ReLU units dead — the loss still goes down, because the remaining live units still learn. It shows up as slightly worse capacity utilization than the parameter count suggests, not as an obvious failure. Smaller, careful initialization and a moderate learning rate reduce how often it happens; activation functions like GELU or Leaky ReLU that don't have a hard zero region sidestep the problem entirely.
Concept Checks
Check yourself
Training loss and validation loss are both dropping steadily. Is the model definitely fine?
It's the healthy pattern, but "still dropping" alone doesn't guarantee the run is finished or optimally configured — it means training hasn't yet reached the point where overfitting or convergence would show up. Keep watching; the diagnostic value is in catching the moment the validation curve stops following the training curve, not in a single snapshot.
What's the first thing to check when loss becomes NaN, and why that one first?
The learning rate — because it's both the most common cause and the cheapest to rule out, and an update that's too large is often what pushes a weight into a range where a downstream operation (like a log or an exponential) produces an invalid value in the first place. Checking it first avoids chasing a numerical-stability bug in the loss function when the real cause is upstream.
Why do residual connections specifically help with vanishing gradients, rather than normalization alone being enough?
Because vanishing gradients come from repeated multiplication through many layers shrinking the gradient exponentially with depth, and a residual connection adds an unobstructed additive path around that multiplicative chain. Even if the multiplicative path's contribution shrinks toward zero, gradient can still flow back through the addition, which normalization alone — which keeps activation scale well-behaved but doesn't add an alternate gradient path — doesn't provide on its own.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Loss curve shapes | Each pattern (healthy, overfitting, undertrained, LR too high/low) has a distinct, recognizable signature |
| Always plot validation alongside training | Training loss alone can't distinguish learning from memorizing |
| NaN triage order | Learning rate, then unstable operations, then missing gradient clipping |
| Vanishing/exploding gradients | Exponential shrink/growth from repeated multiplication through depth, via the chain rule |
| Residual connections | Give gradients an additive path around the multiplicative chain |
| Gradient norm monitoring | Turns "training feels unstable" into an actual logged, checkable signal |
| Dead ReLU | A unit stuck outputting zero, permanently cut off from further learning |
Next
With the diagnostic skills in place, the next page covers the operational discipline around a training run itself — checkpointing, reproducibility, and experiment tracking: Production & Operations.
Distributed Training Internals
The real mechanics of DDP and FSDP2 — what all-reduce and all-gather actually do, why sharding trades memory for communication, and how mixed precision avoids losing updates to rounding
Production & Operations
Checkpointing discipline, honest reproducibility, experiment tracking, and budgeting a training run before committing to it