Production & Operations
Checkpointing discipline, honest reproducibility, experiment tracking, and budgeting a training run before committing to it
Production & Operations
TL;DR
A checkpoint that only saves model weights, not optimizer state, can't correctly resume training. Bit-for-bit reproducibility across different hardware or library versions is usually not actually achievable — set that expectation honestly rather than promising it. Track every run's hyperparameters and metrics from the first one, and estimate a multi-day training run's cost before committing to it, not after.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~16 minutes |
| Prerequisites | Evaluating & Debugging Training |
| You will understand | How to checkpoint correctly, what reproducibility can and can't guarantee, and how to budget a training run |
Area by Area
| Area | The thing you must get right |
|---|---|
| Checkpointing | Save model and optimizer state, not just weights |
| Reproducibility | Seed everything, but be honest about what that does and doesn't guarantee |
| Experiment tracking | Log hyperparameters and metrics from the first run, not the tenth |
| Cost/time budgeting | Estimate a run's duration and cost before committing to it |
| Version pinning | Framework/CUDA/library versions can shift numerical behavior between versions |
Checkpointing: Save the Optimizer State Too
A checkpoint that saves only model.state_dict() cannot correctly resume training. Adam-family optimizers maintain per-parameter running estimates (the first and second moments of the gradient) that took the whole run so far to build up. Resuming from weights alone means restarting those estimates from zero — the optimizer effectively "forgets" the momentum it had accumulated, which can produce a visible bump or instability in the loss curve right after resuming, even though the weights themselves picked up exactly where they left off.
torch.save({
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"step": step,
"scheduler": scheduler.state_dict(),
}, f"checkpoint-{step}.pt")Saving step and the learning-rate scheduler's state matters for the same reason — resuming a cosine or warmup schedule from the wrong point silently changes the effective learning rate the run continues with.
Reproducibility: What Seeding Actually Buys You
import torch, random, numpy as np
def seed_everything(seed):
torch.manual_seed(seed)
random.seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed_all(seed)Seeding makes a run reproducible on the same hardware, with the same library versions, with the same non-deterministic-op settings — it's a real and useful guarantee within that scope.
Bit-for-bit reproducibility across different GPUs, different CUDA versions, or different library versions is usually not actually achievable, and it's worth saying that plainly rather than promising it. Floating-point operations can be reordered differently by different hardware or kernel implementations, and those tiny differences compound over a long training run. The honest, achievable goal is reproducibility within a fixed environment — which is exactly why version pinning (below) matters as much as seeding does.
Experiment Tracking From the First Run
The run you don't bother logging is, with near certainty, the one you'll need to explain or reproduce later. Track, at minimum:
run_id, timestamp, git commit hash
hyperparameters: learning_rate, batch_size, model config, seed
metrics per step/epoch: train_loss, val_loss, gradient_norm
final artifact: which checkpoint, and its evaluation numbersA "quick experiment" that isn't logged has a way of turning into the config the final model was actually trained with, and then nobody can say exactly what it was. Logging costs a few lines of code; reconstructing an untracked run's exact configuration after the fact can cost a full day, if it's possible at all.
Budgeting Before You Commit
Before starting a multi-hour or multi-day run, estimate its actual cost from a short measured sample rather than guessing:
1. Run a few hundred steps, measure throughput (steps/sec or tokens/sec)
2. total_steps = dataset_size × epochs / batch_size
3. estimated_time = total_steps / measured_throughput
4. estimated_cost = estimated_time × (GPU-hours × $/GPU-hour)A five-minute measurement before committing to a three-day run is one of the highest-leverage things you can do — it turns "this training run should take a while" into a number you can actually plan around, or catch as unexpectedly expensive before spending the compute.
Version Pinning
Numerical behavior — not just API surface — can shift between PyTorch, CUDA, and cuDNN versions, as kernel implementations change. Pin exact versions for anything you intend to compare results across (an ablation study, a "did this change actually help" comparison), or an apparent improvement can turn out to be a version-induced numerical shift instead of a real effect from the change you made.
Concept Checks
Check yourself
Why does resuming training from a checkpoint that only has model weights sometimes cause a visible bump in the loss curve?
Because Adam-family optimizers maintain per-parameter running estimates built up over the whole run so far, and a weights-only checkpoint loses that state, forcing the optimizer to restart those estimates from zero. The weights pick up exactly where they left off, but the optimizer's "memory" of the training dynamics does not, which can produce visible instability right after resuming.
Why is it more honest to describe seeding as guaranteeing reproducibility 'within a fixed environment' rather than reproducibility in general?
Because floating-point operations can be reordered or computed slightly differently across different hardware, CUDA versions, or library versions, and those small differences compound over a long run into a genuinely different result — seeding alone doesn't control for that. The achievable guarantee is reproducibility given the same hardware and library versions, which is why version pinning matters alongside seeding, not instead of it.
Why measure throughput on a few hundred steps before committing to a multi-day training run, rather than just starting it?
Because a short measured sample turns a guess about total time and cost into an actual number you can plan around or reconsider — extrapolating steps-per-second to the full dataset and epoch count catches an unexpectedly slow or expensive run before days of compute have been spent on it, rather than after.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Checkpoint model + optimizer state | Weights-only checkpoints lose Adam's accumulated per-parameter estimates |
| Seed for within-environment reproducibility | Cross-hardware/version bit-for-bit reproducibility usually isn't achievable |
| Track from the first run | The unlogged "quick experiment" often becomes the run that mattered |
| Budget before committing | Measure throughput on a short sample, extrapolate cost, before a multi-day run |
| Version pinning | Numerical behavior shifts across framework/CUDA versions, not just APIs |
Next
With the operational discipline in place, the next page catalogues what goes wrong anyway, and how to trace it: Failure Modes & Debugging.
Evaluating & Debugging Training
Reading loss curves as a real diagnostic skill, systematically tracing NaN loss to its cause, and the mechanics behind vanishing gradients and dead ReLUs
Failure Modes & Debugging
Symptom to cause across the whole deep learning stack — the classic bugs that look like mysteries but have mechanical, checkable causes