Distributed Training
accelerate as the abstraction over DDP, FSDP, and DeepSpeed — when a single GPU is enough, and how to rent one instead of owning a cluster
Distributed Training
TL;DR
accelerate is the layer that runs one PyTorch training script across however many devices you point it at, without rewriting the loop per hardware target. Data parallelism (DDP) copies the model per GPU for more throughput; sharded strategies (FSDP, DeepSpeed ZeRO) split the model itself across GPUs when it doesn't fit on one. Reach for either only after a single GPU with LoRA/QLoRA and gradient checkpointing has actually run out of room — and consider renting GPU time with hf jobs before provisioning a cluster.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Alignment with TRL |
| You will understand | The difference between data parallelism and sharding, when each is warranted, and how to scale a training script without rewriting it |
One Script, Any Number of Devices
from accelerate import Accelerator
accelerator = Accelerator()
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
for batch in dataloader:
outputs = model(**batch)
accelerator.backward(outputs.loss)
optimizer.step()accelerator.prepare() wraps the model, optimizer, and dataloader so the same training loop runs correctly whether it's launched on one GPU, eight GPUs, or a multi-node cluster — no manual device placement, no conditional code paths for different hardware. Configuration lives outside the script entirely:
accelerate config # interactive, writes a config file once
accelerate launch train.py --your --usual --argsThe script itself never changes between a laptop GPU and a cluster — only the accelerate config answers do.
Three Strategies, One Decision
DDP, FSDP, or DeepSpeed ZeRO
DDP — Distributed Data Parallel
RecommendedA full copy of the model sits on every GPU. Each GPU processes a different batch slice in parallel, and gradients are synced (averaged) across GPUs after the backward pass. Use this when the model fits comfortably on one GPU and the goal is more throughput, not more memory.
FSDP — Fully Sharded Data Parallel
Shards the model's parameters, gradients, and optimizer state across GPUs, reassembling only the piece each GPU needs at the moment it's needed. Use this when the model does not fit on one GPU at all — its memory footprint is the actual constraint, not just training speed.
DeepSpeed ZeRO
Solves the same sharding problem as FSDP, with a different implementation and deeper configuration knobs — including CPU and NVMe offload for state that doesn't fit even across all available GPU memory. Reach for it over FSDP when you specifically need one of its extra features, not as a default.
| DDP | FSDP / DeepSpeed ZeRO | |
|---|---|---|
| What's split across GPUs | Nothing — full model copy on each | Parameters, gradients, optimizer state |
| Solves | Throughput (more data processed per second) | Memory (a model too big for one GPU) |
| Per-GPU memory | Same as single-GPU training | Falls as you add GPUs |
| Communication overhead | Lower — only gradients sync | Higher — parameters reassembled on demand |
Both FSDP and DeepSpeed are configured through accelerate too — FullyShardedDataParallelPlugin or a DeepSpeed config file, selected during accelerate config, with no change to the training loop itself.
Mixed Precision
Training (and most inference) runs in reduced precision by default now — bfloat16 over float16 on modern GPUs specifically, because bf16 keeps float32's wide exponent range (less risk of the numeric overflow/underflow that can silently corrupt a training run) while still halving memory versus float32. Accelerator takes mixed precision as a setting, applied consistently across whatever parallelism strategy is in use.
Don't Reach for This by Default
A single GPU goes further than people expect
Before adding a second GPU, check whether LoRA or QLoRA (see Fine-Tuning with PEFT) plus gradient_checkpointing=True actually can't fit the job. QLoRA in particular can put a model that needs tens of gigabytes in bfloat16 comfortably on one consumer GPU. Distributed training solves a real problem, but it's a problem worth confirming you actually have — via an out-of-memory error or a measured throughput ceiling — before paying its complexity cost.
Renting Instead of Owning
Not every team wants to own multi-GPU hardware for training that happens occasionally. hf jobs rents compute by the job — GPU or CPU, Docker-based — through the hf CLI, the huggingface_hub Python client, or an HTTP API:
hf jobs run --flavor a10g-large python:3.12 -- python train.py --your --argsChoosing where training actually runs
hf jobs is a practical middle step: it gives you access to larger single- or multi-GPU hardware for the duration of one run, without the commitment of owning a cluster or the complexity of standing up multi-node infrastructure for training that happens a handful of times a month.
Concept Checks
Check yourself
A team's model trains fine on one GPU but they want to finish faster. Which strategy fits, and why not the other one?
DDP — the constraint here is throughput, not memory, and DDP is built exactly for that: full model copies running in parallel on different data, gradients synced afterward. FSDP or DeepSpeed would add sharding and reassembly overhead to solve a memory problem this team doesn't have, making the job slower to set up for no benefit.
Why does the same accelerate-based training script run correctly on both a single GPU and an eight-GPU node with no code changes?
accelerator.prepare() handles device placement and any parallelism-specific wrapping internally, based on the configuration set up separately via accelerate config. The training loop only ever talks to the wrapped model, optimizer, and dataloader — it never needs to know how many devices are actually involved, which is exactly the abstraction accelerate is providing.
A team jumps straight to FSDP across four GPUs for a 3B-parameter model. What question should they have asked first?
Whether QLoRA on a single GPU could have handled it — a 3B model is well within range for single-GPU LoRA/QLoRA training with gradient checkpointing. Reaching for sharded multi-GPU training without first hitting an actual memory or throughput wall adds real complexity (configuration, communication overhead, more failure surface) to solve a problem that may not exist yet.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
accelerate | Runs one training script across any number of devices via accelerator.prepare() |
| DDP | Full model copy per GPU, gradients synced — solves throughput |
| FSDP | Shards parameters/gradients/optimizer state across GPUs — solves memory |
| DeepSpeed ZeRO | Same sharding goal as FSDP, different implementation, extra offload options |
| Mixed precision | bf16 preferred over fp16 on modern GPUs — wider exponent range, less overflow risk |
accelerate config / launch | Configure hardware target once, outside the script; launch without code changes |
| Single-GPU-first rule | Confirm LoRA/QLoRA + gradient checkpointing actually can't fit before scaling out |
hf jobs | Rent GPU time by the job — a middle step between one GPU and owning a cluster |
Next
A trained model is only useful once something can call it — the next page covers turning it into something running: Deployment & Inference.
Alignment with TRL
SFT, reward modeling, DPO, and GRPO — what each stage past supervised fine-tuning teaches a model, and which one your data actually supports
Deployment & Inference
Three shapes for turning trained weights into something callable — Inference Providers, Spaces, and Inference Endpoints — and how to pick between them