Alignment with TRL
SFT, reward modeling, DPO, and GRPO — what each stage past supervised fine-tuning teaches a model, and which one your data actually supports
Alignment with TRL
TL;DR
Supervised fine-tuning teaches a model a format or style from examples, but it can't teach "this response is better than that one" — for that you need preference data. TRL (Transformer Reinforcement Learning) provides a trainer for each stage past SFT: RewardTrainer for scoring preferences, DPOTrainer for optimizing against them directly, and GRPOTrainer for tasks with a checkable, programmatic reward. DPO replaced classic reward-model-plus-PPO RLHF for most teams — it optimizes the same objective with one trainer and nothing else to babysit.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Evaluation |
| You will understand | What SFT can't teach, why DPO mostly replaced reward-model RLHF, and when GRPO's RL approach still earns its cost |
What SFT Can't Teach
Supervised fine-tuning (see Fine-Tuning with PEFT, or trl's SFTTrainer for instruction-formatted data specifically) trains on input–output pairs: given this prompt, produce this response. That's enough to teach a format, a tone, a domain vocabulary, or a new skill demonstrated by example.
It is not enough to teach preference — that one acceptable response is better than another acceptable response. SFT has no mechanism for "both of these are plausible outputs, but the second one is what we actually want," because every training example is a single point, not a comparison.
A model that's only been SFT'd tends to reflect the average of its training examples, not the best of what it's capable of. Preference-based training is how you push a model toward the better end of its own output distribution rather than just imitating demonstrations.
The Progression
From demonstrations to preferences to verifiable correctness
Reward modeling and classic RLHF
RewardTrainer trains a model to score which of two responses (chosen vs rejected) is better, from human- or model-labeled preference pairs. The original RLHF recipe then used that reward model as a stand-in judge and ran PPO — a reinforcement learning algorithm — to push the policy model toward higher-scoring outputs.
from trl import RewardConfig, RewardTrainer
trainer = RewardTrainer(model=reward_model, args=RewardConfig(output_dir="reward-out"), train_dataset=pref_ds)
trainer.train()This works, but it's expensive to get right: a separate model to train and maintain, and PPO is notoriously sensitive to hyperparameters — small changes can destabilize training in ways that are hard to diagnose.
DPO: the direct route
DPOTrainer (Direct Preference Optimization) skips the separate reward model entirely. It reformulates the same underlying preference objective so the policy model can be optimized directly against chosen/rejected pairs, in a single supervised-style training loop.
from trl import DPOConfig, DPOTrainer
trainer = DPOTrainer(
model=model,
args=DPOConfig(output_dir="dpo-out", report_to="trackio"),
train_dataset=pref_ds, # columns: prompt, chosen, rejected
)
trainer.train()Why DPO mostly replaced reward-model-plus-PPO RLHF. Same objective, one trainer, no separate reward model to train and no RL loop to stabilize. For the large majority of alignment work — "make responses more helpful, more concise, more on-brand, safer" — DPO gets there with a fraction of the moving parts. report_to="trackio" is TRL's natively integrated, recommended tracker for watching these runs (see Production & Operations).
GRPO: when the reward is a program, not a preference
GRPOTrainer (Group Relative Policy Optimization) is still an RL approach — but for a different kind of signal: a programmatic, verifiable reward function rather than static human preference pairs. Did the generated code pass its test suite? Does the final numeric answer match? These are checkable at training time, for arbitrarily many generated samples, without needing a human or a judge model in the loop.
def reward_fn(completions, **kwargs):
return [1.0 if passes_tests(c) else 0.0 for c in completions]
from trl import GRPOConfig, GRPOTrainer
trainer = GRPOTrainer(model=model, reward_funcs=reward_fn, args=GRPOConfig(output_dir="grpo-out"), train_dataset=task_ds)
trainer.train()GRPO earns its extra complexity specifically because the reward is verifiable — DPO's static preference pairs can't express "try many candidate solutions and reward the ones that actually work," but a programmatic reward function can be applied to as many generated attempts as you want.
Picking a Trainer From What You Have
Match the trainer to your data and your goal
You have labeled input→output examples, want a format/skill
RecommendedSFT. This is almost always the first and necessary step — DPO and GRPO both assume a model that already roughly knows the task and output format; they refine from there, they don't teach a task from nothing.
You have preference pairs (chosen vs rejected), want better quality/tone/safety
RecommendedSFT, then DPO. The standard modern recipe for most alignment work. One trainer past SFT, no separate reward model, stable to tune.
You have a way to programmatically check correctness (tests pass, math checks out)
SFT, then GRPO. Worth the RL complexity specifically because the reward is verifiable and can be computed on any number of generated attempts, not just a fixed set of labeled pairs.
You only have a handful of preference judgments and no way to check correctness programmatically
Reconsider whether alignment training is the right lever yet. A small preference set may be better spent as a held-out evaluation set (see Evaluation) than as training data — training on too few preference pairs risks overfitting to idiosyncrasies of that small set.
Alignment on Top of PEFT
TRL integrates directly with peft — every trainer above accepts a LoraConfig (or an already-wrapped PEFT model) the same way Trainer does. DPO or GRPO on top of a LoRA adapter is the common path in practice, not the exception; full-parameter alignment training is rarely necessary and carries the same memory cost problems covered in Fine-Tuning with PEFT.
from peft import LoraConfig
trainer = DPOTrainer(
model=model,
args=DPOConfig(output_dir="dpo-out"),
train_dataset=pref_ds,
peft_config=LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"]),
)Concept Checks
Check yourself
Why can't SFT alone teach a model that one acceptable response is better than another?
SFT trains on isolated input-output pairs — each example is a single point to imitate, with no comparison built in. Teaching "this response is better than that one" requires the training signal to actually contain a comparison, which is exactly what preference pairs (chosen vs rejected) provide and plain demonstration pairs don't.
What does DPO remove from the classic RLHF recipe, and why does that make it easier to run?
It removes the separate reward model and the PPO reinforcement learning loop, reformulating the same preference objective so the policy model can be optimized directly against chosen/rejected pairs in one training loop. Fewer moving parts means fewer places for training to become unstable, and no second model to train, evaluate, and keep in sync with the policy.
A team has a large set of math problems with verifiable final answers. Why might GRPO be a better fit than DPO here, despite DPO being the simpler trainer?
DPO needs static preference pairs — a fixed "this one is better" judgment made in advance. A programmatic reward function can instead be applied to any number of freshly generated attempts at training time, checking each one's actual correctness. That's a stronger and more scalable signal for a task where correctness is checkable, which is exactly what GRPO is built to exploit.
A team skips SFT and tries to go straight to DPO on a base model that's never seen the target task's format. What typically goes wrong?
DPO refines preferences between outputs the model can already roughly produce — it doesn't teach a task or output format from nothing. Without SFT first, the model may not reliably produce outputs in the right shape at all, so there's no coherent behavior for DPO's preference signal to sharpen. SFT establishes the baseline capability that later stages refine.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| SFT | Teaches format/style/skill from input-output demonstration pairs |
| Preference pairs | Chosen vs rejected response pairs — the data SFT can't learn from |
RewardTrainer | Trains a model to score which of two responses is better |
| Classic RLHF | Reward model + PPO — the original, expensive-to-tune recipe |
| DPO | Optimizes directly against preference pairs — no separate reward model, one trainer |
| GRPO | RL against a programmatic, verifiable reward — for checkable-correctness tasks |
report_to="trackio" | TRL's recommended, natively integrated experiment tracker |
| PEFT + TRL | DPO/GRPO on a LoRA adapter is the common path, not full-parameter training |
Next
Every trainer here assumes you can fit the model and its training state on your hardware — the next page is about what to do when you can't: Distributed Training.
Evaluation
Metrics with the evaluate library, why a leaderboard score isn't your score, building a held-out test set, and LLM-as-judge for generative tasks
Distributed Training
accelerate as the abstraction over DDP, FSDP, and DeepSpeed — when a single GPU is enough, and how to rent one instead of owning a cluster