Alignment & DPO Math
The Bradley-Terry model of preference, why classic RLHF needs a separate reward model, and the full derivation of DPO's reward-model-free loss
Alignment & DPO Math
TL;DR
Supervised fine-tuning teaches a model to imitate examples; alignment teaches it to prefer better outputs among options it can already produce, using pairs of (chosen, rejected) responses. DPO's key trick is a closed-form relationship between a policy's probabilities and an implicit reward, which lets a preference loss be written directly in terms of the model's own probabilities — with no separate reward model, and no reinforcement learning loop, required at all.
| Property | Value |
|---|---|
| Level | Advanced |
| Reading time | ~22 minutes |
| Prerequisites | Knowledge Distillation Mechanics |
| You will understand | The Bradley-Terry preference model, why classic RLHF is expensive to tune, and the full derivation behind DPO's loss |
The Problem SFT Can't Solve
Supervised fine-tuning (SFT) trains a model to reproduce example outputs — it's imitation learning. But often what you actually want isn't "imitate this exact response," it's "given two responses this model can already produce, prefer the better one." SFT has no mechanism for that: it only ever sees single examples to imitate, never a comparison between two options.
A model can be perfectly capable of writing a great response and a mediocre one, and SFT alone gives it no signal about which it should prefer. Preference data — pairs of (chosen, rejected) responses to the same prompt — is what closes this gap, and how to actually learn from that data is what the rest of this page derives.
The Bradley-Terry Model of Preference
The Bradley-Terry model gives a probabilistic way to relate pairwise preferences to underlying "quality" scores. If response y_w (winner) is preferred over y_l (loser), Bradley-Terry says the probability of that preference is a logistic function of the difference in their latent reward scores:
P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )
where σ(z) = 1 / (1 + e^(-z)) — the logistic sigmoidIn plain terms: the bigger the reward gap between the two responses, the more confidently the model should predict y_w gets preferred — and if the two responses have equal reward, the model predicts a coin-flip, exactly matching intuition.
Classic RLHF: Two Stages, Two Real Costs
The original alignment recipe uses Bradley-Terry in two separate stages:
Classic RLHF pipeline
1. Train a reward model
On preference pairs, using a Bradley-Terry-based loss: maximize log σ(r(x, y_w) - r(x, y_l)) over the dataset
2. Optimize the policy against it
Run PPO (a reinforcement learning algorithm), using the trained reward model to score generations and update the policy toward higher-reward outputs
reward_loss = -F.logsigmoid(reward_model(x, y_w) - reward_model(x, y_l)).mean()This works, but it's genuinely expensive to get right: you need to train and maintain two models (policy and reward model), run an RL loop (PPO) that's notoriously sensitive to hyperparameters, and guard against reward hacking — the policy finding ways to score highly on the reward model's proxy objective without actually producing better outputs, since the reward model is an imperfect stand-in for real human preference.
The DPO Insight: Skip the Reward Model Entirely
DPO's derivation starts from a standard RL objective: maximize expected reward while staying close (in KL divergence) to a reference policy π_ref — this KL constraint is what prevents the optimized policy from drifting arbitrarily far from sensible, coherent behavior:
max_π E_{y~π}[ r(x, y) ] − β · KL( π(y|x) ‖ π_ref(y|x) )The key mathematical fact DPO relies on: this exact objective has a known closed-form solution — the optimal policy π* that maximizes it is:
π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r(x, y) / β )where Z(x) is a normalizing constant. Rearranging this relationship solves for the reward in terms of the (optimal) policy:
r(x, y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)The substitution that makes DPO work
This says the reward is expressible directly in terms of policy probabilities — an "implicit reward." Substitute this expression for r(x, y) back into the Bradley-Terry preference probability from earlier. Critically, the β · log Z(x) term depends only on the prompt x, not on which response (y_w or y_l) is being scored — so when you take the difference r(x, y_w) - r(x, y_l) that Bradley-Terry needs, that term cancels out exactly, leaving an expression that never requires knowing Z(x) — which is exactly the part of the reward model that would otherwise be intractable to compute directly.
The DPO Loss, Term by Term
After the substitution and cancellation, the Bradley-Terry preference loss becomes DPO's final loss, expressed purely in terms of the policy being trained and a frozen reference copy:
L_DPO = -log σ( β · [ log(π(y_w|x)/π_ref(y_w|x)) − log(π(y_l|x)/π_ref(y_l|x)) ] )def dpo_loss(policy_chosen_logp, policy_rejected_logp,
ref_chosen_logp, ref_rejected_logp, beta):
chosen_ratio = policy_chosen_logp - ref_chosen_logp
rejected_ratio = policy_rejected_logp - ref_rejected_logp
return -F.logsigmoid(beta * (chosen_ratio - rejected_ratio)).mean()| Term | Meaning |
|---|---|
log(π(y_w|x)/π_ref(y_w|x)) | How much more (or less) likely the current policy makes the chosen response, relative to the frozen reference |
log(π(y_l|x)/π_ref(y_l|x)) | The same ratio for the rejected response |
| The difference of the two ratios | The loss pushes this to be large and positive — the policy should increase its relative preference for the chosen response and decrease it for the rejected one, both measured against the same fixed reference point |
β | Controls how sharply the loss penalizes preference violations, and implicitly how far the policy is allowed to drift from π_ref before being pulled back — a direct descendant of the same β in the original KL-constrained objective |
No reward model appears anywhere in this final loss — that's the entire point of the derivation, not an approximation of it. The reward was expressed in terms of policy log-probability ratios, substituted into Bradley-Terry, and the intractable normalizing term cancelled exactly because it doesn't depend on which response is being scored. What's left is a loss computable directly from the policy model and a frozen copy of itself — no second model, no RL loop, no reward-hacking surface to defend against.
Concept Checks
Check yourself
Why can't supervised fine-tuning alone teach a model to prefer one of two responses it's already capable of producing?
Because SFT only ever trains on single examples to imitate — it has no training signal that compares two candidate outputs and says one is better. Preference learning requires paired (chosen, rejected) data and a loss built specifically to increase the model's relative preference for the chosen one, which SFT's imitation objective doesn't provide.
What are the two real costs of the classic two-stage RLHF approach that DPO is specifically designed to avoid?
Training and maintaining a separate reward model in addition to the policy, and running a reinforcement learning loop (PPO) that's sensitive to hyperparameters and vulnerable to reward hacking, where the policy learns to exploit the reward model's imperfections rather than genuinely improve. DPO avoids both by deriving a loss expressed purely in policy probabilities, with no reward model or RL loop needed.
Why does the Z(x) normalizing term disappear from the final DPO loss instead of needing to be computed?
Because Z(x) depends only on the prompt x, not on which specific response is being scored, so it's identical whether you're evaluating the chosen or the rejected response for that prompt. When Bradley-Terry's preference probability is computed as a difference between the two responses' implicit rewards, that shared term cancels out exactly — which is precisely why it never has to be computed, sidestepping what would otherwise be an intractable quantity.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Bradley-Terry model | Preference probability as a logistic function of the difference in latent reward scores |
| Classic RLHF | Train a reward model on preferences, then optimize the policy against it with PPO |
| Reward hacking | The policy exploiting the reward model's imperfections instead of genuinely improving |
| KL-constrained objective | Maximize reward while staying close to a reference policy — the objective DPO's derivation starts from |
| Implicit reward | The closed-form relationship expressing reward directly in terms of policy log-probability ratios |
| Z(x) cancellation | The intractable normalizing term drops out because it's identical for both responses to the same prompt |
| β | Controls how sharply preference violations are penalized, and how far the policy may drift from the reference |
| DPO's core result | A preference loss computed purely from policy and reference log-probabilities — no reward model needed |
Next
With alignment's math covered, the next page turns to running training itself across multiple GPUs — the actual mechanics of DDP and FSDP2: Distributed Training Internals.
Knowledge Distillation Mechanics
The full distillation loss — temperature-scaled soft targets, KL divergence, and why the loss needs a T² correction term
Distributed Training Internals
The real mechanics of DDP and FSDP2 — what all-reduce and all-gather actually do, why sharding trades memory for communication, and how mixed precision avoids losing updates to rounding