Reinforcement learning from human feedback (RLHF) was the technique that made instruction-following LLMs viable at scale, but its complexity — a separate reward model, a PPO training loop, careful KL penalty tuning — made it accessible mainly to large research teams. Direct Preference Optimization (DPO) emerged as a simpler alternative that achieves comparable alignment quality without the reinforcement learning machinery. Understanding what each approach actually does, where each fails, and when to use which is increasingly relevant for any team fine-tuning models for real-world deployment.
The Alignment Problem These Methods Solve
Pre-trained language models predict the next token based on patterns in training data. They’re not inherently helpful, harmless, or honest — they’re good at completing text in the style of whatever they were trained on. Alignment techniques fine-tune pre-trained models to follow instructions, refuse harmful requests, and produce outputs that humans prefer. The core challenge is that “what humans prefer” is difficult to specify as a loss function directly, which is why preference learning — training on comparisons between outputs rather than absolute quality scores — became the dominant paradigm.
All three methods (RLHF with PPO, DPO, and its variants) start from the same place: a dataset of preference pairs, where each example consists of a prompt, a preferred response (chosen), and a less-preferred response (rejected). The methods differ in how they use this data to update the model.
RLHF with PPO
Classic RLHF has two stages. First, a reward model is trained on the preference dataset to predict which response humans prefer. The reward model takes a prompt and a response as input and outputs a scalar score. Second, the policy (the language model being aligned) is optimized with PPO to maximize the reward model’s scores while staying close to the original pre-trained model via a KL divergence penalty.
The KL penalty is critical. Without it, PPO will quickly find responses that maximize the reward model’s scores but bear little resemblance to coherent language — reward hacking, where the policy exploits weaknesses in the reward model rather than genuinely improving. The KL penalty keeps the aligned model close to the pre-trained base, limiting how far it can drift. The strength of this penalty (beta) is a key hyperparameter: too small and reward hacking dominates, too large and the model barely moves from the base.
PPO is computationally expensive. It requires four models in memory simultaneously: the policy being trained, a frozen reference policy (for the KL penalty), the reward model, and a value function network used by PPO’s advantage estimation. For a 7B model, this means roughly 4x the memory of a single model inference — 56GB of just model weights before any optimizer states or activations. Running PPO stably requires careful hyperparameter tuning: learning rate, KL coefficient, clipping range, GAE lambda, and rollout length all interact, and instability in any of these can cause training to diverge or collapse.
Despite the complexity, PPO remains the strongest method for applications where the reward signal is rich and well-specified, and where the preference dataset covers diverse failure modes. The online nature of PPO — generating new responses with the current policy and learning from them — allows the model to explore and correct failure modes that aren’t represented in the static preference dataset. This is RLHF’s key advantage over offline methods like DPO.
Direct Preference Optimization (DPO)
DPO, introduced by Rafailov et al. in 2023, shows that the RLHF objective can be optimized directly without a separate reward model or reinforcement learning. The key insight is a mathematical reparameterization: rather than training a reward model and then using it to update the policy, DPO derives an implicit reward from the policy itself and optimizes a binary cross-entropy loss directly on the preference pairs.
The DPO loss increases the log-probability of chosen responses relative to rejected responses, weighted by how much the current policy already prefers each. The reference policy (a frozen copy of the SFT-fine-tuned model) provides the baseline for computing these relative probabilities. No reward model is needed — the preference signal is encoded directly into the policy update.
The practical advantages are significant. DPO requires only two models in memory (the policy being trained and the frozen reference), has no RL hyperparameters to tune, is stable across a wide range of learning rates and batch sizes, and trains in a standard supervised learning loop. A DPO fine-tuning run that would have required weeks of PPO experimentation to stabilize can be completed in a day with straightforward hyperparameters. For teams without dedicated RL engineering expertise, DPO is often the difference between alignment fine-tuning being feasible or not.
DPO’s limitation is that it’s offline: it learns only from the fixed preference dataset and never generates new responses to learn from. If the chosen and rejected responses in the dataset don’t cover a failure mode, DPO won’t fix it. Additionally, when the reference policy is already quite different from the responses in the preference dataset (common when using human-written preferred responses that the model couldn’t have generated), the DPO loss signal becomes noisy and training is less effective.
PPO vs DPO: When to Use Which
Use DPO when: you have a high-quality static preference dataset, your team doesn’t have RL engineering capacity, training stability is a priority, or you’re fine-tuning a model that’s already undergone SFT and you want to further improve its alignment on a specific preference dimension. DPO is the right default for most teams doing alignment fine-tuning in 2026 — the implementation is straightforward, the results are competitive with PPO on most benchmarks, and the reduced complexity is a genuine operational advantage.
Use PPO when: you need the model to improve on tasks where the preference dataset is sparse or incomplete, when you have a reliable automated reward signal (code execution results, factual verification, structured output validity), or when you’re operating at a scale where the engineering investment in a stable PPO pipeline is justified by quality improvements. The companies building frontier models use PPO (or variants) for good reasons — online RL allows the model to discover and correct failure modes that no static dataset covers.
DPO Variants Worth Knowing
IPO (Identity Preference Optimization) modifies the DPO objective to prevent overfitting on the preference dataset. DPO can overfit when chosen responses are assigned very high probability — the model memorizes the preferred responses rather than learning the underlying preference structure. IPO adds a regularization term that prevents the log-ratio of policy to reference from growing too large, producing better generalization on out-of-distribution inputs.
ORPO (Odds Ratio Preference Optimization) eliminates the need for a reference model entirely by incorporating a preference penalty directly into the SFT loss. A single training run simultaneously trains the model on supervised examples and preference pairs, making ORPO more memory-efficient than DPO (one model instead of two) and removing the dependency on a pre-trained SFT checkpoint. For resource-constrained fine-tuning where even two models in memory is a constraint, ORPO is worth evaluating.
SimPO (Simple Preference Optimization) replaces DPO’s reference model with a length-normalized reward based on the average log-probability of the response under the policy itself. This eliminates the reference model while also addressing DPO’s tendency to favor longer responses (which naturally have higher log-probabilities in the raw formulation). SimPO has shown strong results on alignment benchmarks at lower computational cost than DPO, and is increasingly being adopted in production fine-tuning pipelines.
Building a Preference Dataset
The quality of the preference dataset matters more than the choice of alignment algorithm. A clean, diverse dataset of 5,000–20,000 preference pairs typically outperforms a larger but noisier dataset regardless of whether you use DPO or PPO. Human annotation of preferences is the gold standard but expensive — budget 5–15 minutes per example for careful annotation. LLM-generated preferences using a strong judge model (GPT-4o comparing two responses from your model) are faster and cheaper but introduce the judge’s biases. A practical approach is to use LLM-generated preferences for the bulk of the dataset and human annotation for the highest-stakes examples and edge cases where the judge model is least reliable.
Preference datasets should cover the full distribution of inputs your model will see in production, not just the inputs you expect users to ask nicely. Include adversarial inputs, ambiguous instructions, sensitive topics, and the specific failure modes you’ve observed in your base model. A preference dataset that only covers the easy cases produces a model that aligns well on easy cases and fails on hard ones — which is usually the opposite of what you need.
Reward Model Quality Determines RLHF Ceiling
In classic RLHF, the quality of the reward model sets a ceiling on alignment quality that PPO cannot exceed. A reward model that’s biased toward verbose responses will train a policy that’s verbose; a reward model that can be fooled by confident-sounding text will train a policy that sounds confident regardless of accuracy. The reward hacking problem is fundamentally a reward model problem — PPO is optimizing exactly what the reward model scores, and if that doesn’t perfectly reflect human preferences, the policy will diverge from human preferences in proportion to how wrong the reward model is.
This is one of the practical arguments for DPO in resource-constrained settings. DPO implicitly defines a reward through the policy’s own log-probabilities, which means there’s no separate reward model to be wrong. The downside is that DPO’s implicit reward is less powerful and less flexible than an explicit reward model — it can only express the preferences encoded in the static dataset, not learn a generalizable preference function. For alignment tasks where the preference dataset is comprehensive and high-quality, DPO’s implicit reward is sufficient. For tasks requiring nuanced preference modeling across diverse input distributions, an explicit reward model trained on a large, diverse dataset is likely necessary.
Constitutional AI and RLAIF (RL from AI Feedback) approaches address the reward model quality problem by using a strong LLM as the reward signal rather than human annotations. A critique model evaluates policy outputs against a set of principles (the “constitution”) and the policy is trained to satisfy these principles. This scales better than human annotation and can produce reward signals that are more consistent and harder to hack than human-annotated reward models, at the cost of inheriting the biases of the AI judge. For teams without access to large-scale human annotation, RLAIF is an increasingly practical path to alignment fine-tuning.
Evaluating Alignment Quality
Measuring whether alignment fine-tuning actually worked requires evaluation beyond the training metrics. Loss curves and reward scores during training tell you that the model is optimizing the objective you gave it — they don’t tell you whether that objective matches actual human preferences on your target distribution. Post-training evaluation should include: win rate on held-out preference pairs (does the aligned model produce preferred responses more often than the base model?), safety evaluations on adversarial inputs not in the training set, and task performance on benchmarks that weren’t targeted by the alignment training to check for regression.
A common failure mode is alignment tax — degraded performance on downstream tasks after alignment fine-tuning. RLHF with strong KL penalty typically produces smaller alignment tax than weak KL penalty, because the model stays closer to the base. DPO with a small beta (low KL constraint) can produce significant alignment tax, especially when the preference dataset is small relative to the model’s parameter count. Monitor benchmark performance (MMLU, HumanEval, or whatever is relevant for your application) before and after alignment fine-tuning, and tune the KL coefficient or DPO beta to find the Pareto frontier between alignment quality and capability retention.