A pre-trained language model has read an enormous amount of text and can continue almost anything. That does not make it a helpful assistant. It might continue a question with more questions, produce something harmful because similar text existed online, or ramble when a short answer is needed. Preference tuning is the stage of training that shapes a capable model into one that behaves the way people want.
The two best-known methods are RLHF (reinforcement learning from human feedback) and DPO (direct preference optimisation). This explainer covers how each works, how they compare, the newer variants, common failure modes and what it all means if you are considering preference-tuning a model yourself.
Where preference tuning fits
A typical modern training pipeline has three stages:
- Pre-training on large amounts of text to learn language and knowledge.
- Supervised fine-tuning (SFT) on examples of good responses: instructions paired with high-quality answers.
- Preference tuning on comparisons: for a given prompt, which of two responses is better?
Preferences are often easier to collect than perfect demonstrations. People can reliably say which of two answers is more helpful even when they could not easily write the ideal answer themselves.
RLHF: the classic recipe
RLHF was developed in a series of papers including Christiano et al. (2017), Stiennon et al. (2020) on summarisation, and Ouyang et al. (2022), "Training language models to follow instructions with human feedback," the InstructGPT paper. A notable result from that paper: human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter GPT-3, showing how much behaviour depends on this training stage.
The recipe has three steps:
- Collect comparisons. For many prompts, generate two or more responses and ask people which is better.
- Train a reward model. A separate model learns to predict a score that reflects human preferences: higher for responses people preferred.
- Optimise the policy with reinforcement learning. The language model generates responses, the reward model scores them, and an algorithm, typically PPO (proximal policy optimisation), updates the model to increase reward. A KL penalty keeps the model close to its SFT starting point, so it does not drift into strange outputs that happen to score well.
RLHF is powerful and flexible, but complex: several models in memory at once (policy, reference, reward and often a value model), sensitive hyperparameters and unstable training.
DPO: preferences without a separate reward model
Rafailov et al. (2023), "Direct Preference Optimization: Your Language Model is Secretly a Reward Model," showed that the RLHF objective with a KL penalty has a closed-form relationship between the optimal policy and the reward. That means you can skip training a separate reward model and the RL loop entirely, and optimise the language model directly on preference pairs with a simple classification-style loss.
Intuitively, DPO increases the probability of the preferred response and decreases the probability of the rejected one, relative to a frozen reference model, with a parameter (β) controlling how far the model may move from the reference.
For each (prompt, chosen, rejected):
margin = β × [ (log p_model(chosen) − log p_ref(chosen))
− (log p_model(rejected) − log p_ref(rejected)) ]
loss = −log sigmoid(margin)DPO is much simpler to implement and run, needs fewer models in memory and is generally more stable. That simplicity made it very popular, especially for open-model fine-tuning.
Side by side
| Aspect | RLHF (reward model + PPO) | DPO |
|---|---|---|
| Separate reward model | Yes | No |
| Training loop | Online RL: generate, score, update | Offline: train on fixed preference pairs |
| Models in memory | Policy, reference, reward, often value | Policy and reference |
| Complexity and stability | High; sensitive to tuning | Lower; more stable |
| Uses fresh samples during training | Yes | Not in the basic form |
| Flexibility | Reward model can score any new output | Limited to the preference data provided |
The online nature of RLHF is a real advantage: the model is trained on its own current outputs, which can help it improve beyond the quality of the original preference data. Many practical pipelines combine approaches, or run DPO iteratively with fresh preference data generated from the latest model.
Newer variants and AI feedback
The field has produced many variants:
- IPO (identity preference optimisation) addresses some overfitting issues in DPO.
- KTO (Ethayarajh et al., 2024) learns from individual good or bad labels instead of pairs, which can be easier to collect.
- ORPO (Hong et al., 2024) combines supervised fine-tuning and preference optimisation in one step.
- RLAIF and Constitutional AI (Bai et al., 2022) use AI-generated feedback guided by written principles, reducing dependence on human labels for some behaviours.
Reasoning-focused models also use reinforcement learning with verifiable rewards, such as whether a maths answer is correct or code passes tests, which avoids some problems of learned reward models; see chain of thought and self-consistency for background on reasoning.
Failure modes
- Reward hacking: the model finds ways to score highly that people would not actually prefer, such as excessive length, flattery or confident tone. Length bias is a well-documented example.
- Sycophancy: preference data that rewards agreeable answers can teach models to agree with users even when they are wrong.
- Over-refusal: strong safety preferences can make models refuse harmless requests.
- Preference data quality: noisy, inconsistent or biased labels are learned faithfully. The same calibration discipline used for model graders applies to human and AI labellers.
- Distribution shift: preferences collected on one kind of prompt may not generalise to others.
Collecting good preference data
Whatever method you use, the data decides the outcome. Write labelling guidelines with concrete examples of better and worse responses, and include the criteria that matter for your product, such as accuracy, completeness, tone and policy compliance. Measure agreement between labellers on a shared sample; low agreement means the guidelines are unclear, and the model will learn the confusion. Control for length by including pairs where the shorter answer is better. Generate candidate responses from the model you intend to tune, so the preferences describe its actual behaviour, and refresh the data as the model improves.
Should you preference-tune your own model?
For most product teams, preference tuning is not the first lever. Prompting, retrieval and supervised fine-tuning, often with LoRA, solve most problems. Consider preference tuning when:
- you need to shift style or judgement in ways that are easier to compare than to demonstrate;
- you have, or can collect, thousands of reliable preference pairs from your domain;
- you have an eval set that measures the behaviour you want and checks for regressions.
DPO with LoRA adapters is the most accessible starting point. Always compare against the SFT-only model on your eval set, and check for length inflation and over-refusal. Preference tuning also pairs naturally with distillation: distil capability first, then tune behaviour.
Frequently asked questions
What is RLHF?
Reinforcement learning from human feedback is a training method in which people compare model responses, a reward model learns to predict their preferences, and the language model is then optimised with reinforcement learning to produce responses the reward model scores highly.
What is DPO and how is it different from RLHF?
Direct preference optimisation trains the model directly on pairs of preferred and rejected responses using a simple loss, without a separate reward model or reinforcement learning loop. It is simpler and more stable, while RLHF can learn from fresh samples during training.
Is DPO better than RLHF?
Neither is universally better. DPO is easier and cheaper to run and works well in many settings; online RL methods can achieve stronger results when well tuned. Many pipelines combine them or run DPO iteratively.
What is reward hacking?
Reward hacking occurs when a model learns to exploit weaknesses in the reward signal, for example by producing longer or more flattering answers, rather than genuinely improving the behaviour people care about.
