Direct Preference Optimization (DPO)
- Training
- Standard term
Direct Preference Optimization (DPO) trains a model on pairs of answers, one preferred and one rejected, nudging it toward the preferred kind without first training a reward model, a separate model that scores answers.
For each pair, training raises how likely the model is to write the chosen answer compared with the rejected one. Both likelihoods are measured against a frozen copy of the model from before training, which keeps the model close to what it already did and limits what else training disturbs.
Where you’ll meet it
You're comparing open models, the kind anyone can download and run, and read one model's release notes. They say it was tuned with DPO on a large set of preference pairs to make its answers more helpful.
What to do
Someone picked the preferred answer in every pair, so the model leans toward their taste. Look in the notes for who picked, such as paid raters or another AI, and for results on tasks like yours. If they don't say, try it on a few of your own prompts first.
Easy to mix up
RLHF trains a reward model first, which then scores fresh answers as the main model trains. DPO skips that step and learns from pairs of answers ranked in advance.
Check yourself
A team builds DPO pairs by taking a large model's answer to each question as the preferred one and a small model's as the rejected one. The large model's answers also tend to run twice as long. What will tuning likely teach?
Go deeperDPO rewrote the RLHF goal, and its variants loosen the recipe
Rafailov and colleagues introduced DPO in 2023 in “Your Language Model Is Secretly a Reward Model.” They showed that under the goal used in RLHF, reinforcement learning from human feedback, the reward can be written in terms of the model's own probabilities, so training reduces to sorting each chosen answer above its rejected partner. A setting called “beta” sets how strongly the frozen copy pulls back.
Pairs written by people or other models may hold answers the tuned model would rarely write. Iterative versions of DPO sample fresh answers from the current model, have them ranked, and train again. Other variants simplify further. KTO learns from single answers marked good or bad, with no pairs, and ORPO and SimPO drop the frozen reference copy.