Direct Preference Optimization (DPO)

A preference-optimization objective that updates a policy directly from chosen and rejected response pairs relative to a reference model.

DPO derives a classification-style loss from the same preference-modeling setup that motivates some RLHF pipelines. It avoids training an explicit reward model and running a separate on-policy reinforcement-learning loop, which can simplify implementation. Results still depend on hyperparameters, reference policy, data distribution, and how preferences were collected.

Builder example

DPO and variants are common in open-model post-training tools. Availability does not make DPO the best method for every dataset, and preference pairs can encode label noise, stylistic shortcuts, or missing task coverage. Compare against supervised and other preference baselines.

You have examples of answers your users prefer and answers they reject.

Direct Preference Optimization (DPO) can tune the model toward that preference pattern, while factual checks remain separate.

Common confusion: The simpler training process does not eliminate value judgments. Someone still has to decide which response in each pair is the preferred one, and those choices shape model behavior just as much as reinforcement learning from human feedback (RLHF) choices do.