Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Reinforcement Learning from Human Feedback (RLHF)

  • Training
  • Standard term
2 steps

Reinforcement Learning from Human Feedback (RLHF) is a training method where people rate or compare a model's answers, and the model is then trained to give answers more like the ones they preferred.

In a common setup, the comparisons first train a separate “reward model” to predict which of two answers people would pick. The main model then writes answers, the reward model scores them, and training pushes it toward higher scores. Training also subtracts a penalty that grows as its answers drift from how it wrote before.

Whatever raters reliably favored becomes part of the score, including habits of tone and length they never meant to reward.

Where you’ll meet it

You ask a chat assistant whether you can take an allergy pill alongside your blood pressure medication. It says yes in a warm, assured paragraph, with no mention of your dose and no source.

What to do

Treat the confident tone as style, since it tells you nothing about whether anything was checked. Ask which interactions it considered and where that information comes from, then confirm with a pharmacist before taking the two together.

Easy to mix up

RLAIF has a model do the judging, so computing power limits how much feedback it gets. RLHF is limited by the hours of the people doing the comparing.

Check yourself

A team keeps training its model against one fixed reward model, round after round, and the scores keep climbing. What should the team expect from the answers people get?

Go deeperRLHF began with robots and games, and DPO drops the reward model

Paul Christiano and colleagues trained agents from human preferences in 2017, in simulated robots and video games. OpenAI's InstructGPT paper in 2022 applied the method to instruction-following language models, and the method was central to ChatGPT. Its last stage used proximal policy optimization (PPO), which caps how far each update moves the model, a second limit beside the drift penalty.

A 2023 Anthropic study of preference data found that an answer matching the person's stated views was more likely to be preferred, which helps explain sycophancy, a model's habit of telling people what they want to hear. Direct Preference Optimization (DPO) skips the separate reward model and trains the main model on the comparisons directly.

Practice Modern AI

Or take the learning quiz on a mix of every set