Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Reinforcement Learning from AI Feedback (RLAIF)

  • Training
  • Standard term
2 steps

Reinforcement Learning from AI Feedback (RLAIF) trains a model using judgments from another AI system, which compares or scores answers against a rubric or written principles, in place of some ratings from people.

The judge's guide holds lines such as “prefer answers that explain risks plainly.” Thousands of its picks usually train a scoring model, a second model that learns to predict which answer the judge would choose, and that score becomes the reward the model in training learns to raise.

Where you’ll meet it

On a podcast during your commute, a startup founder says their new model was trained with AI feedback, so its answers are objective and free of human bias.

What to do

Hear “objective” as a sales word. People wrote the judge's guide and chose the judge, and AI judges have measured leanings of their own, such as favoring longer answers. Ask which principles the judge followed, and try the model on questions where you already know a fair answer.

Easy to mix up

In RLHF, new ratings arrive only as fast as people can read and compare answers. In RLAIF, a judge model does the rating, so ratings arrive as fast as that model can run.

Check yourself

A team will use an AI judge to compare 100,000 pairs of answers for training. Before they start, how can they find out whether its picks are any good?

Go deeperIn one study, people rated AI-feedback and human-feedback models about equally

Anthropic's 2022 Constitutional AI work used AI feedback for the harmlessness part of training. A 2023 Google study by Lee and colleagues found that people rated summaries and dialogue from RLAIF-trained models about as highly as those from models trained on people's ratings (RLHF).

Lee and colleagues also tested having the judge score answers directly during training, skipping the scoring model, which keeps rewards current as the model changes. A judge from the same family as the model it trains may share its blind spots, so mistakes both make can pass without penalty.

Practice Modern AI

Or take the learning quiz on a mix of every set