Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Process reward model vs Outcome reward model

  • Reasoning
  • Research term

An outcome reward model is a trained scorer that judges an AI's finished answer; a process reward model judges each intermediate step. Developers use both to train models and to pick among candidate answers.

An outcome reward model learns from finished answers labeled right or wrong. A process reward model learns from solutions in which each step was marked valid or invalid, so it can point to the step where the work went wrong. Step labels give more feedback per solution and are harder to define and collect.

Where you’ll meet it

You read a launch post for a new math model. It says a process reward model checked every step of the solutions it trained on, and suggests this makes the reasoning the model shows you trustworthy.

What to do

The claim is narrower than it sounds. During training, a scorer judged whether written steps looked valid. Steps can look valid without being how the model reached its answer, and that scoring happened before your question, so an answer you depend on still needs its own check.

Easy to mix up

Chain-of-thought is reasoning written out before the answer, often where you can read it. Process and outcome reward models score that work behind the scenes, and their scores stay with the developer.

Check yourself

Two solutions reach the same right number. One gets there through two wrong steps that cancel each other out. Which scorer was trained to mark that one down?

Go deeperStep labels helped on math, and any scorer can be gamed

OpenAI's 2023 paper “Let's Verify Step by Step” trained a process reward model on about 800,000 human judgments of individual solution steps. Used to pick the best of many candidate solutions to competition math problems, it chose correct ones more often than an outcome reward model did.

Where a program can check the final answer, such as code tests, training on that check is called RLVR (reinforcement learning with verifiable rewards). Those checks can be gamed too, as when a model's code recognizes the test inputs and returns the expected outputs. Against a learned scorer, a model may learn steps the scorer favors and careful readers reject.

Practice Modern AI

Or take the learning quiz on a mix of every set