Reinforcement Learning from Human Feedback (RLHF)

Using human feedback to define a reward signal and reinforcement learning to update a model toward higher-reward behavior.

A common language-model pipeline collects human comparisons, trains a reward model to predict them, and optimizes the policy against that reward, usually with additional constraints. RLHF can also refer more broadly to reinforcement learning informed by human ratings or demonstrations. Dataset design, rater instructions, reward modeling, and optimization all affect the outcome.

Builder example

RLHF can influence instruction following, tone, refusal, and preference alignment, but a product observer cannot trace one behavior to raters alone. Pretraining, supervised data, system instructions, decoding, and later safety work also contribute. Evaluate the behavior rather than reverse-engineering an undocumented pipeline from style.

Raters prefer a fluent answer over a cautious one, even when the cautious answer is more accurate.

Product teams still need factual evals, calibration checks, and domain review.

Common confusion: Human preference is neither a direct truth label nor one stable objective. Ratings vary by rubric and population, and reward models can generalize or be exploited in ways the original raters did not intend.