Reasoning / Standard term
Eval (evaluation)
A repeatable evaluation of an AI system against defined cases, criteria, and scoring or review procedures.
An eval defines what is being tested, supplies representative inputs, records the system configuration, and judges results against explicit criteria. Some tasks have exact answers and automated scorers. Open-ended tasks may use rubrics, blinded human comparison, model graders, or downstream outcome data. A threshold is useful only when the score is calibrated to the decision it controls.
Builder example
Repeatable evals make prompt, model, retrieval, and workflow changes comparable. Their evidence is bounded by case coverage and judge quality, so production monitoring, incident review, and user outcomes still matter. A high aggregate score can hide a serious subgroup or edge-case failure.
Common confusion: A benchmark is a shared eval used for comparison across systems. A product eval is usually narrower and private. Neither automatically predicts the live workflow if its data, scoring, or operating conditions differ from production.