Decision-maxxing
- Maxxing
- Practitioner slang
Decision-maxxing is improving an AI-assisted decision process by judging the evidence it uses, the choices it makes, and their consequences, rather than speed or volume alone.
Decision quality depends on the evidence available at the time, the cost of different mistakes, and what follows. Linking outcomes to earlier choices can reveal failures that speed and volume miss. A favorable outcome by luck does not by itself establish that the decision process was sound.
Where you’ll meet it
At a team meeting, the vendor of your hiring tool demonstrates its new version. It ranks job applicants twice as fast, and a chart of applicants processed per week climbs steeply. Nobody mentions how the people it ranked highly last year have done.
What to do
Ask what evidence would show that the rankings help interviewers choose well. Before signing, agree to follow hires through their first year using the same job-relevant measures as earlier hires, while recording changes in roles or applicants that could affect the comparison.
Easy to mix up
Eval-maxxing chases a higher test score, even when everyday usefulness stays flat. Decision-maxxing judges a process by the choices it leads to and what those choices then cause.
Check yourself
Your sales team starts using an assistant that ranks which potential customers to call first. Which number is more relevant evidence about decision quality, while still needing a fair comparison?
Go deeperYou only see outcomes for the options you chose
A lender normally sees repayment outcomes only for approved loans, leaving the outcomes of rejected applications unknown. Researchers call this the “selective labels” problem. Following your hires likewise leaves out people you rejected. Carefully designed trials or expert review of sampled rejections can add evidence, but neither reveals every unchosen outcome.
Calibration asks whether stated probabilities match observed frequencies across comparable cases. If cases assigned 90 percent probability succeed about nine times in ten, those estimates are calibrated. That helps distinguish a 55 percent estimate from a 95 percent one, but the appropriate action still depends on the costs of being wrong.
Sources
- The Selective Labels Problem, www.cs.cornell.edu
- Fair Decisions Despite Imperfect Predictions, arXiv