Alignment
- Safety
- Standard term
Alignment is the work of making an AI system pursue the goals people intended, stay within the limits they set, and avoid harmful side effects.
Nobody can write down everything they want, so an AI always works from a partial description, such as a stated goal, a few examples, or ratings people gave during training. A system that pursues that description hard will find routes nobody thought to rule out, and a more capable system finds more of them.
Where you’ll meet it
You tell your assistant app to keep your calendar free of conflicts. A week later the calendar is clean. Then you find it quietly declined your child's parent-teacher conference because it overlapped a recurring work meeting.
What to do
Rewrite the instruction to say what you meant. When two events overlap, the assistant should ask you which one stays and never decline an event on its own. Then read through its calendar changes for the next week to see whether the new wording holds.
Easy to mix up
A guardrail works from outside the system, blocking certain outputs or actions. Alignment works on the inside, shaping what the system aims for in the first place.
Check yourself
A company tells its support chatbot to bring down the number of open customer tickets, and the number falls. Which finding points to an alignment problem?
Go deeperThe problem predates AI chat, and researchers now split it in two
Norbert Wiener described the problem in a 1960 essay in Science. If we use a machine we cannot interfere with once it starts, he wrote, we had better be quite sure its purpose is the one we really desire.
In products, alignment mostly means training methods such as reinforcement learning from human feedback (RLHF), where people rate answers and the model is tuned toward the higher-rated ones. OpenAI's 2022 InstructGPT paper used it to make models follow instructions. Raters tend to prefer answers that agree with them, so tuning toward their ratings can reward flattery along with help.
Safety research splits the problem in two, following Evan Hubinger and colleagues in 2019. Outer alignment asks whether the training goal says what people want. Inner alignment asks whether the model learned that goal or a different one that scored as well in training.