Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Representation engineering

  • Control
  • Research term
4 steps

Representation engineering is an umbrella term for methods that read or change a model's internal representations, the patterns of activity linked to concepts, to study or influence its behavior.

Researchers record the model's internal activity, long lists of numbers, on contrasting examples like honest and dishonest answers. The average difference between the two sets is called a “direction,” since it points from dishonest toward honest, as a compass bearing points across a map. Reading measures how far new activity points that way, and changing adds or subtracts it while the model runs.

Where you’ll meet it

A consultant pitches your company “representation engineering” to make its customer chatbot more honest. The chatbot runs on a model from a hosted provider, whose service takes your prompts and sends back replies.

What to do

Ask the consultant how they would reach the model's internal activity. Unless your provider offers that access, they have no way to apply it to your chatbot, so ask what they would do instead.

Easy to mix up

Mechanistic interpretability starts from the network's small parts and works up to behavior. Representation engineering starts from a concept people name, such as honesty, and looks for the activity linked to it.

Check yourself

Researchers find a direction that a model's activity points along more strongly whenever its answer is false, on the prompts they tested. Which use does that finding alone support?

Go deeperDirections may track less than their concept, and pushing has side effects

Zou and colleagues named the approach in a 2023 paper, “Representation Engineering: A Top-Down Approach to AI Transparency.”

A direction found on one set of prompts may follow something narrower than the concept, such as the wording of those examples, so it needs testing on prompts unlike them. Pushing along a direction can also change unrelated behavior, and too strong a push makes the output incoherent.

Practice Modern AI

Or take the learning quiz on a mix of every set