Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Activation steering / activation engineering

  • Control
  • Research term
4 steps

Activation steering is a research technique that changes a model's behavior while it runs, by adding a chosen direction to its internal activity that pushes it toward or away from a trait.

Inside the model, each word is held as a long list of numbers, and a direction is a fixed set of amounts added to that list. Researchers get one, a steering vector, by running the model on cautious and careless answers and taking the difference. That this works suggests the model holds traits as directions, which add together like arrows.

Where you’ll meet it

A research write-up says a lab made its model more honest by steering it, without retraining it. A chart shows the steered model scoring better on a set of test questions.

What to do

Read it as a promising result on those questions. A convincing post also shows the model on questions unlike the ones used to build the steering, and says whether it got worse at anything ordinary along the way.

Easy to mix up

Fine-tuning trains the model further, so the change stays in the model itself. Activation steering works only while the push is switched on; turn it off and the model behaves as before.

Check yourself

A lab finds a steering vector that makes its model less flattering, then adds the same vector to another company's model of the same size. What will it most likely do there?

Go deeperSeveral labs found the method in parallel, and the push needs dosing

Several research groups published versions of the idea around 2023, including activation addition (Turner and colleagues), representation engineering (Zou and colleagues), and contrastive activation addition (Rimsky and colleagues). Anthropic later used related “persona vectors” to monitor traits such as flattery and to keep training from strengthening them.

The researcher chooses which layer to push, meaning which stage of the model's step-by-step calculation, and how hard. A weak push changes little, and too strong a push makes answers incoherent.

A related method steers with features, patterns in those numbers that interpretability tools, programs built to look inside a model, have matched to a single idea. Anthropic's 2024 “Golden Gate Claude” turned up one such feature until the model brought the Golden Gate Bridge into almost every answer.

Practice Modern AI

Or take the learning quiz on a mix of every set