Sparse autoencoder (SAE)
- Control
- Research term
A sparse autoencoder (SAE) is a separate model trained to rebuild another model's internal activity from a larger set of features, using only a few of those features at a time.
A comparison: Like matching paint colors with a few pigments from a big shelf: each mix uses a handful, and some colors no mix quite reaches.
A model's activity is the output of many small units, its neurons, and one neuron often responds to several unrelated things, so single neurons are hard to read. Limiting each rebuild to a few features from a much larger set pushes each feature toward one narrower kind of input, which is easier to name.
Where you’ll meet it
You open a public website where researchers have posted the features an SAE found in a freely downloadable model. One feature is labeled “legal language,” and its top examples are all contract clauses. Lower down, a list of weaker examples shows it also firing on recipe instructions.
What to do
Scroll through the weaker examples before trusting the label, which was written from the top ones. A feature that fires on both contracts and recipes may be tracking something they share, such as numbered steps, so look for one description that fits every example.
Easy to mix up
A probe is trained to detect one concept that someone chose in advance. A sparse autoencoder is told nothing of the kind, and finds its own features in the model's activity.
Check yourself
A paper reports an SAE with a million features and notes that 40% of them never switched on across ten million test inputs. What does that most likely mean?
Go deeperWhat the rebuild misses can be large, and probes sometimes do better
SAEs scaled from small test models to production models in 2023 and 2024, in Anthropic's “Scaling Monosemanticity” work on a Claude model and in similar work by OpenAI and Google DeepMind on their own models.
Researchers measure what the rebuild misses by putting the rebuilt activity back into the model in place of the original. The model often gets noticeably worse at predicting text, and whatever it computes in the part left out has no feature to read.
Results on practical tasks are mixed. Google DeepMind's interpretability team reported that simple probes beat SAE features at spotting harmful intent in requests, and moved some of its research to other methods.
Sources
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Anthropic
- Toy Models of Superposition, Anthropic Transformer Circuits