Feature (interpretability)
- Control
- Research term
In interpretability research, a feature is a pattern of activity inside a model that researchers link to something meaningful, such as a topic, a writing style, a named person, or a behavior.
Researchers record a model's activity on many inputs and look for patterns. With a probe, a small detector, they choose a concept first and train the probe to spot it. A sparse autoencoder, a second model that rebuilds the activity from many simpler parts, finds patterns unprompted, and researchers label them from the inputs that switch them on most.
Where you’ll meet it
A news story reports that an AI company found a “deception feature” in its model and now uses it to flag dishonest answers before users see them. The story shows a few answers that set it off.
What to do
Look for how the feature did as a detector on answers already labeled honest or dishonest: how many dishonest ones it caught, and how many honest ones it flagged by mistake. A few answers that set it off show only that it fires sometimes.
Easy to mix up
A product feature, such as dark mode, is a setting anyone can switch on. An interpretability feature is a pattern inside the model, which only researchers with access to its insides can adjust.
Check yourself
Researchers turn a “sarcasm” feature down in their model, and its replies become much less sarcastic. What does that result show about the feature?
Go deeperLabels are tested unevenly, and features change with the autoencoder's size
Anthropic's “Scaling Monosemanticity” extracted millions of features from a production Claude model, from concrete ones such as the Golden Gate Bridge to broad ones such as sycophantic praise. How thoroughly each label was tested varied from feature to feature.
In Anthropic's earlier work on a small model, sparse autoencoders given more features to draw on split one broad feature into several narrower ones. So a feature reflects how one autoencoder of one size divided the activity, and whether the model itself uses such a division is an open question.