Polysemanticity

  • Control
  • Research term
2 steps

Polysemanticity is when a single neuron or other part of a model responds to several unrelated kinds of input, such as both legal citations and baking recipes.

A neuron is a small unit of the model's calculation whose activity, a number, rises and falls with the input. Researchers usually label a neuron by the passages that push that number highest. The number reads the same whichever input raised it, so the rest of the model is thought to separate the meanings by which other neurons fire alongside it.

Where you’ll meet it

You read an article saying researchers found the neuron in a chatbot that handles sarcasm. Its figure shows the twenty passages that excite the neuron most, and every one is sarcastic.

What to do

Treat “the sarcasm neuron” as a nickname for now, and look for two more pieces of evidence in the article. One is how the neuron responds to ordinary passages, well below the top twenty. The other is what changes in the chatbot's answers when researchers turn the neuron down.

Easy to mix up

Superposition is a proposed reason: many ideas sharing the same neurons. Polysemanticity is what researchers see, a single neuron responding to unrelated kinds of input.

Check yourself

A developer sends customer emails to a manager whenever one neuron fires strongly, chosen because the angriest emails in a sample raised it most. A request to change a delivery address gets flagged, and the manager finds nothing angry in it. What's the likely reason?

Go deeperSparse autoencoders split mixed neurons into features that are easier to name

Olah and colleagues described mixed neurons in image models in 2020, including one that responds to cat faces and to the fronts of cars. Anthropic's 2022 “Toy Models of Superposition” showed small networks storing more features, the separate ideas a network tracks, than they have neurons when those features rarely occur together, which would leave individual neurons mixed.

Anthropic's 2023 “Towards Monosemanticity” trained a sparse autoencoder, a second network that rewrites a layer's activity as a much larger set of features with only a few active at once. Many of those features had a single clear reading. The split depends on choices, since a larger autoencoder breaks one feature into finer ones, so no decomposition is final.

Practice Modern AI

Or take the learning quiz on a mix of every set