Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Adversarial suffix

  • Attacks
  • Research term
3 steps

An adversarial suffix is text, often gibberish to people, that an attacker appends to a request so a model is more likely to comply when it would usually refuse. An automated search finds it.

Researchers find one with a program that keeps editing a string of tokens, the word pieces a model reads, against an open model whose inner workings they can inspect. Each round keeps the edits that push the model further toward answering. Nothing pulls the string toward readable words, so it ends up a jumble.

Where you’ll meet it

A news story reports that researchers added a string of nonsense words and symbols to requests and got several popular chatbots to give instructions they normally withhold. The headline says one string breaks every chatbot's safety.

What to do

Look in the study itself for which chatbots and requests the team tested, and when. A string published months ago may already be blocked, and success on several chatbots says nothing yet about the rest.

Easy to mix up

Most jailbreaks are prompts a person writes, such as a role-play that talks the model past its rules. An adversarial suffix comes from an automated search, run on an open model for hours.

Check yourself

Your company's support chatbot can look up any customer's orders by name. A security reviewer warns about adversarial suffixes. Which change protects customers better?

Go deeperScreening out gibberish slows the attack without ending it

Andy Zou and colleagues published Greedy Coordinate Gradient in 2023. Its target is a reply that opens with agreement, such as "Sure, here is," since a model that starts that way usually goes on. The model's gradients, which estimate how swapping each token would raise or lower the chance of that opening, pick the swaps to try next.

Strings tuned against open models also worked some of the time on closed chatbots the researchers never saw inside. The authors suggested one reason: some of the open models had been trained on conversations with a closed chatbot, so they share its habits.

Because the early strings look odd, one defense scores how surprising a text is to a language model, its perplexity, and blocks high scorers. Later methods produce suffixes that read as ordinary sentences and pass that screen.

Sources

Practice Modern AI

Or take the learning quiz on a mix of every set