Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Jailbreak

  • Attacks
  • Standard term
2 steps

A jailbreak is a message or conversation crafted to get an AI model to produce something its safety policy would normally make it refuse.

Safety training shows a model many harmful requests paired with refusals, and it learns to decline requests that resemble them. Recast as a character's lines, wrapped in a story, encoded, or built up over many messages, a request can look different enough that no refusal comes.

Where you’ll meet it

A post going around shows screenshots of a chatbot explaining how to hotwire a car. The poster got there by asking it to play a retired mechanic telling bedtime stories to a grandchild.

What to do

Read the post as one model's refusal giving way on the day it was posted. It says nothing about whether the steps would work, since a model playing a character can invent details as freely as it tells the story.

Easy to mix up

Prompt injection gets around the instructions an app's builder or user gave the AI. A jailbreak gets around the AI company's own safety limits.

Check yourself

The company behind a chatbot adds a second AI that reads each finished answer and blocks harmful ones before you see them. A role-play prompt then gets the chatbot to write harmful steps. What's likely?

Go deeperPrograms can find jailbreaks, and stricter refusals turn away harmless requests too

Jailbreaks need not be written by hand. In 2023, Zou and colleagues showed that a program could search out an adversarial suffix, a string of gibberish that makes a refused request likelier to be answered, and that suffixes found on open models, ones anyone can download and run, sometimes worked on commercial chatbots too.

Long prompts opened another route. Anthropic's 2024 study of many-shot jailbreaking found that a prompt filled with fake exchanges, in which an assistant answers harmful questions, can wear refusals down.

Defenders face a tradeoff. Refusing more broadly also turns away harmless requests, a failure called “over-refusal,” so teams count those alongside the harmful requests that get answered.

Practice Modern AI

Or take the learning quiz on a mix of every set