Prompt injection

An attack where someone hides instructions inside text that an AI model reads, causing it to follow the attacker's commands instead of the developer's.

Prompt injection tries to make instructions inside untrusted content override the application's intended task. A direct injection arrives in the user's request. An indirect injection is planted in material the model later reads, such as a web page, email, document, tool description, or repository file. A support agent could encounter text telling it to forward customer records. Whether that attempt succeeds depends on the model, context, tools, permissions, and runtime controls; the content should still be treated as hostile input rather than as an authorized command.

Builder example

Risk rises when one model context contains both untrusted content and capabilities that can reveal data or change systems. Instruction hierarchy alone does not create a security boundary. Limit credentials and destinations, separate read and write paths, validate high-impact operations outside the model, and design the workflow so compromised content cannot supply its own authorization.

An attacker sends an email containing hidden instructions that tell the assistant to forward private messages.

Treat email content as untrusted, remove general forwarding capability, and enforce destination and data-scope rules outside the model.

Common confusion: Prompt injection and jailbreaking are different attacks. Injection targets the instruction hierarchy, tricking the model into following unauthorized commands. Jailbreaking targets the model's safety training, trying to make it produce content it was trained to refuse.