Autonomy budget
- Agents
- Industry term
An autonomy budget is a set of measurable limits on an agent's reach, such as allowed tools, data, time, steps, spending, and destinations, plus the largest allowed effect before the system stops or changes route.
Rules written only in prompts depend on the model following them. Limits enforced by tools and accounts can refuse an action or hold it for an authorized decision even when the model proposes it. The limit and the route taken when it is reached need to be defined together.
Where you’ll meet it
At work, you're setting up an agent to reorder office supplies. It compares vendors, asks for quotes, and can place orders on the company account. In a trial run, it tries to buy 40 boxes of printer paper from a site nobody on your team has used before.
What to do
Before it runs alone, have the purchasing-account owner enforce two controls through the account or purchasing tool: an approved list of vendors and item types, and a total purchasing limit for the run. Try an unapproved vendor and an order above the spending limit, confirming that both are refused.
Easy to mix up
An approval gate holds an action until an authorized person decides. The limits in an autonomy budget can act on their own, blocking, stopping, or rerouting work with no one asked.
Check yourself
Your support agent may issue refunds of up to $50 each, but the intended budget is $50 per order. A customer asks for three $40 refunds on one order, and all three go through. Which limit enforces the budget even if refunds are split into smaller pieces?
Go deeperLeast privilege helps set an agent’s limits
In their 1975 paper, Jerome Saltzer and Michael Schroeder describe least privilege: each part of a system receives only the access its task needs. That principle can help set an agent’s limits. OWASP calls excessive functions, permissions, or freedom to act “excessive agency,” a risk in applications built on language models.
Limits set too tight can delay routine, reversible work and encourage people to approve requests without reading. Match the limits to the effects you need to contain and the work that should proceed. A $10 test purchase and a message to every customer need different boundaries.
Sources
- LLM06:2025 Excessive Agency, OWASP GenAI Security Project
- The Protection of Information in Computer Systems, web.mit.edu
- MCP Security Best Practices, Model Context Protocol