Computer use / browser use
- Agents
- Industry term
Computer use is an AI ability to operate software through its interface by clicking, typing, scrolling, and moving between screens. Browser use applies this approach to websites.
The system supplies a view of the interface, such as a screenshot or a structured description of page elements. The model proposes a click or other action, software carries it out, and the updated state informs the next step. Working through an interface can reach apps without a suitable program-facing connection.
Where you’ll meet it
You ask an assistant to sign your kids up for a summer camp through the camp's website. It fills in names and birthdays across four pages. On the last page, a pop-up offers a paid add-on, and its close button sits right next to "Add to cart."
What to do
Use an isolated session with access limited to the camp site, where the tool supports that restriction, and keep purchases subject to your approval. A separate browser profile can reduce shared sign-ins, but does not itself enforce those limits. Before paying, check the cart and each child's details.
Easy to mix up
Through an API, a program sends defined requests straight to the app. Computer use works the app the way a person does, through its on-screen buttons, menus, and fields.
Check yourself
Your browser agent reports that it renewed your parking permit on the city's website. Which evidence that the renewal went through do you trust more?
Go deeperPage content can try to redirect the agent
Anthropic released computer use as a public beta in October 2024, and other labs have followed. Some products supply an accessibility tree instead of screenshots or alongside them: a structured description of interface elements that assistive software can use. It can make controls clearer to identify, though the information may be incomplete.
Page content is outside your instructions, but the model may treat words in it as directions. A malicious page can exploit this with prompt injection, instructions intended to redirect the agent. Separately, an interrupted run may leave a partially completed booking or order, so recovery includes checking the site's actual state.
Sources
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku, Anthropic
- Microsoft Playwright MCP README, github.com
- Function calling and other API updates, OpenAI
- Stripe API: Idempotent requests, docs.stripe.com