What Is a Browser Agent? – Definition and How It Works
Learn what a browser agent is, how it decides actions from page state, how it differs from a scripted automation, and where it earns its cost.
A browser agent is a language model given control of a browser and a goal, deciding each action from what the current page shows. The script is written at run time rather than in advance.
1. What Is a Browser Agent?
It joins three parts: a browser session, a set of actions, and a model that chooses between them.
- Key idea: navigation is inferred from the page, not encoded in selectors.
- Mechanism: observe the page, pick an action, execute it, observe again.
- Goal: operate sites whose layout was never inspected by the person writing the task.
2. How the Agent Sees the Page
| Representation | Strength | Cost |
|---|---|---|
| Cleaned text | Cheapest for reading | Cannot be clicked |
| Accessibility tree | Interactive nodes with roles and labels, addressable by reference | Moderate |
| Screenshot | Sees visual-only content | Needs a vision model and matching coordinates |
The accessibility tree is the usual default, because each node it returns is something the browser can act on without pixel coordinates.
3. The Action Set
Agents are given a small, constrained vocabulary — navigate, click, type, select, scroll, extract, and finish. Keeping it small matters: every additional action increases the chance of a wrong choice and consumes context in the tool list.
Actions are executed by the harness, never by the model, which is where validation and authorisation live.
4. Agent Versus Script
- Script — fixed selectors, cheapest per page, breaks when the DOM changes, perfectly repeatable.
- Agent — adapts to unfamiliar layouts, costs a model call per step, varies between runs.
The economics decide it. One site read a million times deserves a script; a hundred sites read once each are where an agent avoids writing a hundred parsers.
5. What Makes Agents Fail
- Step accumulation. Per-step reliability multiplies, so long tasks degrade sharply.
- Interstitials. Cookie dialogs and modals consume steps intended for the task.
- Ambiguous targets. Several plausible elements share a label, and the agent picks the wrong one.
- Injected instructions. Page text can be phrased as a command; permission must be enforced in the harness, not in the prompt.
6. Design Rules That Work
- Cap steps and wall-clock time per task.
- Define an explicit success test rather than letting the model declare completion.
- Return structured output so downstream systems receive fields, not prose.
- Keep the action set minimal and the tool descriptions specific.
- Log every observation and action, because failures are only debuggable from the trace.
7. Real-World Examples
- Collecting one field from many differently built sites.
- Completing a lookup that only exists behind a form.
- Checking a user journey after a release and reporting where it broke.
- Exploring a site whose structure is unknown, to decide whether a scripted scraper is worth writing.
8. Summary
A browser agent trades the cost and determinism of a script for tolerance of unfamiliar layouts. It observes, decides, acts, and repeats, with the harness executing every action and enforcing every limit.
Use it where variety is the problem and volume is modest, keep the action set small, bound the loop, and treat page content as untrusted input throughout.