How Does an AI Agent Use the Web? – Tools and the Action Loop
Learn how an AI agent reaches the web: the tools it is given, how a page is represented, how actions are chosen, and what stops it going wrong.
An agent has no network access of its own. Everything it does on the web happens because the application exposed a tool, the model requested that tool, and the application executed it. The web-facing behaviour is entirely a property of the tool set.
1. The Tools an Agent Is Given
Most web-capable agents are built from three primitives, sometimes packaged behind a tool server.
| Tool | Input | Returns |
|---|---|---|
| Search | A query | Ranked results with titles, links, snippets |
| Fetch | A URL | Page content as clean text or Markdown |
| Browser actions | Navigate, click, type, extract | Whatever a user would see, after scripts run |
Search answers "where is it", fetch answers "what does it say", and browser actions reach content that only exists after interaction.
2. How a Page Is Represented
The model cannot see pixels or HTML natively. The application chooses a representation, and that choice drives both cost and capability.
- Cleaned text or Markdown — cheapest, and enough for reading.
- Accessibility tree — interactive elements with roles and labels, each addressable by reference rather than coordinates. The usual default for agents that act.
- Screenshot — needed for canvas-rendered or visually encoded content, and requires a vision model.
- Raw HTML — complete and mostly noise; it exhausts the context window quickly.
3. The Action Loop
observe page state → choose one action → execute via tool
↑ |
└────────────── observe result ──────────┘
Each pass costs one model call. The loop ends when a success test passes, a step cap is reached, or a budget is exhausted. Without an explicit stop condition an agent will either declare victory early or wander.
4. What the Application Controls
Every consequential decision stays outside the model.
- Execution. The model requests; the application runs the tool.
- Authorisation. Whether this user may perform this action is checked in code.
- Validation. Tool arguments are validated against a schema before use.
- Budget. Step count, wall-clock time, and spend are capped per task.
5. Why Page Content Is Untrusted
An agent reads pages written by other people, and text on a page can be phrased as an instruction. If the agent's permissions come from the prompt, a page can rewrite them.
The defence is architectural rather than textual: keep authorisation in application code, scope tools to the narrowest capability that completes the task, and never let retrieved content decide whether an action is allowed.
6. Real-World Examples
- Answering a current-events question — search, open two results, compose an answer with citations.
- Collecting a field from many sites — visit each, extract one value, normalise, and store, with no per-site parser.
- Completing a lookup behind a form — navigate, type, submit, read the result page.
- Verifying a flow after a release — walk the steps and report where the expected element failed to appear.
7. Summary
An AI agent uses the web through tools the application defines, on a representation the application chooses, inside a loop the application bounds. The model contributes the decision about what to do next; everything else is ordinary software.
That division is also the safety model. Treat pages as untrusted input, keep permission checks in code, and cap the loop — the failure modes that matter are all failures of the harness rather than of the model.