What Is GUI Grounding? – Definition and How It Works
Learn what GUI grounding is, how a model maps an instruction to a screen element, why coordinates are fragile, and how element references solve it.
GUI grounding is connecting a described intention — "click the submit button" — to a specific, actionable target in an interface. It is the step between deciding what to do and being able to do it.
1. What the Problem Is
A model reasoning about a screen produces a description. Executing it requires an unambiguous target: a coordinate, an element reference, or a selector.
Grounding is that translation, and it is where a large share of agent failures originate — the plan is right and the target is wrong.
2. Three Grounding Strategies
| Strategy | Target | Strength | Weakness |
|---|---|---|---|
| Coordinates | A point on screen | Works on anything visible | Breaks on scaling, scrolling, and layout shifts |
| Element reference | A node from the accessibility tree | Stable and precise | Misses purely visual content |
| Selector | A CSS or XPath expression | Precise and reusable | Requires knowing the markup |
Element references are the usual default for web agents, because each one is directly actionable and survives repositioning.
3. Why Coordinates Are Fragile
A coordinate is only correct if the model's view and the executing environment agree exactly.
- A screenshot scaled before inference shifts every coordinate.
- A viewport that differs from the captured frame puts targets elsewhere.
- Content that loads or animates between capture and click moves the target.
- Scrolling changes the mapping entirely.
Any of these produces a click that lands on the wrong element while looking successful in the trace.
4. Set-of-Mark Prompting
A middle path overlays numbered labels on interactive elements in a screenshot, and asks the model to name a label rather than a coordinate.
The model reasons visually, and the harness resolves the label back to a real element. This keeps visual understanding while removing coordinate arithmetic from the model's job.
5. Disambiguation
Interfaces repeat labels — several "Edit" buttons, several "Next" links. Grounding must resolve which one, using position relative to a container, surrounding text, or the section heading above it.
Instructions that name a container as well as a target — "the Edit button in the billing section" — ground far more reliably than a bare label.
6. Verifying the Action
Grounding failures are quiet. The click succeeds, something happens, and the agent proceeds with a wrong assumption.
Confirm the effect rather than the action: after clicking, check that the expected state appeared. That single check converts a silent misgrounding into a caught error.
7. Practical Guidance
- Prefer element references over coordinates wherever a tree is available.
- Use labelled overlays when visual reasoning is genuinely needed.
- Include container context in target descriptions.
- Verify the resulting state after every consequential action.
- Log the resolved target, not just the intent, so failures are diagnosable.
8. Summary
GUI grounding maps an intended action to an actionable target. Coordinates work anywhere and break easily; element references are stable but blind to purely visual content; labelled overlays combine the two.
Disambiguation and post-action verification matter as much as the mapping itself, because a wrongly grounded action fails silently and poisons every step after it.