What Is Set-of-Mark Prompting? – Definition and How It Works
Learn what set-of-mark prompting is, how numbered overlays replace coordinates, why it improves grounding accuracy, and what its limits are.
Set-of-mark prompting overlays numbered labels on the interactive elements of a screenshot and asks the model to name a label rather than produce a coordinate. It removes coordinate arithmetic from the model's job.
1. The Problem It Solves
A vision model asked to click something must output where. Producing pixel coordinates from an image is error-prone, and any mismatch between the captured frame and the live viewport makes a correct-looking coordinate land somewhere else.
Errors of that kind are silent: the click succeeds, the wrong thing happens, and the agent continues confidently.
2. How It Works
- Enumerate the actionable elements on the page, usually from the accessibility tree.
- Draw a numbered marker on each in the screenshot.
- Send the annotated image along with the numbered list.
- The model responds with a number and an intent.
- The harness maps that number back to the real element and acts on it.
Model: click 14
Harness: element 14 = button "Submit order" → click by reference
3. Why Accuracy Improves
The model performs a task it is good at — visually identifying the right element — and does not perform one it is poor at, which is precise coordinate estimation.
Resolution back to an element reference also means scrolling, resizing, and minor reflow do not invalidate the choice, because the target was never a point on screen.
4. What to Mark
Marking everything produces a cluttered image and a long list. Practical filtering:
- Only visible, actionable elements.
- Deduplicate overlapping markers.
- Cap the count, prioritising elements in the viewport.
- Include a short label with each number, so the list is meaningful without the image.
5. Compared With Other Approaches
| Approach | Model outputs | Robustness |
|---|---|---|
| Raw coordinates | A point | Low — breaks on any frame mismatch |
| Set-of-mark | A label number | High — resolved to a real element |
| Accessibility tree only | An element reference | High, but no visual context |
| Selectors | A CSS or XPath expression | High, requires knowing the markup |
Set-of-mark occupies the useful middle: visual reasoning, element-level execution.
6. Limits
- Enumeration must be accurate. Elements missed by the extractor cannot be selected.
- Canvas-rendered interfaces expose nothing to enumerate.
- Dense interfaces produce crowded overlays that are hard to read.
- Cost. Every step still sends an image for inference.
7. Practical Notes
Keep the mark-to-element mapping for the exact screenshot sent, since a later capture renumbers everything. Log both the number chosen and the element it resolved to, so a wrong action can be traced to either misreading or misenumeration.
Verify the resulting state after each action, because grounding errors otherwise stay invisible.
8. Summary
Set-of-mark prompting labels interactive elements in a screenshot and asks the model to choose a label, converting a coordinate problem into a selection problem.
It raises grounding accuracy substantially and depends entirely on accurate enumeration — the model can only choose from what the harness marked.