How to Read robots.txt – Syntax and Precedence
Learn how robots.txt groups directives by user agent, how Allow and Disallow precedence works, what wildcards mean, and what it does not control.
robots.txt sits at a site's root and tells automated clients which paths they should not request. Reading it correctly means understanding grouping and precedence, both of which are easy to get wrong.
1. The Structure
User-agent: Googlebot
Disallow: /private/
Allow: /private/public-summary
User-agent: *
Disallow: /admin/
Disallow: /cart
Crawl-delay: 5
Sitemap: https://example.com/sitemap.xml
Directives are grouped by User-agent. A group applies until the next User-agent line begins another.
2. Only One Group Applies
A client matches the most specific group for its user agent, and only that group. If a named group exists for your client, the wildcard group does not also apply.
This is the most common misreading: a named group with two rules does not inherit the wildcard group's ten rules. Whatever is not repeated in the named group is not in effect for that client.
3. Allow and Disallow Precedence
Where both could match, the longer, more specific rule wins — not the one that appears first.
Disallow: /private/
Allow: /private/public-summary
/private/public-summary is permitted because the Allow path is longer. Order in the file does not decide it.
4. Wildcards
| Pattern | Meaning |
|---|---|
* |
Any sequence of characters |
$ |
End of the URL |
Disallow: /*.pdf$ |
Any URL ending in .pdf |
Disallow: /*? |
Any URL containing a query string |
Wildcards are an extension rather than part of the original specification, though support is now effectively universal among major crawlers.
5. Empty and Missing Values
Disallow:with nothing after it permits everything for that group.Disallow: /forbids the entire site.- A missing file means no restrictions are stated.
- A file returning a server error is conventionally treated as forbidding everything, since the site's wishes cannot be determined.
6. What It Does Not Do
- It is not access control. Compliant clients honour it; nothing enforces it.
- It does not hide anything. Listing a path in it advertises that path's existence.
- It says nothing about what may be done with data that is fetched — terms of service govern that.
- It does not remove pages from search on its own; that is a different mechanism.
7. Reading It Properly Before Collecting
- Fetch it and read the whole file.
- Find the group matching your client, and use only that group.
- Apply longest-match precedence between
AllowandDisallow. - Honour any
Crawl-delay. - Note the
Sitemaplines — they are usually the most efficient discovery route. - Re-check periodically; the file changes.
8. Summary
robots.txt groups directives by user agent, only the most specific matching group applies, and the longest matching path rule wins regardless of order.
It expresses a site's wishes rather than enforcing them, says nothing about permitted use of data, and its Sitemap lines are frequently the most useful thing in the file.