Scrapeless Wiki

What Is robots.txt? – Definition and How It Works

Informational P1 what is robots txt
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/robots-txt. Publishing both risks cannibalisation — decide before this goes live.

Learn what robots.txt is, how user-agent groups and matching rules work, what it does not do, and how AI crawler directives changed its role.

robots.txt is a plain text file at the root of a domain telling automated clients which paths they should not request. It is a request, not a control — and the difference is the most misunderstood thing about it.

1. What Is robots.txt?

It is a published crawling policy, always at /robots.txt.

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /cart/help
Crawl-delay: 1

Sitemap: https://example.com/sitemap.xml
  • Key idea: the site states its preferences; well-behaved clients honour them.
  • Enforcement: none. It is a convention backed by reputation, not by any mechanism.
  • Scope: one file per host and port. https://example.com and https://shop.example.com have separate files.

2. How the Rules Work

Groups. Each User-agent line starts a group. A crawler uses the single most specific matching group and ignores the rest — it does not merge them.

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /internal/

Googlebot here is disallowed from /internal/ only. It does not inherit /private/ from the wildcard group. This trips people up constantly: adding a specific group silently exempts that crawler from every general rule.

Matching is prefix-based on the path, with two wildcards:

  • * matches any sequence — Disallow: /*.pdf$
  • $ anchors the end — Disallow: /*?$ blocks URLs ending in a query mark

Allow carves exceptions out of a Disallow. Where both match, the more specific rule wins.

An empty Disallow: permits everything. Disallow: / forbids everything.

3. What It Does Not Do

Four persistent misconceptions:

It is not access control. A disallowed path is still publicly reachable. Anyone can request it. Listing /admin/ in robots.txt advertises its existence to everyone reading the file — which is everyone.

It does not remove pages from search. Disallow prevents crawling, not indexing. A page linked from elsewhere can still appear in results, listed without a description because the crawler was never allowed to read it. To keep a page out of an index, allow the crawl and use noindex — a directive the crawler must fetch the page to see.

It does not stop anyone. Compliance is voluntary. Malicious clients ignore it, and so, deliberately, do some legitimate ones.

It is not legally binding on its own. It may inform a dispute about authorisation; it is not a contract.

4. Crawl-delay and Rate

Crawl-delay: 10 asks for ten seconds between requests. It was never part of the original specification, and support is inconsistent — Google ignores it entirely, preferring its own rate controls, while Bing and others honour it.

Where it is absent, one request per second per host is a defensible default. Where it is present and you are not Google, honouring it is both correct and a good way to stay welcome.

5. AI Crawlers

The file has taken on a new role as publishers respond to model training and retrieval:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Different bots serve different purposes, and blocking indiscriminately has consequences worth knowing. Some collect training data; others fetch pages live to answer a user's question with a citation. Blocking the second kind removes you from answers where you would have been cited and linked — a real visibility cost, not just a principled stance.

Worth deciding per bot rather than as one sweeping rule.

6. Reading One as a Crawler Operator

  • Fetch it first, before anything else on the host, and cache it — typically for 24 hours.
  • Handle failures deliberately. A 404 conventionally means no restrictions. A 5xx means unknown, and the cautious reading is to back off rather than assume permission.
  • Match your own user agent honestly. Selecting the least restrictive group by claiming to be something else defeats the point.
  • Take the Sitemap: line. It is often the fastest route to a complete URL list and saves the site your discovery traffic.
  • Log your decisions. When someone asks why you fetched a path, "our robots parser matched this group" is a much better answer than a shrug.

7. Real-World Examples

  • A /admin/ disallow functioning as a directory listing for anyone curious.
  • A disallowed page appearing in search results with no description, because blocking the crawl also blocked the noindex it contained.
  • A specific Googlebot group accidentally exempting Google from every wildcard restriction.
  • A publisher blocking all AI crawlers and disappearing from cited answers along with the training corpora.
  • A Sitemap: line that turned a planned multi-day crawl into a single file download.

8. Summary

robots.txt is a plain text file at a domain's root that declares which paths automated clients should avoid. Rules are grouped by user agent, the most specific matching group applies exclusively, and matching is prefix-based with * and $ wildcards.

It is not security, not an indexing control, and not enforceable — a disallowed path stays public, and a disallowed page can still be indexed while remaining unreadable to the crawler that would have seen its noindex.

For crawler operators it is both a courtesy and a practical asset: honour the groups and delays, treat a 5xx as unknown rather than permitted, and take the Sitemap: line, which frequently makes link discovery unnecessary.