What Is ClaudeBot? – Definition and How to Control It
Learn what ClaudeBot is, how AI crawler user agents differ by purpose, how to allow or disallow them in robots.txt, and how to verify what visits.
ClaudeBot is the crawler operated by Anthropic to collect web content. Like other AI operators, Anthropic publishes distinct user-agent tokens for different purposes, so sites can permit one activity and restrict another.
1. Why Purpose Matters
AI crawling is not one activity. Publishers increasingly distinguish between them because the exchange of value differs.
| Purpose | What it does | What the site gets |
|---|---|---|
| Training collection | Gathers text that may inform model training | No direct return |
| Live retrieval | Fetches a page to answer a specific question | Potential citation and referral |
| User-directed fetch | Retrieves a page a user explicitly asked about | Direct engagement |
Operators publish separate tokens precisely so these can be controlled independently.
2. Controlling It in robots.txt
Directives are matched per user agent, so each token is addressed separately:
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
To permit some paths and not others, disallow selectively rather than wholesale:
User-agent: ClaudeBot
Disallow: /members/
Disallow: /checkout/
Allow: /
Always consult the operator's current documentation for the exact tokens in use, since names and purposes change.
3. robots.txt Is a Directive, Not Enforcement
Compliant crawlers honour it voluntarily. It is not an access control, and anything requiring enforcement needs edge rules or authentication.
That said, major operators do comply, so the file is the right first tool — and it is the only one that expresses intent rather than merely blocking traffic.
4. Verifying What Actually Visited
User agents are self-declared, so a string alone proves nothing. Verification uses the operator's published address ranges, or a reverse-then-forward DNS check where supported.
Check server logs rather than assuming: the mix of automated traffic on a site is frequently different from what its operators expect.
5. Deciding What to Allow
Considerations that usually settle it:
- Do you want to appear in AI answers? Restricting retrieval fetchers reduces the chance of accurate citation.
- Is your content your product? Training collection may warrant a different answer than retrieval.
- Is licensing on the table? Some publishers restrict crawling and negotiate separately.
- What is the load? Crawl volume is an operational cost worth measuring before deciding.
6. Making Content Usable When You Do Allow
- Serve content server-side, since a fetcher that receives an empty shell cannot quote you.
- Use real headings and structure so passages chunk cleanly.
- Keep claims specific and dated so they are safe to repeat.
- Provide structured data where it applies.
7. Common Misconceptions
- "Blocking one bot blocks AI entirely." Operators run several crawlers, and other providers exist.
- "robots.txt hides content." It requests that compliant clients stay away; the content remains public.
- "Allowing crawling means giving up rights." Copyright is unaffected by a crawl directive.
- "The user agent proves identity." Only address verification does.
8. Summary
ClaudeBot is Anthropic's web crawler, one of several tokens published so that sites can distinguish training collection from live retrieval and control each separately.
Express your decision per token in robots.txt, verify real traffic against published address ranges rather than trusting user-agent strings, and remember that blocking retrieval fetchers also removes the path to being cited accurately.