What Is GPTBot? – Definition and How to Control It
Learn what GPTBot is, how it differs from OpenAI's other crawlers, how to allow or disallow each in robots.txt, and how to verify real traffic.
GPTBot is OpenAI's crawler for collecting web content. It is one of several user agents the operator publishes, each corresponding to a different purpose, so sites can permit one and restrict another.
1. Why There Is More Than One
The value exchange differs by purpose, and publishers increasingly want to distinguish them.
| Purpose | Behaviour | What the site gets |
|---|---|---|
| Training collection | Bulk gathering that may inform model training | No direct return |
| Live retrieval for answers | Fetches pages to answer a question now | Potential citation and referral |
| User-directed fetch | Retrieves a page a user explicitly asked about | Direct engagement |
Publishing separate tokens is what makes selective permission possible at all.
2. Controlling It
Directives are matched per user agent:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Selective permission is usually the better setting:
User-agent: GPTBot
Disallow: /members/
Disallow: /internal/
Allow: /
Check the operator's current documentation for the exact token names, since they change as products change.
3. robots.txt Is Voluntary Compliance
It expresses intent and is honoured by major operators. It is not an access control — enforcement requires edge rules or authentication.
That distinction matters when the goal is protection rather than preference: a Disallow on genuinely sensitive content is the wrong tool.
4. Verifying What Visits
User-agent strings are self-declared, so any client can claim to be a crawler. Verification uses the operator's published address ranges, or a reverse-then-forward DNS check where supported.
Reading server logs is worth doing before making policy: the actual mix of automated traffic is often not what operators assume.
5. Deciding Your Policy
- Do you want to be quoted? Restricting the retrieval fetcher makes accurate citation less likely.
- Is the content the product? Training collection may deserve a different answer from retrieval.
- Is licensing being negotiated? Some publishers restrict crawling and license separately.
- What does it cost? Measure crawl volume before deciding it is a burden.
6. If You Allow It
Make the content usable: render server-side so a fetcher does not receive an empty shell, use real headings so passages chunk cleanly, keep claims specific and dated, and provide structured data where it applies.
A page that is allowed but unreadable to a fetcher gets the downside of access with none of the benefit.
7. Common Misconceptions
- "Blocking one token blocks AI." Operators run several, and other providers exist.
- "Disallow removes content from a model." It affects future collection, not what was already gathered.
- "Allowing crawling waives copyright." It does not.
- "The user agent proves identity." Only address verification does.
8. Summary
GPTBot is OpenAI's web crawler and one of several published tokens, separated so that training collection, live retrieval, and user-directed fetching can be permitted independently.
Set policy per token in robots.txt, verify traffic against published address ranges, and weigh restriction against the loss of accurate citation before blocking everything.