What Is PerplexityBot? – Definition and How to Control It
Learn what PerplexityBot is, how answer-engine crawlers differ from search crawlers, how to control them, and why citation depends on access.
PerplexityBot is the crawler operated by Perplexity to index web content for its answer engine. Like other AI operators, it publishes user-agent tokens so sites can decide what to allow.
1. Answer-Engine Crawling Is a Different Bargain
A search crawler indexes pages so they can be ranked and clicked. An answer engine retrieves passages so they can be summarised and cited.
| Aspect | Search crawler | Answer-engine crawler |
|---|---|---|
| Output | A ranked link | A cited passage inside prose |
| Referral | A click through | A citation that may or may not be clicked |
| Slots | Ten or more per page | A handful per answer |
| Value to publisher | Traffic | Attribution, and some traffic |
That difference is why the allow-or-block decision is genuinely contested rather than obvious.
2. Controlling It
Address the token directly in robots.txt:
User-agent: PerplexityBot
Disallow: /premium/
Allow: /
Consult the operator's published documentation for current token names and their purposes, since operators run more than one and rename them as products change.
3. Verification Matters Here
Because answer-engine traffic is valuable to imitate, verifying identity matters. Check the operator's published address ranges rather than trusting the user-agent string.
Traffic claiming to be an answer engine from an unlisted range is something else, and treating it as verified because of its user agent is a common misconfiguration.
4. The Access-Citation Link
An engine cannot cite a page it cannot read. Blocking the crawler does not remove a brand from answers — engines still describe companies from other sources — but it does remove your own words from the pool that shapes the description.
The practical consequence is that blocking often trades a small amount of control for a larger loss of accuracy in how you are represented.
5. Making Content Citable
- Render server-side so a fetcher receives real content.
- Answer one question per section, directly, near its heading.
- Use specific, checkable claims rather than general description.
- Keep structure clean so passages chunk well.
- Date claims that depend on time.
6. Measuring the Effect
Whether a policy helps or hurts is measurable: run a frozen prompt set against the engine repeatedly, record whether your domain is cited and how it is described, and compare before and after a change.
Single checks are not measurements — generated answers vary between identical runs.
7. Common Misconceptions
- "Blocking removes us from the engine." It removes your pages as sources, not your brand as a subject.
- "robots.txt enforces access." It is a directive honoured voluntarily.
- "One rule covers all AI crawlers." Each operator, and often each purpose, has its own token.
- "Citation follows from allowing access." Access is necessary, not sufficient; quotability decides the rest.
8. Summary
PerplexityBot collects content for an answer engine, where the return is citation rather than a ranked link. Control it per token in robots.txt and verify real traffic by address.
Weigh a block against the loss of accurate representation, and measure the outcome with repeated sampling rather than assuming what a policy change did.