What is PerplexityBot?
PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results. It follows robots.txt, and Perplexity says it is not used to crawl content for AI foundation models. A separate fetcher, Perplexity-User, visits pages when a user asks a question, and Perplexity says that one generally ignores robots.txt.
At a glance
- Operator
- Perplexity
- Type
- Search crawler (not a training crawler)
- User-agent token
- PerplexityBot
- Follows robots.txt
- Yes. Changes may take up to 24 hours to apply
- How to verify
- Source IP in Perplexity's published list, perplexity.ai/perplexitybot.json
- Blocking it means
- The bot that surfaces sites in Perplexity search stops crawling yours
How to spot it
- In your logs: the user agent contains
PerplexityBot, for exampleMozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). Match the token, not the full string. - To verify it: the source IP must be in
perplexity.ai/perplexitybot.json. ForPerplexity-User, checkperplexity.ai/perplexity-user.jsoninstead. - If it is missing: check robots.txt, your CDN and your WAF. Perplexity recommends allowing PerplexityBot in robots.txt.
PerplexityBot vs Perplexity-User
| PerplexityBot | Perplexity-User | |
|---|---|---|
| Job | Surfaces and links sites in Perplexity search | Visits a page to answer a user’s question |
| robots.txt | Follows it | Generally ignores it, because a user triggered the fetch |
| IP list | perplexity.ai/perplexitybot.json |
perplexity.ai/perplexity-user.json |
Similar bots from other AI companies
OAI-SearchBot OpenAI's crawler for ChatGPT search. ChatGPT-User OpenAI's user-triggered fetcher, the counterpart to Perplexity-User. ClaudeBot Anthropic's training crawler, with separate search and user bots.
Quick answers
Does PerplexityBot train AI models?
Perplexity says no. It states PerplexityBot is not used to crawl content for AI foundation models.
How do I block Perplexity-User?
robots.txt is not reliable for it, because Perplexity says it generally ignores robots.txt. Use an edge or firewall rule against its published IP list instead.
Is a PerplexityBot user agent proof it is Perplexity?
No. Anyone can send that string. Check the source IP against perplexity.ai/perplexitybot.json.