What is ClaudeBot?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

ClaudeBot is Anthropic's web crawler for collecting content that could contribute to training its AI models. It follows robots.txt and supports Crawl-delay. Blocking it signals that your future content should be left out of Anthropic's training data. Anthropic runs two other bots with their own tokens: Claude-SearchBot for search indexing and Claude-User for user requests.

At a glance

Operator
Anthropic
Type
Training data crawler
User-agent token
ClaudeBot
Follows robots.txt
Yes, including the non-standard Crawl-delay extension
How to verify
Source IP in Anthropic's published list, claude.com/crawling/bots.json
Blocking it means
Your future content is excluded from Anthropic's training datasets

How to spot it

  • In your logs: the user agent contains ClaudeBot. Claude-SearchBot and Claude-User are different bots, so count them separately.
  • To verify it: the source IP must be on the list Anthropic publishes at claude.com/crawling/bots.json. A ClaudeBot user agent from any other IP is user-agent spoofing.
  • To control it: use robots.txt. Anthropic honors it for its bots and supports Crawl-delay.
# Opt out of training, stay in Claude search
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Anthropic’s three bots

Token Job Blocking it
ClaudeBot Collects content that may be used for training Future content excluded from training data
Claude-SearchBot Indexes pages to improve search results May reduce visibility and accuracy in Claude’s search results
Claude-User Fetches pages when a Claude user asks Claude cannot retrieve your content for those queries

Anthropic-hosted browsers acting for a user add a separate Claude-Agent token. See agentic browser.

Similar bots from other AI companies

GPTBot OpenAI's training crawler. Also separate from its search bot. OAI-SearchBot OpenAI's crawler for ChatGPT search, the counterpart to Claude-SearchBot. PerplexityBot Perplexity's search crawler. Perplexity says it does not train foundation models.

Quick answers

Does blocking ClaudeBot remove me from Claude's answers?

Not by itself. ClaudeBot is about training. Claude-SearchBot and Claude-User handle search and user requests, and each has its own robots.txt token.

Should I block ClaudeBot by IP?

Anthropic recommends robots.txt. It warns that IP blocking may not work correctly or guarantee a lasting opt-out.

Does ClaudeBot respect Crawl-delay?

Yes. Anthropic says it supports the non-standard Crawl-delay extension to robots.txt.

Sources