What is an AI crawler?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

An AI crawler is an automated bot run by an AI company that fetches web pages. It does one of three jobs: collecting content for model training, indexing pages for AI search answers, or fetching a page because a user asked. GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are examples, each with its own robots.txt token.

At a glance

Run by
AI companies such as OpenAI, Anthropic and Perplexity
Main jobs
Model training, AI search indexing, user-triggered fetching
How it identifies itself
A user-agent token per bot, e.g. GPTBot, ClaudeBot, PerplexityBot
Follows robots.txt
Training and search crawlers do, per their operators. Some user-triggered fetchers may not
How to verify
Match the source IP to the operator's published IP list
Blocking it means
Depends on the bot: training opt-out, or dropping out of that AI's search answers

How to spot it

  • In your logs: look for the operator’s token in the user agent, such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot or PerplexityBot. Match the token, not the full string.
  • To verify it: check the source IP against the list the operator publishes. A known AI token from any other IP is user-agent spoofing.
  • Know the job: training, search indexing and user-triggered fetching are separate bots with separate tokens, so one robots.txt rule rarely covers an operator.
# Allow AI search, opt out of training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
From Lume's data

Which AI crawlers are busiest

It varies
No single AI crawler was the busiest everywhere. The leader changed from site to site
At least 2x
as many requests from ClaudeBot as from GPTBot, on every opted-in site
More
requests from ChatGPT-User, which fetches for people's questions, than from GPTBot's training crawl, on every opted-in site

Method: Requests to sites that opted in to this anonymous analysis, August, September, October and November 2026 (to 4 November). Each site is summarized separately, then figures are rounded. Counts are requests claiming each bot name.

Common AI crawlers

OAI-SearchBot OpenAI's crawler for ChatGPT search. Blocking it keeps you out of ChatGPT search answers. ClaudeBot Anthropic's training crawler, with separate Claude-SearchBot and Claude-User bots. PerplexityBot Perplexity's search crawler. Perplexity says it is not used to train foundation models.

Quick answers

Does blocking GPTBot remove me from ChatGPT search?

No. OpenAI says each bot setting is independent. ChatGPT search uses OAI-SearchBot, so that is the token that controls it.

Is Google-Extended an AI crawler?

No. Google calls it a standalone product token with no separate user agent. Google's crawling uses its existing crawlers, and the token only controls how content is used.

Should I block all AI crawlers?

Decide per bot. Blocking training crawlers is an opt-out from training. Blocking search crawlers can remove you from the AI answers you may want to appear in.

Sources