What is GPTBot? A field guide to AI crawlers
On this page
GPTBot is OpenAI’s official web crawler for collecting public webpage text to train its generative AI models. It requests pages like any browser, but it announces itself with the user-agent token GPTBot and exists for one purpose: gathering training data. It does not index your site for search, and it does not fetch pages on behalf of a person typing into ChatGPT. Those are separate bots with separate names, and treating them as one crawler is the most common and most expensive mistake site owners make.
This guide defines GPTBot precisely, then maps it against the other major AI crawlers so you can decide what to allow, what to limit, and what you should never block by accident.
What GPTBot is and what it does
GPTBot is the crawler OpenAI runs to collect publicly accessible content that may be used to train its foundation models. It was introduced in August 2023. It reads ungated text, follows links, and skips paywalled pages, personal data, and sources that require login. Its full user-agent string looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The version number changes over time, so match on the GPTBot token rather than the whole string. OpenAI publishes the IP ranges GPTBot crawls from at https://openai.com/gptbot.json, which matters for verification (more on that below). If you are new to how automated agents move through a site, our primer on how a web crawler actually works covers the mechanics.
The key point about GPTBot: it feeds model training only. Blocking it keeps your content out of future training data. It has no effect on whether ChatGPT can cite you in a live answer, because live answers come from different bots.
Training vs search-index vs user-fetch crawlers
AI crawlers fall into three jobs, and the job determines whether blocking one helps or hurts you.
A training crawler collects text to teach a model; Common Crawl’s CCBot, an open web-archive crawler whose dataset is widely reused to train models, is one example. Blocking it is a data-licensing decision, not a visibility decision.
A search-index crawler builds the index an AI assistant reads from when it answers a live query. Block this one and you disappear from that assistant’s answers, the same way blocking Googlebot removes you from Google.
A user-fetch crawler retrieves a single page in real time because a person asked the assistant to read that specific URL. It is closer to a human click than to bulk crawling.
Blocking the wrong category is silent. Nothing errors. You simply stop showing up in AI answers while believing you only opted out of training. Understanding the split is the foundation of any sensible policy, and it pairs directly with work on optimizing your site for AI search.
The major AI crawlers compared
The table below lists the crawlers you will see most in server logs, what each does, its user-agent token, and how to control it.
| Crawler | Operator | Job | User-agent token | How to allow or limit |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | GPTBot | robots.txt honored |
| OAI-SearchBot | OpenAI | Search index for ChatGPT search | OAI-SearchBot | Allow to stay in ChatGPT search |
| ChatGPT-User | OpenAI | User-triggered page fetch | ChatGPT-User | Fires only on a user action |
| ClaudeBot | Anthropic | Training | ClaudeBot | robots.txt honored |
| Claude-User | Anthropic | User-triggered page fetch | Claude-User | Fires on a Claude user request |
| PerplexityBot | Perplexity | Search index | PerplexityBot | Allow to appear in Perplexity |
| Google-Extended | Training control token | Google-Extended | robots.txt token, not a crawler | |
| Applebot-Extended | Apple | Training control token | Applebot-Extended | robots.txt token, not a crawler |
| CCBot | Common Crawl | Open crawl reused for training | CCBot | robots.txt honored |
Two rows deserve a note. Google-Extended and Applebot-Extended are not crawlers you will see hitting your server. They are robots.txt tokens that tell Google and Apple whether they may use content they already fetched (with Googlebot and Applebot) for AI training. Disallowing Google-Extended stops Gemini training use without touching your Google Search ranking. That separation is deliberate, and it is why a blanket block is rarely what you want.
How to allow and limit each crawler
You control most of these through robots.txt, using the exact user-agent token. To keep your content out of OpenAI training while staying fully visible in ChatGPT’s search answers:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Apply the same logic per operator: disallow the training token, allow the search token. For a more expressive, machine-readable statement of what agents may do with your content, an llms.txt file sits alongside robots.txt and is increasingly read by AI tooling. Note that user-fetch bots like ChatGPT-User and Claude-User act on a direct human request, so robots.txt crawl rules may not apply to them the way they do to bulk crawlers.
Changes are not instant. OpenAI notes it can take around 24 hours for a robots.txt edit to propagate through its systems, so verify with logs rather than assuming an immediate effect.
Why user-agent strings cannot be trusted on their own
A user-agent string is just a text header the client chooses to send, and anything can send any string. A scraper can set its user-agent to GPTBot and impersonate OpenAI to slip past rules meant to allow legitimate crawlers, or to make its traffic look benign in your analytics. Nothing in the request itself proves the sender is who it claims to be.
Real verification confirms identity independent of the header. The two standard methods are checking the request’s source IP against the operator’s published ranges (like OpenAI’s gptbot.json), and running a reverse-then-forward DNS lookup so the IP resolves back to the operator’s domain. Our walkthrough on verifying Googlebot shows the same reverse-DNS technique that applies to AI crawlers. Lume runs these checks continuously: it tracks every operator and user-agent token in a live agent catalog and verifies which requests are genuine, so a spoofed GPTBot is flagged rather than counted as the real thing.
Verification also turns raw logs into a decision you can act on. Once you can trust the label, you can see real AI crawler and referral activity in agent analytics and set robots.txt policy on facts instead of guesses.
Frequently asked questions
Is GPTBot the same as ChatGPT?
No. GPTBot is the crawler that gathers training data. ChatGPT is the product. When a ChatGPT user asks the assistant to open a specific link, that page is fetched by ChatGPT-User, not GPTBot.
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot only affects training data. ChatGPT’s search answers are built from OAI-SearchBot. Block GPTBot and you stay in ChatGPT search; block OAI-SearchBot and you vanish from it.
Is GPTBot the same as OAI-SearchBot?
No. They are separate crawlers with separate tokens and separate jobs. GPTBot trains models; OAI-SearchBot indexes pages for ChatGPT’s live search feature.
How do I verify that a request is really GPTBot?
Check the request’s IP against OpenAI’s published ranges at openai.com/gptbot.json, or run a reverse DNS lookup that resolves back to an OpenAI domain and confirm it with a forward lookup. The user-agent string alone is not proof.
Should I block GPTBot?
That depends on your goal. Block it if you do not want your content used in future model training. Keep it allowed if you want your material represented in those models. Either way, do not block OAI-SearchBot unless you also want out of ChatGPT’s search answers.