Bot, crawler and AI agent glossary

A real web browser driven by an AI agent that browses, clicks and fills in forms on a person's behalf.

How to check it: Some sign their requests with Web Bot Auth. Others look like ordinary Chrome and need IP or behavior checks.

A bot run by an AI company that fetches web pages to train models, build an AI search index, or answer a user's question.

Good to know: Each AI crawler has its own robots.txt token, so you can allow search bots and block training bots.

The user agent OpenAI uses when ChatGPT visits a page because a user asked it to. It does not crawl the web on its own.

Good to know: OpenAI says robots.txt rules may not apply to it.

Anthropic's crawler that collects web content that may be used to train its AI models. Claude-SearchBot and Claude-User are separate bots.

Good to know: Anthropic runs three bots with three tokens. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.

A robots.txt product token that controls whether Google may use your content for Gemini training and grounding. It is not a separate crawler.

In your logs: You will never see it. Google says it has no separate HTTP user agent, so a request claiming to be Google-Extended is fake.

Google's main crawler for Google Search.

How to check it: Reverse DNS or Google's published IP ranges. The name alone is often faked.

OpenAI’s crawler that collects public pages which may be used to train its models.

Good to know: Blocking it does not remove you from ChatGPT search. That is OAI-SearchBot.

A web browser that runs without a visible window, controlled by code. Used for testing, rendering, automation and scraping.

Why it matters: Headless browsers run JavaScript, so bots built on them can be counted as people by client-side analytics.

OpenAI's crawler that surfaces websites in ChatGPT's search features. It is separate from GPTBot, which is about model training.

Why it matters: OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

Perplexity's crawler that surfaces and links websites in Perplexity search results. Perplexity says it is not used to train foundation models.

Good to know: Perplexity-User is a separate fetcher that Perplexity says generally ignores robots.txt.

A program that discovers and fetches pages by following links, for search engines, AI companies or SEO tools.

In your logs: Most announce a bot name in the user agent. Check the claim before trusting it.

A program that extracts specific data from web pages, such as prices, listings or article text, for use elsewhere.

In your logs: Scrapers rarely name themselves. Look for generic user agents, cloud-hosting IPs and repeated hits on the same page types.

Forward-confirmed reverse DNS. A reverse lookup on the IP, then a forward lookup on the hostname, which must return the same IP.

How to check it: Reverse, match the domain suffix exactly, forward-confirm. All three must pass.

RFC 9421, the IETF standard for signing selected parts of an HTTP message and carrying the result in Signature-Input and Signature headers.

Why it matters: It is the cryptography under Web Bot Auth, the way bots and AI agents prove who they are.

Machine-readable lists of the IP blocks a crawler or AI bot sends requests from, published by its operator so sites can verify it.

How to check it: Match the source IP against the exact CIDR blocks. A nearby address on the same cloud is a fail.

A DNS query that turns an IP address back into a hostname using a PTR record. The first half of crawler verification.

Good to know: On its own it proves nothing. Whoever runs the IP block can write any hostname into its PTR record.

A Web Bot Auth request header that points the verifier to the HTTPS URL where the signer's public keys are published.

Status: Defined in an IETF working-group draft, not an RFC yet. Expect older and newer forms in the wild.

A bot or AI agent that cryptographically signs its HTTP requests (Web Bot Auth), so its identity can be checked from the request itself.

In your logs: Look for Signature, Signature-Input and Signature-Agent headers. The user agent may just say Chrome.

The User-Agent request header, where a browser, crawler or script names the software making the request.

Good to know: It is self-reported text. Treat it as a claim to check, never as proof.

Sending a fake User-Agent header, usually a trusted crawler's name, so a request gets treated as something it is not.

How to check it: Run the operator's own verification method. A famous name that fails it is a spoof.

A bot whose claimed name has been confirmed against something only the real operator controls. It proves who, not whether you want it.

Good to know: Verified means the name is real. It says nothing about whether the bot is welcome.

An emerging IETF approach that lets bots and agents prove who they are by signing requests with HTTP Message Signatures.

Status: Draft standard, already used by some agents.

The set of URLs a search engine can and wants to crawl on a site in a given time. It mostly matters for very large or fast-changing sites.

How to check it: Your server logs show exactly which URLs crawlers spend their requests on.

A proposed Markdown file, usually at /llms.txt, that gives AI models a short summary of a site and links to its most useful pages.

Status: A community proposal, not an IETF or W3C standard. Google Search ignores it.

A plain-text file at the root of a site that tells crawlers which paths they may and may not fetch. It is a request, not access control.

Good to know: Some user-triggered AI fetchers say robots.txt rules may not apply to them.

A firewall for HTTP traffic that checks each request against a set of rules and blocks, challenges or allows it before it reaches your application.

Why it matters: A WAF rule can block a crawler that robots.txt allows. The bot sees an error, you see nothing.

Human visitors who arrive on a site by clicking a link or citation in an answer from an AI assistant such as ChatGPT, Perplexity or Gemini.

Good to know: Clicks with no referrer land in Direct, so the labeled number is a floor.

Requests to a website made by software rather than a person, from search crawlers and AI bots to uptime monitors, scrapers and attack tools.

In your logs: Most bot traffic never runs JavaScript, so it shows up in server logs, not in page-tag analytics.

The address of the page that linked to or loaded a resource, sent by the browser in the Referer request header (spelled with one r).

Good to know: Modern browsers send only the origin to other sites by default, and nothing from HTTPS to HTTP.

A file where a web server records every request it handles, usually one line per request with the IP, time, URL, status code, referrer and user agent.

Why it matters: Bots that never run JavaScript are invisible to page-tag analytics but still appear here.