How to verify Googlebot (and catch spoofed bots)

The fastest way to verify Googlebot is a forward-confirmed reverse DNS lookup: take the IP address from your server logs, run a reverse DNS lookup, confirm the hostname ends in googlebot.com or google.com, then run a forward DNS lookup on that hostname and confirm it resolves back to the same IP. If both directions match, the request is genuinely from Google. If the user-agent says “Googlebot” but the IP fails this check, you are looking at an impostor. The user-agent header alone is never proof, because any client can set it to any string in a single line of code.

That gap between what a request claims and what it can prove is where most bot problems live. Fake Googlebot is used to scrape content, bypass rate limits, and evade paywalls, because site owners whitelist “Googlebot” for SEO reasons and attackers know it. It is common enough that verifying Googlebot has become standard practice rather than an optional precaution. The exact same trick now targets AI crawlers: a scraper sets its user-agent to GPTBot and rides the goodwill site owners extend to OpenAI. This guide covers the full verification stack that closes both holes: reverse DNS with a forward-confirm, Google’s published IP ranges, and the newest layer, Web Bot Auth cryptographic signatures. It ends with a worked example you can run against a real log line.

Why the user-agent string proves nothing

A user-agent string is a self-reported label, not an identity. It is a plain-text HTTP header the client chooses, and there is no rule that it be truthful. A request carrying Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) might come from Google, or from a scraper on a residential proxy, or from someone testing with a one-line curl command.

This matters because so much infrastructure keys off the string. Site owners relax rate limits for Googlebot to protect crawl coverage. Some serve fuller markup to it. Analytics tools bucket traffic by user-agent. Every one of those behaviors becomes an attack surface the moment the label is trusted on its own. Distinguishing a genuine crawler from a browser or a scraper is the core problem of telling real bots apart, and it is worth understanding what a web crawler actually is and how a crawler differs from a scraper before you decide what to allow.

Google is blunt about this in its own documentation: the user-agent can be spoofed, so verification has to rest on something the impostor cannot fake. That something is the network path the request came from, and, increasingly, a cryptographic signature.

The three layers of real verification

Verifying a bot means proving the request originated from infrastructure the named operator actually controls. No single signal does this cleanly on its own, so production verification stacks three layers, each stronger than the last.

Layer	What it proves	Strength	Main limitation
Reverse DNS + forward-confirm (FCrDNS)	The IP resolves to and from an operator-owned domain	Strong for Google	Needs a live DNS lookup per IP; only works for operators who set PTR records
Published IP ranges	The IP sits inside a range the operator publishes	Strong, fast	Ranges change; you must refresh the list; not every operator publishes one
Web Bot Auth signatures	The request carries a valid cryptographic signature from the agent	Strongest	New standard; only signing agents are covered

The layers are complementary, not redundant. Reverse DNS and IP ranges answer “did this come from Google’s network?” Web Bot Auth answers “is this request cryptographically signed by the agent it claims to be?” - a question that holds even when an agent runs from cloud IPs it does not publish. Use them together and the naive spoof has nowhere to hide.

Layer 1: forward-confirmed reverse DNS (FCrDNS)

Forward-confirmed reverse DNS is the method Google has recommended since 2006, and it remains the gold standard for Googlebot. It works in two directions so a spoofer cannot fake it with a single crafted DNS record.

The steps:

1. Take the source IP from your log line, for example 66.249.66.1.
2. Run a reverse DNS (PTR) lookup on it. A genuine Googlebot IP returns a hostname like crawl-66-249-66-1.googlebot.com.
3. Confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com.
4. Run a forward DNS (A/AAAA) lookup on that hostname and confirm it resolves back to the original IP.

Both directions must agree. An attacker might control the reverse record for their own IP, but they cannot make crawl-66-249-66-1.googlebot.com forward-resolve to their server, because they do not control the googlebot.com zone. This two-way handshake is what makes reverse DNS trustworthy, and the mechanics are worth reading in full in our guide to reverse DNS lookup for bot verification.

The command-line version, using host on Linux or macOS:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward lookup returns the same IP you started with, so this request is genuinely Googlebot. If step 2 had returned a hostname on some unrelated domain, or step 4 had resolved to a different IP, you would reject it.

Layer 2: Google’s published IP ranges

Google publishes the exact IP ranges its crawlers use as machine-readable JSON, so you can verify without a DNS round-trip on every request. This is faster than reverse DNS at scale and useful when you want to pre-filter traffic in a firewall or edge worker.

The relevant files:

File	Covers
developers.google.com/static/crawling/ipranges/common-crawlers.json	Googlebot
developers.google.com/static/crawling/ipranges/special-crawlers.json	AdsBot, other special crawlers
developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json	Google-triggered fetchers (Google infrastructure)
developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json	User-triggered fetchers (may run from user devices)

Each entry is a CIDR block, for example 66.249.64.0/27. To verify, check whether the source IP falls inside any range in the matching file. Google notes that these ranges change, so you cannot cache the list forever. Refresh it on a schedule (daily is reasonable) and diff it, or you will start rejecting real Googlebot from a newly added block.

Two cautions. First, membership in an IP range proves the request came from Google’s network, not that the specific product is Googlebot rather than, say, a user-triggered fetch; match against the right file for the crawler you expect. Second, IP ranges verify origin but say nothing about intent, which is why range-matching pairs best with the behavioral view of what the crawler actually requested once inside.

Layer 3: Web Bot Auth signatures

Web Bot Auth is an emerging standard that lets a bot cryptographically sign its requests, so verification no longer depends on DNS or IP lists at all. It is built on HTTP Message Signatures (RFC 9421) and public-key cryptography. The agent holds a private key, publishes the corresponding public key in a directory, and signs each request; your server verifies the signature against the published key.

Mechanically, a signing agent sends a Signature header and a Signature-Agent header pointing to its key directory. You fetch the directory, find the key identified in the signature, and verify. A valid signature proves the request was produced by the holder of that private key, which no user-agent spoof and no IP forgery can replicate. It works even when an agent operates from generic cloud IP space it never published, which is exactly the case that defeats IP-range matching.

This is the direction the agent web is moving, because IP ranges and reverse DNS were designed for a handful of large search crawlers, not for the thousands of AI agents now fetching pages. Google, Cloudflare, and others are backing the approach. For the full mechanics of keys, directories, and the signing flow, see our explainer on what Web Bot Auth is. For now, treat it as the strongest layer where it is available, and fall back to reverse DNS and IP ranges for agents that do not yet sign.

A worked example: verifying one Googlebot hit

Here is the full process against a single log line, start to finish. Suppose your access log contains this entry:

66.249.66.1 - - [25/Jul/2026:09:14:22 +0000] "GET /pricing HTTP/1.1" 200 5120
"-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The user-agent claims Googlebot. Do not trust it yet. Verify:

Step 1 - reverse DNS. Look up the IP:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

The hostname ends in googlebot.com. So far so good, but a reverse record alone is not enough.

Step 2 - forward-confirm. Resolve that hostname back:

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

It resolves to 66.249.66.1, the same IP. Forward and reverse agree. This is genuine Googlebot.

Step 3 - cross-check the IP range (optional belt-and-suspenders). Confirm 66.249.66.1 falls inside a CIDR block in common-crawlers.json, for example 66.249.64.0/27 covers .64.0 through .64.31… in practice you check the file for the matching block. If it is listed, you have two independent confirmations.

Now compare that to a spoof. A scraper on a data-center IP such as 45.130.xx.xx sends the identical Googlebot user-agent. Step 1 returns a hostname that does not end in googlebot.com (or returns nothing), so it fails immediately. The user-agent matched; the network path did not. That is the entire point of verification, and it is why a system that only pattern-matches user-agent strings will happily wave the impostor through.

Spoofed GPTBot: the same problem, a new name

Spoofed GPTBot is the AI-era version of fake Googlebot, and it slips past the same naive checks. As site owners started allowing or tracking OpenAI’s crawler, scrapers began setting their user-agent to GPTBot to borrow that trust, exactly as they long borrowed Googlebot’s. The label is just as free to copy.

The good news is the verification pattern transfers directly. OpenAI publishes IP ranges for its crawlers as JSON, mirroring Google’s approach:

• GPTBot, the training crawler, publishes its ranges (see what GPTBot is for the full profile and how it differs from OpenAI’s other agents).
• OAI-SearchBot, which powers ChatGPT search results, publishes a separate range.
• ChatGPT-User, the user-triggered fetcher, publishes its own.

To verify a claimed GPTBot hit, match the source IP against OpenAI’s published GPTBot ranges, the same way you match Googlebot against common-crawlers.json. A request with the GPTBot user-agent from an IP outside those ranges is spoofed. And because OpenAI, Google, and others are adopting Web Bot Auth, signature verification is becoming the single method that covers every signing agent at once, instead of a per-operator scramble for IP lists. If you want to see which AI systems are actually reaching your pages and sending referral traffic, that is a question of tracking AI referral traffic on top of verifying the crawlers themselves.

This is the layer of the web Lume is built to measure. Lume captures the non-human half of your traffic server-side and cryptographically verifies which agents are genuine, combining Web Bot Auth, reverse DNS, and published IP ranges so a spoofed GPTBot is flagged rather than counted as the real thing. The same engine confirms real Googlebot, so your analytics reflect verified agents rather than whatever a user-agent header claimed.

Putting it into practice at scale

Verifying one log line by hand is easy; verifying millions is an engineering task. A few principles keep it reliable.

• Cache verification results by IP. Reverse DNS is a network round-trip. Verify an IP once, cache the verdict with a sensible TTL (an hour is common), and reuse it. Do not look up the same Googlebot IP on every hit.
• Refresh IP-range files on a schedule. Google and OpenAI both change their published ranges. Pull the JSON daily, and alert if a file fails to load rather than silently falling back to a stale copy.
• Verify at the point of decision. If you serve or block based on bot identity, verify before you act, not in a nightly batch. A batch job tells you yesterday’s scrapers; inline verification stops today’s.
• Log the verdict, not just the user-agent. Store whether each request was verified, spoofed, or unverifiable. That turns your logs into an audit trail and makes spoofing patterns visible. Learning to detect bot traffic in your logs starts with recording an honest verdict on every hit.

Done well, verification changes what your data means. A “Googlebot: 40,000 requests” line in an unverified report is a guess. The same line after verification is a fact, split into real Googlebot and the impostors that were riding its name.

FAQ

Is checking the user-agent enough to verify Googlebot?

No. The user-agent is a self-reported header that any client can set to any value, so it identifies nothing on its own. Verification requires confirming the request came from Google’s network via forward-confirmed reverse DNS or a published IP range, and ideally a Web Bot Auth signature. Treat the user-agent as a claim to be checked, never as proof.

What is FCrDNS?

FCrDNS stands for forward-confirmed reverse DNS. You run a reverse DNS lookup on the source IP to get a hostname, confirm the hostname belongs to the operator’s domain (such as googlebot.com), then run a forward DNS lookup on that hostname and confirm it resolves back to the original IP. Both directions must agree, which is what stops a spoofer who controls only their own reverse record.

Which domains should a real Googlebot IP resolve to?

A genuine Googlebot IP resolves to a hostname ending in googlebot.com, google.com, or googleusercontent.com. Googlebot itself typically returns a crawl-*.googlebot.com hostname. If the reverse DNS lands on any other domain, or the forward lookup does not return the original IP, the request is not authentic Googlebot.

Can I verify Googlebot without doing a DNS lookup on every request?

Yes. Google publishes its crawler IP ranges as JSON (Googlebot lives in common-crawlers.json), so you can check whether a source IP falls inside a published CIDR block without a DNS round-trip. Refresh the file regularly because the ranges change. Many teams combine both: IP-range matching for speed, reverse DNS as a fallback and cross-check.

How do I catch a spoofed GPTBot?

Match the source IP against OpenAI’s published crawler IP ranges, exactly as you match Googlebot against Google’s ranges. A request carrying the GPTBot user-agent from an IP outside those ranges is spoofed. Because OpenAI and others are adopting Web Bot Auth, verifying a cryptographic signature is becoming the most durable way to confirm any AI agent, GPTBot included.

Does Web Bot Auth replace reverse DNS and IP ranges?

Not yet, but it is the strongest layer where it is available. Web Bot Auth proves a request is cryptographically signed by the agent it claims to be, which holds even when the agent runs from cloud IPs it never published. Until every agent signs, use it alongside reverse DNS and published IP ranges rather than instead of them.