What is Web Bot Auth? A plain-English guide to signed bot traffic

Web Bot Auth is an emerging IETF standard that lets automated bots and AI agents cryptographically sign their HTTP requests, so a website can verify who is actually knocking instead of trusting a self-reported label. Each request carries a digital signature that a site checks against a public key the bot operator publishes at a known URL. If the math checks out, the visitor is provably the agent it claims to be. If it does not, the request is an impostor.

This matters because the fastest-growing slice of web traffic is now non-human: crawlers, scrapers, AI training bots, and AI agents fetching pages for a user. Site owners want to welcome the legitimate ones and block the fakes, but the old signals for telling them apart were never built to resist forgery. Web Bot Auth fixes that with public-key cryptography. This guide explains what it is, how the signature handshake works step by step, and how it fits alongside the verification methods you already use.

What Web Bot Auth actually is

Web Bot Auth is a way for a bot to prove its identity by signing each HTTP request with a private key only it holds. The signature is verified against a matching public key the operator publishes openly, so anyone receiving the request can confirm it came from the real key holder and was not tampered with.

The approach is being formalized by the IETF’s Web Bot Auth working group (acronym webbotauth), which sits in the Web and Internet Transport area. Its charter is narrow: standardize methods for cryptographically authenticating automated clients and for conveying information about their operators to websites. In scope are search crawlers, web archivers, link checkers, AI training crawlers, and AI agents acting for a user. Explicitly out of scope are end-user authentication, agent-to-agent APIs, and bot reputation scoring. The group aims to send its core authentication specification to the IESG in April 2026 and an operational best-practice document in August 2026.

Underneath, Web Bot Auth builds on an existing standard rather than inventing new cryptography. It uses HTTP Message Signatures, defined in RFC 9421, which specifies exactly how to sign chosen parts of an HTTP request and carry the result in headers.

Why user-agent strings and IP allowlists fall short

The problem Web Bot Auth solves is that the traditional bot signals are either forgeable or fragile. A user-agent string is just text the client sends about itself. Anyone can set their user agent to Googlebot or GPTBot in a single line of code, so the string is a claim, never a proof. Our explainer on what GPTBot is and how to control it walks through how often that label gets spoofed in real logs.

The stronger legacy method is verifying the network path: confirm the request came from an IP the operator actually owns. That is the basis of reverse DNS verification for bots and of the classic technique for confirming Googlebot is genuine. Reverse DNS is far better than a user-agent string because it is hard to fake, but it has real limits. It depends on the operator maintaining correct forward-confirmed DNS records, it adds DNS lookups to your hot path, and published IP ranges drift as fleets scale, move clouds, or add regions. Every time an operator changes infrastructure, every site relying on a hardcoded IP list has to catch up.

Web Bot Auth decouples identity from the network. The proof travels inside the request as a signature, so it holds no matter which IP or data center the bot uses. The key does the vouching, not the address.

How the Web Bot Auth handshake works

The handshake is a signing step on the bot side and a verification step on the server side, tied together by a public key the operator publishes at a well-known location. Here is the full sequence:

1. Generate a key pair. The operator creates an asymmetric key pair, typically using the Ed25519 algorithm. The private key stays secret; the public key will be shared.
2. Publish the public key. The operator hosts the public key in JSON Web Key (JWK) form at a standard path, /.well-known/http-message-signatures-directory, served over HTTPS. This directory is itself signed so it cannot be quietly swapped.
3. Sign the request. For each outbound request, the bot selects components to sign (such as @authority, the target host) and produces a signature per RFC 9421. It attaches a Signature-Input header naming the covered components, algorithm, key ID, creation timestamp, and expiry; a Signature header with the signature bytes; and a Signature-Agent header pointing to its key directory.
4. Read the headers. The website, CDN, or WAF sees the request and reads Signature-Agent to learn where the operator’s public keys live.
5. Fetch and verify. The server retrieves the public key from that directory and checks the signature against the signed components. It also confirms the timestamp is fresh, which stops an attacker replaying a captured signature later.
6. Allow, log, or block. A valid signature means the request is provably from the named operator, so the site can allowlist it, apply a rate policy, or record it as verified. An invalid or missing signature falls back to your other checks.

Because the signature covers request components and includes a short expiry, a captured header cannot be reused against another site or replayed after it lapses. Identity is bound to the request, not merely asserted next to it.

Where adoption stands in 2025 and 2026

Web Bot Auth has moved from proposal to real deployments across major infrastructure providers, even while the spec is still an IETF draft. Cloudflare shipped verification and publishes an open web-bot-auth helper library and directory tooling on GitHub, plus a public test endpoint. AWS WAF added support in November 2025. Vercel’s bot verification reads the signature, and Akamai has described cryptographic bot authentication as the direction for trustworthy automated traffic.

On the bot side, Google has begun signing a subset of requests from its Google-Agent user agent, authenticated as https://agent.bot.goog, though it labels the feature experimental. That caveat is the key operational point: adoption is partial, so verification should be additive, and Google itself advises sites to keep falling back to established methods when a request is not signed. This shift toward machine-readable trust sits alongside conventions like the llms.txt standard and the broader practice of generative engine optimization.

How Web Bot Auth fits your existing verification stack

Web Bot Auth is best treated as the strongest layer in a stack, not a replacement for everything below it. A practical order: check the cryptographic signature first when present, fall back to reverse DNS and forward-confirmed lookups, then compare against published IP ranges, and use the user-agent string only as a hint. Each layer covers the gaps of the one beneath it, so no single signal carries the whole decision. If you are auditing today, our guide to spotting bot traffic in server logs shows what those signals look like in raw data.

This layered model is exactly how Lume approaches agent identity. Lume verifies agents using Web Bot Auth signatures alongside reverse DNS and published IP ranges, then records a verdict for each request so you can see which of your non-human visitors are provably authentic and which are spoofing a trusted name. You can read the full technical breakdown on the Web Bot Auth product page or explore how the agent verification feature turns those checks into a clear signal in your analytics.

Frequently asked questions

Is Web Bot Auth an official standard yet?

Not fully. It is an active IETF working group with published drafts and a charter, targeting an April 2026 submission of its core authentication specification. In the meantime, major providers already implement the draft, so it is usable in practice while the text is finalized.

What is the difference between Web Bot Auth and reverse DNS verification?

Reverse DNS proves a request came from an IP the operator controls, which ties identity to the network. Web Bot Auth proves the request was signed by the operator’s private key, which ties identity to the key regardless of IP. Signatures survive infrastructure changes that would break a hardcoded IP list, so the two methods pair well.

Does Web Bot Auth stop user-agent spoofing?

Yes, for participating bots. A spoofer can copy a GPTBot user-agent string, but it cannot produce a valid signature without the operator’s private key. Any request claiming to be a signed agent but lacking a verifiable signature can be flagged or blocked.

What headers does Web Bot Auth add to a request?

Three: Signature-Input lists the signed components, algorithm, key ID, and timing; Signature carries the signature bytes; and Signature-Agent points to the operator’s public key directory at /.well-known/http-message-signatures-directory.

Do I need to change my site to accept signed bots?

If you run behind a CDN or WAF that already supports Web Bot Auth, verification can happen at the edge with no code from you. If you verify in your own stack, you read the signature headers, fetch the published key, and validate per RFC 9421, keeping reverse DNS and IP checks as a fallback for unsigned traffic. A platform like Lume can handle that verification and surface the verdict for you.