Which AI crawlers are taking your content — and is AI search sending readers back?
Publishers get hit hardest: crawlers that ignore robots.txt, content scraped for training, and search clicks replaced by AI answers. Lume shows exactly which AI crawlers hit your content and how much, verifies they’re really GPTBot or ClaudeBot and not a spoofer, and measures the humans ChatGPT, Perplexity, and other AI answers send back. It won’t block a crawler, charge for a crawl, or get you cited. It gives you the truth, and you decide what to do with it.
Sound familiar?
- GPTBot and ClaudeBot hammered my origin hundreds of thousands of times last month. I’m the one eating the bandwidth, and robots.txt clearly isn’t stopping them.
- If GPTBot or Googlebot started getting 403s after a deploy, I’d never know until my AI answers and search clicks quietly fell off a cliff.
- Is ChatGPT or Perplexity actually sending readers to my articles, or just answering with my work? It all lands in “Direct.”
- My content’s being scraped for training, and half these “GPTBot” hits could be spoofers, and I can’t even tell who’s really taking it.
What Lume does for you
Every AI crawler on your content, identified
Split every crawler by operator, type, and page: 56 cataloged bots, crawlers, and AI agents across 36 operators, sorted into 17 types. See GPTBot, ClaudeBot, and PerplexityBot for what they are (AI-training vs. AI-search vs. traditional search) and how much each takes. Server-side and unsampled, not the fraction client JS catches.
Catch crawler access regressions
See the status codes crawlers hit, per URL, so the day GPTBot or Googlebot starts getting 403s or 429s after a deploy shows up here, not weeks later when your AI-answer inclusion and search clicks have quietly dried up.
Is AI search sending readers back?
Measure the humans ChatGPT, Perplexity, Claude, and Gemini send you, detected across 18 referrer domains and 15 assistants by referrer host and utm_source, even when the referrer is stripped and it would otherwise land in “Direct.” Read it per article, against a clean, bot-free human baseline.
Decide with data, block nothing
Lume tells you who’s really taking your content and how much; you act via robots.txt or your CDN. We don’t block, throttle, paywall, or toll a single crawler. (Blocking AI-training crawlers won’t drop you from search, but may cut AI-answer inclusion. Your call, made on data.)
Fits your stack
Report traffic straight from WordPress, your CDN, or the edge. No rebuild, no client script.
Publisher FAQ
Should I block AI crawlers?
Your call: Lume is data, not a kill switch. It shows who’s crawling, how much, and whether they’re verified, so you decide and then act via robots.txt or your CDN. Blocking AI-training crawlers won’t remove you from search, but may cut your inclusion in AI answers.
Does this stop scraping, or let me charge crawlers?
No. Lume is the identity and analytics layer, not a paywall, a toll booth, or a blocker. It tells you exactly who’s taking what, then pairs with whatever enforcement or licensing you choose to run.
Is AI search actually replacing my traffic?
Measure it directly. Lume shows how much each AI crawler takes and how many humans those same assistants send back (your crawl-to-referral ratio, per article), so you can see whether AI answers are replacing your clicks or driving them.
How do I tell a real GPTBot from a spoofed one?
Verification: HTTP Message Signatures (Web Bot Auth), forward-confirmed reverse DNS, and published IP ranges, live for 7 major operators. A “GPTBot” from an IP that isn’t OpenAI’s is flagged unverified: identity proven, not assumed. “Verified” proves who a crawler is, not that it’s safe.
Does adding this slow my site or need a developer?
No. Start from the edge or CDN logs you already emit, with zero client JS and zero added latency. Prefer code? A dependency-free TypeScript SDK or a plain REST call. No full visitor IP is ever stored.
See who’s taking your content, and who’s sending readers back.
Free plan, one ingest token. Know your AI crawlers, verify who’s real, and measure the AI traffic you’re already getting.
Start for free →