How to detect bot traffic in your logs
On this page
Roughly half of all web traffic is not human, and almost none of it shows up in the analytics dashboard you check every morning. Search crawlers, AI agents, uptime monitors, scrapers, and outright malicious bots all hit your site every day, yet the tools most teams rely on are built to measure people. The result is a large, active, and completely invisible half of your traffic.
The good news is that the honest record already exists. Every request to your site, human or not, leaves a line in your server logs. That log is where bot traffic is actually visible, and with a handful of signals you can not only count the bots but tell them apart: which are legitimate search and AI crawlers, which are scrapers copying your content, and which are attackers probing for weaknesses.
This guide covers a practical, server-log-first method to detect and classify bot traffic yourself: why client-side analytics misses bots, the specific signals that identify one, how to separate good crawlers from bad, and why a user-agent string alone can never be trusted without verification.
Why client-side analytics misses most bot traffic
Client-side analytics misses most bots because it only records visitors that execute JavaScript. Tools like Google Analytics work by loading a tracking script in the browser; that script fires an event, and the event becomes a session in your reports. Bots that never run the script are never counted.
Most automated traffic falls into exactly that blind spot. Search crawlers, AI training and retrieval agents, feed readers, and the majority of scrapers request your raw HTML over HTTP and move on. They do not render the page, do not execute the analytics tag, and therefore leave no trace in any client-side tool. A scraper can pull ten thousand pages from your site in an afternoon while your analytics shows a quiet day.
Server logs have the opposite property. They are written by your web server or CDN the moment a request arrives, before any JavaScript runs and regardless of whether the client is a browser at all. That makes the log the only complete record of who and what actually reached your site. If you want to understand non-human traffic, reading your log files is the foundation, and it is a discipline that also underpins how you diagnose crawling and indexing for search and AI engines.
A second reason to work from logs is trust. Client-side metrics can be inflated or distorted by bots that do render pages, and they can be blocked by privacy tooling. The server log is closer to ground truth: one line per request, written by infrastructure you control.
What detecting bot traffic in your logs actually means
Detecting bot traffic in your logs means reading raw request records and deciding, line by line, whether each request came from a human or an automated client, then labelling the automated ones by type and intent. It is a classification task, not a blocking one. The goal is visibility first: to see the non-human half of your traffic clearly enough to make decisions about it.
A typical log line contains the fields you need to start: a timestamp, the client IP address, the requested path and method, the HTTP status code returned, the number of bytes sent, the referrer, and the user-agent string. Combined access logs from Nginx and Apache include all of these, as do the edge logs from most CDNs.
None of those fields on its own is decisive. A user-agent can lie. An IP can be shared. A single fast request looks like a human clicking quickly. Detection works by reading the fields together, across many requests, and looking for the combinations that only automated clients produce. Before you can classify anything, it helps to be clear on what a web crawler is and how systematic automated fetching differs from a person browsing.
The signals that identify a bot in your logs
The signals that identify a bot fall into four families: what the client claims to be, where it comes from, how it behaves, and whether it can prove its identity. Strong detection reads all four together, because each one is defeatable on its own and reliable only in combination.
The table below lists the practical signals you can extract from standard logs, what each one tells you, and how easily a bot can fake it.
| Signal | What it tells you | Where it lives in logs | Spoofability |
|---|---|---|---|
| User-agent string | The identity the client claims (browser, crawler name, bot version) | User-agent field | Very high; trivially forged |
| Declared bot token | Names a known crawler such as a search or AI agent | User-agent field | Very high; scrapers copy real tokens |
| Client IP and ASN | The network and organisation the request came from | IP field, enriched with ASN lookup | Low for the network owner; the ASN is hard to fake |
| Reverse and forward DNS | Whether the IP genuinely belongs to the claimed operator | Derived from the IP | Low; this is the basis of verification |
| Request rate and volume | Human-impossible speed or page counts from one source | Timestamps grouped by IP | Medium; can be slowed to blend in |
| Path patterns | Systematic sweeps, sitemap crawling, or endpoint probing | Requested path over time | Medium |
| Asset loading | Whether CSS, JS, and images are fetched alongside HTML | Requested paths per session | Medium; most bots skip assets |
| HTTP status mix | High rates of 404s or 403s suggest scanning or blocked bots | Status code field | Low |
| Header fingerprint | Missing or ordered-wrong headers real browsers always send | Extended log formats | Medium |
| Cryptographic signature | A signed proof of identity the client attaches to the request | Signature headers, where present | Very low; the point of the mechanism |
Read across a row and you get a claim; read down the columns and you get corroboration. A request that claims to be a search crawler, comes from that search engine’s verified network, fetches HTML without assets at a steady rate, and passes a DNS check is almost certainly the real thing. A request that claims the same identity but comes from a consumer broadband IP in an unrelated country is almost certainly a scraper wearing a disguise.
Behavioural signals deserve particular attention because they are hard to fake without giving up efficiency. A human session loads a page and then pulls its stylesheets, scripts, fonts, and images in a burst, while automated clients that only want your content usually request the HTML and nothing else, producing a distinctive lonely-HTML pattern in the logs. Likewise, no person visits four hundred product pages in ninety seconds in perfect URL order, but a scraper working through your sitemap does exactly that.
Why the user-agent string alone is unreliable
The user-agent string is unreliable because it is self-reported and any client can set it to anything. It is a plain text header the requester chooses. A scraper can send Googlebot in the user-agent as easily as it can send a real Chrome string, and many do precisely that to slip past rules that trust the name.
This matters because user-agent filtering is still the most common way teams try to detect bots, and it fails in both directions. It produces false positives when a legitimate but unfamiliar agent gets flagged, and far more dangerous false negatives when a hostile scraper impersonates a trusted crawler. If your only rule is match the name, the bots you most want to catch are the ones that walk straight through. Spoofed search-crawler traffic is common enough that learning to confirm Googlebot is genuine has become a standard, documented practice.
The problem is growing as AI agents multiply. New crawlers appear constantly, some declaring honest tokens like GPTBot and others copying well-known names to avoid scrutiny. A token in the user-agent is a claim to investigate, never a fact to act on. Treat the string as the beginning of detection, not the end of it.
The fix is to corroborate the claim against something the client cannot control: the network it actually came from. That is what turns a guess into a verdict, and it is the subject of the next two sections.
How to separate good crawlers from scrapers and malicious bots
You separate good bots from bad by scoring intent and legitimacy, not just by confirming that a request is automated. All bots are non-human, but they are not all unwelcome, and a blunt block-everything approach will cut off the search and AI crawlers that bring you traffic. The useful split has three buckets.
Good bots are the crawlers you want. Search engine crawlers index your pages, AI assistants retrieve and cite your content, and monitoring services check that your site is up. These declare themselves honestly, come from stable published networks, respect robots directives, and crawl at reasonable rates. The distinction between a crawler and a scraper is less about the technology and more about consent and behaviour.
Grey-area bots are the ones whose value depends on your business. SEO tools, brand-monitoring services, and academic researchers sit here. They are automated and they consume resources, but they may also be doing work you benefit from. You want to see them clearly and decide case by case.
Bad bots are the ones acting against you. Content scrapers copying your material, credential-stuffing scripts hammering your login, vulnerability scanners probing for weak endpoints, and inventory or price bots. These reveal themselves through the combination of signals in the table: forged or mismatched identity, unverifiable networks, aggressive request rates, heavy 404 and 403 counts, and paths that make no sense for a human.
In practice you classify by asking a short sequence of questions of each source. Does it claim to be a known crawler? If so, does that claim survive verification against the operator’s network? If it does not verify, how does it behave: steady and shallow like a feed reader, or fast and systematic like a scraper? Is it requesting real content or scanning for admin panels and login forms? A source that fails verification and behaves aggressively is your priority. It is also worth watching whether legitimate crawlers are being blocked by accident, because that quietly starves your search and AI visibility while looking like nothing is wrong.
Verifying which agents are real
Verification is the step that confirms a claimed bot identity by checking it against the network the request genuinely came from. It is what converts the untrustworthy user-agent string into a reliable label, and it is the single most important technique in honest bot detection.
The classic method is reverse DNS confirmation. You take the request’s IP address, perform a reverse DNS lookup to get a hostname, confirm the hostname belongs to the operator’s domain, and then perform a forward lookup on that hostname to confirm it resolves back to the same IP. A request claiming to be a major search crawler should reverse-resolve to that search engine’s own domain and forward-resolve back to the same address. If it does not, the claim is false no matter what the user-agent says. This forward-confirmed reverse DNS check is the documented, operator-endorsed way to verify a legitimate crawler.
A second and stronger layer is now emerging: cryptographic verification. Instead of relying on DNS, newer agents sign their requests, attaching a signature that proves they hold a private key tied to a published identity. Emerging standards such as Web Bot Auth let an agent prove who it is on every request, and the presence of a valid signature is close to unforgeable. As AI agents proliferate, cryptographic identity is becoming the durable answer to spoofing that user-agent filtering never could be.
Verification at scale is where doing this by hand stops being realistic. Running DNS checks and signature validation across millions of log lines, keeping an up-to-date list of every crawler operator and their networks, and re-verifying as new agents appear is exactly the job Lume exists to do. It reads the non-human half of your traffic server-side, classifies each agent, and cryptographically verifies which ones are authentic rather than impostors, so you see a labelled, trustworthy view instead of a raw log. If you want to understand the verification layer specifically, our overview of how agent identity is checked walks through it.
Turning detection into a repeatable process
Bot detection becomes useful when it is a routine rather than a one-off investigation. A simple loop works well: collect your access logs into one place, parse each line into structured fields, enrich the client IP with ASN and reverse-DNS data, score each source against the four signal families, verify every request that claims a known identity, then group the results into your good, grey, and bad buckets and watch how they change week to week.
What you learn from that loop compounds. You find out which AI crawlers read your content and how often, you spot scrapers before they copy your whole catalogue, and you catch the day a deploy starts returning 403 to a search crawler. The core idea is small and durable: your logs already hold the truth about bot traffic, and detection is the practice of reading them with the right signals while refusing to trust any identity you have not verified.
Frequently asked questions
What is the difference between bot detection and bot mitigation?
Bot detection is identifying and classifying automated traffic so you can see what is hitting your site. Bot mitigation is acting on that knowledge to block, challenge, or rate-limit specific bots. Detection comes first and is valuable on its own, because you cannot make good decisions about traffic you cannot see. Many teams need clear detection and classification long before they need aggressive mitigation, and blocking without detection tends to cut off the good crawlers along with the bad.
Can I detect bots without buying a security product?
Yes. The signals that identify bots live in standard server and CDN logs that you already generate, so you can parse them, group requests by IP, check request rates, and run reverse DNS lookups with your own tooling. The limits are scale and freshness: verifying millions of requests, maintaining an accurate list of every crawler operator, and keeping up with new AI agents is where a dedicated analytics and verification layer saves substantial effort. The method is open to anyone; the labour is the reason people automate it.
How can I tell a real search crawler from a fake one?
Verify the request rather than trusting the user-agent. Take the IP address, run a reverse DNS lookup, confirm the hostname belongs to the crawler operator’s domain, then run a forward lookup to confirm it resolves back to the same IP. A genuine crawler passes this forward-confirmed check; an impostor sending the crawler’s name from an unrelated network fails it. Cryptographic request signatures, where an agent provides one, give an even stronger confirmation that does not depend on DNS.
Why do bots spoof their user-agent?
Bots spoof the user-agent because the string is self-reported and many sites grant trusted crawlers easier access. By claiming to be a well-known search or AI crawler, a scraper hopes to bypass rules that would otherwise slow or block it. This is precisely why identity claims must be verified against the network of origin, since the claim itself costs the bot nothing to fabricate.
Do AI crawlers show up in my server logs?
Yes. AI crawlers request your pages over HTTP just like any other client, so they appear in your access logs with their own user-agent tokens and source networks. They usually do not run client-side JavaScript, which is why they are absent from analytics dashboards but fully present in logs. Reading logs is currently the most reliable way to see how often AI agents fetch your content and which pages they favour.