What is a web scraper?
A web scraper is a program that extracts specific data from web pages, such as prices, product listings or article text, so it can be reused elsewhere. A crawler discovers pages by following links; a scraper pulls data out of them. Many scrapers use generic user agents or headless browsers and do not follow robots.txt.
At a glance
- Type
- Data extraction tool (not a discovery crawler)
- Operator
- Anyone: companies, researchers, resellers, attackers
- How it identifies itself
- Often it does not, or it claims to be a normal browser
- Follows robots.txt
- Voluntary. RFC 9309 says robots.txt rules are not a form of access authorization
- OWASP name
- OAT-011 Scraping: collecting application content or data for use elsewhere
- How to limit it
- Rate limits, WAF rules and verifying bots that claim a known identity
How to spot it
- In your logs: generic user agents (HTTP libraries, or a browser string with no matching browser behavior), many requests from cloud-hosting IPs, and steady hits on the same page templates such as product or listing pages.
- Claimed identities: a scraper may claim to be Googlebot or an AI crawler to avoid blocks. Verify the claim with reverse DNS or published IP ranges.
- Browser-based scrapers: tools built on a headless browser run JavaScript, so they can show up in client-side analytics as people.
Related terms
Headless browser A browser driven by code with no visible window. A common tool for scrapers. User-agent spoofing Claiming to be a browser or a known bot to avoid being blocked. AI crawler Named bots from AI companies that publish tokens and IP lists.
Quick answers
Is web scraping the same as crawling?
No. A crawler follows links to discover and fetch pages. A scraper extracts specific data from pages. Many tools do both, but the goals differ.
Does robots.txt stop scrapers?
Only the ones that choose to obey it. RFC 9309 says the rules are not access authorization, so enforcement takes rate limits or a WAF.