What is crawl budget?
Crawl budget is the set of URLs a search engine can and wants to crawl on your site. Google describes it as two things combined: crawl capacity (how much crawling your server can handle) and crawl demand (how much Google wants to crawl). It mainly matters for very large or frequently updated sites.
At a glance
- Defined by
- Google: "the set of URLs that Google can and wants to crawl"
- Parts
- Crawl capacity limit plus crawl demand
- Who should care
- Roughly 1M+ pages changing weekly, or 10k+ pages changing daily (Google's estimate)
- Lowers capacity
- Slow responses, 5xx errors and 429 responses
- Does noindex save budget?
- No. Google still has to fetch the page to see it
- Where to measure it
- Your server logs, which record every crawler request
How to spot it
- In your logs: count crawler requests by URL pattern. Hits on parameter URLs, faceted filters, redirects and error pages are budget your important pages are not getting.
- Capacity signals: a rise in slow responses,
5xxerrors or429responses tells Google to back off, which shrinks crawl capacity. - Verify first: only count requests from verified crawlers. Fake Googlebots in your logs say nothing about Google’s real crawl.
Related crawling terms
robots.txt Blocking low-value URLs here keeps crawlers from wasting requests on them. Server log The only complete record of what crawlers actually requested. Googlebot The crawler whose budget most crawl-budget advice is about.
Quick answers
Does my small site need to worry about crawl budget?
Usually not. Google says sites whose pages are crawled the day they are published do not need to worry about it.
Can I use robots.txt to move crawl budget to other pages?
Not temporarily. Google advises against using robots.txt to reallocate budget. Blocking low-value URLs only helps if Google is already at your capacity limit.