Crawl budget: what it is and how to protect it
On this page
Crawl budget is one of the few technical-SEO concepts that most site owners can safely ignore until the day it starts costing them indexed pages and revenue. The classic advice, still true, is that small sites rarely need to think about it. What changed in 2025 and 2026 is who spends it. Googlebot used to be the main visitor deciding how much of your server to consume. Now GPTBot, ClaudeBot, PerplexityBot, and a growing fleet of AI crawlers hit the same URLs, hammer the same origin, and draw down the same finite serving capacity. Crawl budget stopped being a pure Googlebot problem and became a whole-of-traffic problem.
This guide covers what crawl budget actually is, what wastes it, how your server logs expose the leaks that dashboards hide, and how the AI-crawler wave reshapes the whole calculation. There is a table of the most common budget wasters with the specific fix for each, and a short FAQ at the end.
What crawl budget actually is
Crawl budget is the number of URLs a crawler can and wants to fetch from your site in a given time window. Google frames it as two forces multiplied together: the crawl capacity limit and crawl demand. Neither is a dial you set directly, but both respond to how you run your site.
The crawl capacity limit is the ceiling on how hard a bot will push your server without degrading it. Google limits the total time it holds connections open, balancing parallel requests against your response times. When your origin answers quickly and cleanly, the limit rises. When responses slow down or you start returning 5xx errors, Google backs off automatically to avoid knocking you over.
The crawl demand is how badly the crawler wants your pages. It scales with your site’s size, how often content changes, and how popular or fresh a URL appears. A news homepage updated hourly earns high demand. A five-year-old terms-of-service page earns almost none.
Google updated its crawl budget documentation in July 2026 with a detail that matters more than ever: every site now starts with the same default, conservative capacity limit, and that limit is shared across all of its crawlers. High demand from one crawler reduces the capacity available to the others. That principle, written about Google’s own bots, is exactly what makes the AI-crawler surge a crawl budget problem rather than a footnote. A web crawler does not care whose budget it spends; your server has one finite pool of capacity, and every bot draws from it.
Why crawl budget matters, and when it does not
Crawl budget matters when the number of URLs on your site is large enough that bots cannot fetch all of them as often as you would like. If a page is not crawled, it is not indexed. If it is not indexed, it cannot rank or be cited in an AI answer. Wasted budget therefore shows up as a delay: new products, updated prices, and fresh articles take longer to appear in search, and low-value URLs get crawled instead of the pages that make money.
The rough threshold most practitioners use is around 10,000 URLs. Below that, a healthy site with a clean sitemap and a fast server almost never runs out of budget, and optimizing it is a poor use of time. Above that, and especially on large e-commerce catalogs, publishers, marketplaces, and any site with faceted navigation, crawl budget becomes a real constraint worth managing deliberately.
Consider a mid-size store with 2,000 products. That sounds small. But add faceted navigation with filters for color, size, price band, brand, and sort order, and the crawlable URL space explodes. Every filter combination becomes its own URL, often with its own parameters, and the site now exposes 40,000 or more crawlable addresses for 2,000 products worth of actual content. Bots discover those combinations through internal links, follow them, and spend the day crawling ?color=blue&sort=price-asc variants while your newest arrivals sit undiscovered. That is crawl budget waste in its purest form, and it is invisible in a normal analytics dashboard because none of those crawls are human visits.
What wastes crawl budget
Crawl budget waste is any bot request that consumes capacity without moving a valuable URL closer to being crawled, indexed, or served. Most waste falls into a handful of recurring patterns, and each has a specific fix. The table below maps the common wasters to what to do about them.
| Crawl-budget waster | What it looks like | The fix |
|---|---|---|
| Faceted navigation and URL parameters | Thousands of filter and sort combinations from a small catalog | Block parameter patterns in robots.txt, use canonical tags, and avoid linking to filtered URLs internally |
| Redirect chains | Bot follows 3+ hops before reaching the final URL | Collapse chains so each old URL points directly to the final destination in one hop |
| Duplicate content | Same content on multiple URLs (www vs non-www, trailing slashes, print versions) | Consolidate with canonicals and 301s, and serve one authoritative URL |
| Soft 404s and thin pages | Empty category pages or “no results” pages returning 200 | Return a real 404 or 410, or add genuine content and noindex the rest |
| Broken internal links | Bots crawl to dead 404 URLs discovered through your own links | Fix or remove the linking anchors so budget is not spent on dead ends |
| Slow server responses | High time-to-first-byte drags down the capacity limit | Improve origin speed and caching so bots fetch more URLs per session |
| 5xx errors under load | Server returns errors when crawl volume spikes | Add capacity or rate-limit aggressive bots so healthy crawling continues |
| Orphan and low-value pages | Old, unlinked, or expired URLs still crawled repeatedly | Remove them, consolidate, or block them so demand shifts to live pages |
| Uncontrolled AI-crawler volume | GPTBot or PerplexityBot fetching low-value URLs at high frequency | Identify the pattern in logs, then rate-limit or disallow the low-value paths |
Redirect chains deserve a specific note because they are so common and so quietly expensive. Every hop is a separate request that a bot must make and wait on before reaching real content, and long chains can cause crawlers to give up before the destination loads. Cleaning them up is one of the highest-return crawl budget fixes on most large sites; the mechanics of doing it well are covered in this guide to 301 redirects.
The new dimension: AI crawlers now spend your budget too
AI crawlers are automated agents that fetch your pages to train models, build retrieval indexes, or answer live user questions, and they consume the same server capacity that search crawlers do. This is the shift that classic crawl budget articles were written before, and it is why the topic deserves a fresh look.
The volume is not trivial. On many sites, AI crawlers now account for a substantial and rising share of all bot requests, and they are not always polite about it. Where Googlebot has decades of tuning to respect your capacity limit and back off under strain, newer AI crawlers can request pages aggressively, ignore crawl-delay hints, and fetch the same low-value URLs that waste Googlebot’s time. Because your server has one shared pool of serving capacity, a burst of GPTBot traffic that slows your origin can drag down the crawl capacity limit that Googlebot operates under. The AI crawler spends budget, and the search crawler pays for it in reduced capacity.
There are three distinct costs here, and they are worth separating:
- Serving cost. AI crawlers consume real bandwidth and CPU. On a large catalog, an aggressive crawler fetching 40,000 faceted URLs is a measurable hosting bill and a source of latency for real users.
- Search crawl cost. Because capacity is shared, AI-crawler load that slows your server reduces how deeply and quickly Googlebot can crawl, delaying indexing of your important pages.
- Opportunity cost. Every AI-crawler request spent on a junk URL is a request not spent on the content you actually want represented in AI answers.
The response is not to block every AI crawler by reflex. Many of them feed the answer engines where you want your brand cited, and there is a real strategy to being crawled well for generative engine optimization. The response is to control where that budget goes: let AI crawlers reach your valuable, canonical URLs, and keep them out of the faceted, parametered, and duplicate URL space that wastes everyone’s budget. That decision requires knowing which bots are hitting which URLs, which is exactly what dashboards cannot tell you and logs can.
How server logs reveal where crawl budget goes
Server logs are the only complete record of every bot request your site received, and they are where crawl budget waste becomes visible instead of theoretical. Google Search Console’s Crawl Stats report is useful for Googlebot trends, but it only shows Google, it samples and aggregates, and it tells you nothing about GPTBot, ClaudeBot, or PerplexityBot. Your raw access logs miss nothing.
A log file analysis answers the questions that matter for crawl budget directly. Which URLs did bots request most? What share of crawls landed on parametered or filtered URLs versus real content? How many requests hit 404s, redirect chains, or 5xx errors? How does crawl frequency on your money pages compare to the junk? A Reddit TechSEO thread this year put it plainly: crawl-log analysis revealed Google was wasting budget on low-value pages while ignoring revenue pages entirely. You cannot find that pattern any other way.
The second question logs answer, which is newly essential, is who is crawling. Splitting requests by user agent shows the true mix of Googlebot, Bingbot, and the AI crawler fleet, and it exposes the volume each one is spending. The catch is that user agent strings are self-reported and easily faked, so counting them naively overstates real search traffic and hides abuse. Reliable numbers require verifying that a request claiming to be Googlebot actually came from Google, and the same for other declared bots. The techniques for separating genuine crawlers from imposters are covered in detecting bot traffic in your logs and in reverse DNS verification.
This is where a server-side view earns its place. Lume reads your traffic at the origin and shows exactly where crawl budget goes, broken down by verified crawler across both search engines and AI agents, so you can see which bot spent how much of your budget on which URLs. Because it classifies and verifies bots server-side rather than trusting the user agent string, the agent analytics view distinguishes a real crawler from one wearing its name, and surfaces the low-value URL patterns quietly draining capacity. It is the difference between guessing at your crawl budget and reading it.
How to protect your crawl budget
Protecting crawl budget comes down to spending it on the URLs you care about and starving the ones you do not, then confirming in logs that bots actually changed behavior. A practical sequence:
- Pull your logs and rank URLs by crawl frequency. Find the most-crawled paths and check how many are parametered, filtered, duplicated, or dead. This is your waste inventory.
- Fix the structural leaks first. Collapse redirect chains to single hops, resolve duplicate URLs with canonicals and 301s, and return real 404 or 410 status codes for content that is gone.
- Contain faceted navigation. Block parameter patterns in robots.txt, avoid linking internally to filtered combinations, and canonicalize filtered pages to their parent category.
- Keep your sitemaps honest. Include only canonical, indexable URLs and keep
<lastmod>accurate so demand concentrates on pages that actually changed. - Speed up your origin. Faster, cacheable responses raise the capacity limit and let every crawler fetch more real pages per session. Supporting 304 responses for unchanged content saves budget on both sides.
- Audit and shape AI-crawler access. Identify the AI crawlers in your logs, decide which you want, and rate-limit or disallow the ones fetching low-value URLs at high frequency. Blocking is a scalpel, not a hammer.
- Watch for silent blocking. Aggressive rate limits or CDN rules can accidentally throttle Googlebot along with the bots you meant to restrict, and the symptom is a quiet drop in crawl volume. Catch it early with a routine like detecting intermittent crawler blocking.
None of these steps is one-and-done. Crawl budget optimization is an ongoing habit because catalogs grow, new parameters appear, and the AI-crawler mix shifts month to month. The sites that keep budget under control are the ones that treat their logs as a standing report, not a one-time audit.
Frequently asked questions
Does crawl budget matter for small websites?
For most sites under roughly 10,000 URLs with a clean structure and a reasonably fast server, crawl budget is not a limiting factor and optimizing it is not worth the effort. It becomes worth managing on large catalogs, publishers, sites with faceted navigation, and anywhere log data shows bots repeatedly crawling low-value URLs while important pages go stale.
How do I check my crawl budget?
Start with Google Search Console’s Crawl Stats report for a Googlebot-only view of total requests, response times, and status-code breakdowns over the trailing 90 days. For the complete picture, including AI crawlers, analyze your raw server logs, which record every bot request and let you see exactly which URLs and status codes are consuming capacity.
Do AI crawlers like GPTBot affect crawl budget?
Yes. AI crawlers consume the same server capacity as search crawlers, and because that capacity is shared, aggressive AI-crawler load that slows your origin can reduce the crawl capacity available to Googlebot. They also spend their own crawling effort, which is wasted when it lands on low-value or duplicate URLs instead of the content you want represented in AI answers.
Should I block AI crawlers to save crawl budget?
Not by default. Many AI crawlers feed the answer engines where you want to be cited, so blocking them wholesale can cost you visibility. The better approach is to shape access: verify which bots are hitting your site, let them reach your valuable canonical URLs, and rate-limit or disallow only the paths, such as faceted and parametered URLs, that waste budget for every crawler.
What is the fastest way to reduce crawl budget waste?
Collapsing redirect chains and containing faceted navigation usually deliver the largest gains fastest. Redirect chains force bots to make several requests to reach one page, and uncontrolled faceted navigation can multiply a small catalog into tens of thousands of crawlable URLs. Fixing those two, then confirming the change in your logs, resolves the majority of waste on most large sites.
How is crawl budget different from indexing?
Crawling and indexing are separate steps. Crawl budget governs whether and how often a bot fetches a URL. Indexing is Google’s later decision about whether a crawled page is worth storing and ranking. A page can be crawled and still not indexed, but a page that is never crawled can never be indexed, which is why wasted crawl budget delays or prevents pages from appearing in search at all.