What Is Crawl Budget?
Crawl budget is the number of URLs Googlebot will crawl on your site within a given timeframe. It is determined by two factors: crawl rate limit (how fast Googlebot can crawl without overloading your server) and crawl demand (how much Google wants to crawl your site based on its freshness and popularity).
Why Crawl Budget Matters
For small sites (under a few thousand pages), crawl budget is rarely a concern. For large sites — e-commerce stores with faceted navigation, news sites with millions of articles, SaaS platforms with dynamic URLs — crawl budget is critical. If Googlebot exhausts its budget before reaching your important pages, those pages won't be indexed.
What Wastes Crawl Budget
- Faceted navigation URLs — filter combinations like
/products?color=blue&size=Mcan generate millions of near-duplicate URLs - Session IDs in URLs —
/page?session=abc123creates unique URLs for each visitor - Duplicate pages — the same content accessible at multiple URLs
- Soft 404s — pages that return 200 but show "not found" content
- Redirect chains — multi-step redirects slow down crawling
- Low-quality pages — thin pages that don't contribute to the index
How to Optimize Crawl Budget
- Block low-value URL patterns in robots.txt
- Use canonical tags to consolidate duplicate pages
- Eliminate redirect chains — make redirects point directly to final URLs
- Fix soft 404s — return proper 404 status codes
- Improve page speed — faster servers allow Googlebot to crawl more pages per session
- Use internal linking to guide Googlebot to your most important pages
- Submit key pages via the Google Indexing API to ensure they're crawled regardless of budget — use the Indexing API guide or bulk URL indexing for large batches
Monitoring Crawl Budget
Google Search Console's Crawl Stats report (Settings → Crawl stats) shows crawl activity over the past 90 days including total crawl requests, average response time, and how Googlebot found each URL. Use this to identify crawl budget waste.