Crawl Budget

What it means

Crawl budget is the practical limit on how many pages Googlebot will fetch from your site in a given period. It's a function of two factors: crawl rate (how many requests your server can handle without slowing down) and crawl demand (how interested Google is in re-crawling your content).

For small sites — under 10,000 URLs — crawl budget rarely matters. Google has more than enough capacity to crawl everything that needs crawling. For large ecommerce sites with 100K+ URLs, faceted navigation, paginated listings, and millions of parameter-based variants, crawl budget becomes a bottleneck that determines what gets indexed.

Optimizing crawl budget means making it easy for Google to find what matters and ignore what doesn't. The main levers are: clean URL architecture, smart canonical and robots directives, sitemap prioritization, and removing low-value pages from the crawl path entirely.

Key takeaways

• Crawl budget matters mainly for sites with 10,000+ URLs
• Server response time directly affects crawl capacity — faster servers get more crawls
• Log file analysis is the only way to see what Googlebot actually crawls
• Most wasted crawl budget on ecommerce comes from faceted nav and parameter URLs