Crawl budget is roughly how many pages Google is willing and able to crawl on your site in a given time. Google has said most small and medium sites do not need to worry about it.
When it matters
- Large sites with tens of thousands of URLs or more.
- Online stores with endless filter combinations.
- Sites that auto-generate many pages.
- Sites with many errors or slow servers.
How to use crawl budget well
- Keep the site fast and stable.
- Block or control useless URL variations (filters, sorting, session IDs).
- Fix redirect chains and errors.
- Keep XML sitemaps clean; generate them with our sitemap generator.
- Use robots.txt carefully for low-value areas; build it with our robots.txt generator.
- Avoid thin and duplicate pages.
How to check
Search Console’s Crawl stats report shows how often Google crawls your site and any problems.
Read getting pages indexed.
Want the full picture? This article is part of SEO: the complete guide, our in-depth guide with everything in one place.
