Crawl budget describes how much crawling a search engine will do on your site. Google explains it as the combination of crawl capacity (how much your server can handle without slowing down) and crawl demand (how much Google wants to crawl, based on popularity and freshness). For most sites it is not a constraint. It starts to matter on very large sites, fast-changing sites, and sites that generate huge numbers of low-value URLs.
Who should care
Google's own guide on crawl budget is aimed at large sites, giving rough examples such as sites with a million or more unique pages that change about weekly, or ten thousand or more pages that change daily, as well as sites with many pages stuck as "Discovered, currently not indexed". A typical SaaS marketing site with a few hundred or a few thousand pages rarely hits a crawl budget limit.
The exception is when a site accidentally creates millions of URLs: faceted filters, calendar pages that go on forever, session IDs, or a public app route that generates a URL per user object. Then crawl budget becomes very real.
What wastes crawl budget
Anything that gives crawlers more URLs than you have real pages uses up crawl capacity that could go to pages that matter.
- Faceted navigation and filter parameters producing endless combinations
- Infinite spaces such as calendars or paginated search results
- Duplicate URLs from tracking parameters or inconsistent links
- Long redirect chains and soft 404s
- Slow server responses and frequent 5xx errors, which lower crawl capacity
How to improve it
Reduce the URL space: block infinite or low-value parameter spaces in robots.txt, keep internal links pointing at canonical URLs, and remove soft 404s. Make the server fast and stable; Google raises crawl capacity when responses are quick and healthy and lowers it after errors. Keep sitemaps clean and lastmod accurate so Google can find changed pages without re-crawling everything.
Note that noindex does not save crawl budget: Google still has to crawl a page to see the tag. Blocking in robots.txt is what prevents the request.
Measuring it
Search Console's Crawl Stats report shows total requests, response codes, file types and average response time over the last 90 days. Server log analysis is more precise, showing exactly which URLs Googlebot fetched and how often.
Example
Public share links that multiplied
Consider a diagramming SaaS, Boardly, that lets users publish public share links at boardly.example/s/[id]. Millions of these exist, most are tiny or empty, and Googlebot spends much of its activity on them. Meanwhile, new template and comparison pages take a long time to be crawled.
Adding noindex to empty boards, blocking low-value share paths where appropriate, and linking new marketing pages from the main navigation shifts crawling toward the pages the company wants found.
Common mistakes
Worrying about crawl budget on a small site
If your site has a few thousand pages and Google crawls new ones promptly, crawl budget is not your problem. Look at content quality or internal links instead.
Using noindex to save crawl budget
Noindexed pages are still crawled. Use robots.txt to stop requests to large low-value URL spaces.
Questions
Not directly. You can make crawling cheaper (faster server, fewer junk URLs) and pages more worth crawling (links, freshness). Google can lower crawl rate on request, but not raise it.
Not directly. It affects whether and how quickly pages are crawled, which comes before indexing and ranking.
Every crawler limits how much it fetches, but third-party AI crawlers do not document their scheduling in the detail Google does, so treat their behavior as unknown and watch your logs.
