# Does a SaaS Site Need to Care About Crawl Budget?

> Crawl budget rarely limits a typical SaaS site. See Google's rough thresholds, the exceptions that apply to SaaS, and what to fix if URLs sit undiscovered.

Source: https://comppound.com/blog/crawl-budget-does-a-saas-site-need-to-care (published 2026-10-30)

Crawl budget is the number of URLs Google can and wants to crawl on your site in a given period, and most SaaS sites don't need to manage it. [Google's crawl budget guide](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) is aimed at sites with more than a million unique pages changing about weekly, or more than 10,000 pages changing daily. Google calls those numbers rough estimates. If your marketing site, blog and docs add up to a few thousand pages, a page that isn't getting indexed almost certainly has a different problem.

There is one exception that catches SaaS companies, and it is the reason this topic is worth ten minutes of your time. Google's guide also applies to sites with many URLs reported as "Discovered - currently not indexed", whatever their size. A SaaS site can reach that state by accident, because products create URLs faster than anybody notices.

## How Google decides how much to crawl

Two things set the budget, and neither is a number you can request.

**Crawl capacity** is how hard Google can hit your server without harming it. The limit rises while the site responds quickly and falls when it slows down or returns 5xx or 429 errors. It is shared across Google's crawlers.

**Crawl demand** is how much Google wants to crawl. It depends on the URLs Google knows about, including duplicates, how popular those pages are, and how often they need rechecking.

Most small sites are limited by demand, not capacity. Google could crawl far more of your site than it wants to. This is why tricks to "save budget" usually change nothing: freeing up capacity doesn't help when capacity was never the constraint. It also explains the most common real fix, which is reducing the junk URLs that inflate what Google thinks it should check.

## Where SaaS sites accidentally create crawl problems

A brochure site has a fixed set of pages. A SaaS product can generate an almost unlimited set of them, usually without anyone on the marketing team knowing. Say you run a 60-person collaboration SaaS with 3,000 real public pages. These are the places the other URLs come from.

**Public workspaces and shared links.** If customers can publish a board, form or report on your domain, every one of them is a URL Google can find. Some of these have value. Most are empty, duplicated or abandoned.

**Faceted template and integration directories.** A template gallery filterable by industry, team and use case can produce thousands of filter combinations with near-identical content. See our post on [template galleries as an SEO asset](/blog/template-galleries-as-a-saas-seo-asset) for how to choose which combinations deserve a page.

**Programmatic pages published ahead of demand.** Large sets of generated integration or use-case pages can outrun what Google is willing to crawl, particularly on a young domain.

**Docs versioning.** Every old product version kept live multiplies the docs, and most of those copies will never be useful to a searcher.

**Parameters and session links.** UTM parameters on internal links, sort orders, calendar pages and in-app links with tokens all look like separate URLs.

**Staging and preview deployments.** A preview host left open to crawlers can duplicate the whole site on another domain.

## How to tell whether crawl budget is your actual problem

Check these in order. Each takes minutes, and the first answer often ends the investigation.

1. **Compare your real page count to the indexing report.** In Search Console, open the page indexing report and look at the total known URLs. If you have 3,000 real pages and Google knows of 90,000 URLs, you have a URL generation problem whatever you call it.
2. **Read "Discovered - currently not indexed".** Google found these URLs and hasn't crawled them yet. Sample the list. If it is full of filter pages and customer workspaces, you have found the source. If it contains your new blog posts, look at quality and internal links first, since those pages may lack signals showing they are worth crawling.
3. **Separate it from "Crawled - currently not indexed".** That status means Google fetched the page and chose not to index it. More crawling won't fix it, because crawling wasn't the blocker. Look at the content instead.
4. **Open Crawl Stats.** Under Settings in Search Console, this report shows requests over time, average response time and the breakdown by response code and purpose. Rising response times, or lots of 5xx errors, mean capacity may be a real limit for you.
5. **Look at server logs if you have them.** Logs show what Googlebot fetches on a normal day. Seeing it spend its visits on `/workspace/` URLs and parameter variations is the clearest possible diagnosis.

Once this is clean, see [how to get new SaaS blog posts indexed faster](/blog/how-to-get-new-saas-blog-posts-indexed-faster).

## Fixes that work, and fixes that don't

Google is unusually direct here, and some of its advice contradicts what people commonly do.

**Don't use noindex to save crawl budget.** Google still requests the page, then drops it after seeing the tag, so the crawl has already been spent. Noindex is the right tool for keeping a thin page out of results. It isn't a crawling control.

**Don't use robots.txt as a temporary switch.** Google advises against temporarily blocking pages to shift budget elsewhere, and says freed budget only gets reallocated if the site is already hitting its capacity limit. Use robots.txt for sections that should never be crawled, such as internal search results, filter combinations or infinite calendars.

**Do choose the right tool for each URL type.** The table below is a reasonable starting point for a SaaS site.

| URL type | Treatment |
| --- | --- |
| Filter combinations with no search demand | Block in robots.txt, and stop linking to them |
| Customer workspaces and shared boards | Noindex by default, with an opt-in for ones worth indexing |
| Duplicate URLs with parameters | Canonical to the clean URL, and clean up internal links |
| Old docs versions | Canonical to the current version, or remove them |
| Removed or expired pages | Return 404 or 410 |
| Preview and staging hosts | Password protection |

**Do speed up server responses.** Faster responses let Google fetch more with the same capacity. This matters most for server-rendered directories, where one slow database query can affect thousands of pages.

**Do strengthen the pages you care about.** Link to new and important pages from pages Google already crawls often, and keep [your XML sitemap](/blog/xml-sitemap-for-saas-sites-what-to-include-how-to-submit) limited to canonical URLs that return a 200. Popularity feeds demand, and internal links are the part of that you control.

**Do flatten redirect chains.** Long chains can have a negative effect on crawling, and they usually accumulate through past redesigns.

## What this means for AI crawlers

Google's guidance covers Google, and other crawlers set their own pace. The same cleanup still helps. A site with fewer junk URLs and quick responses is cheaper for any crawler to read, and your useful pages are less likely to be lost among thousands of empty workspaces. Your server logs will show which ones visit and what they spend time on, which is useful context when deciding [whether to block AI crawlers in robots.txt](/blog/should-saas-sites-block-ai-crawlers-in-robots-txt).

Going back to the collaboration SaaS, the sensible plan is modest: noindex customer workspaces by default, block the filter combinations nobody searches for, and link to the real templates from the gallery's main pages. Then watch the known URL count fall and the "Discovered" list shrink over the following weeks. If neither moves, look again at page quality, since crawl budget wasn't the constraint after all.

Keeping product-generated URLs out of the way of the pages that sell is a small example of the work behind [the system we run for SaaS sites](/system).

## Frequently asked questions

### Does crawl budget affect rankings?

Not directly. Being crawled doesn't improve a page's ranking. Crawl budget matters only because a page Google hasn't crawled can't be indexed or updated in results.

### How do I check my crawl budget?

Google doesn't report a budget figure. Use the Crawl Stats report in Search Console for requests and response times, and the page indexing report for URLs marked "Discovered - currently not indexed".

### Does a small SaaS site need to worry about crawl budget?

Usually not, unless the product creates large numbers of public URLs. If Google knows about far more URLs than you have real pages, deal with where those URLs come from.

### Can I ask Google to crawl my site more often?

There is no setting that raises crawl demand. Faster responses, better internal links, popular pages and fewer junk URLs are what you can influence.

## About Comppound

Comppound runs SEO and AI search for B2B SaaS companies: a person sets the plan and reviews samples and every high-risk page, and AI speeds up production.

- [Check a page free](https://comppound.com/tools/seo-checker): Check one of your pages before you change anything else
- [See pricing](https://comppound.com/pricing): Want the content written, reviewed and published for you?