Crawlability Problems on SaaS Sites and How to Fix Them

Crawlability issues are anything that stops a search engine reaching a page or following links through your site. On SaaS sites, the usual causes are a broad robots.txt rule, bot protection on the CDN, navigation that relies on JavaScript, and important pages with no internal links. Each is detectable in an afternoon, and many can be fixed by changing a single rule or template.

Crawlability is a separate question from indexability. A page can be perfectly crawlable and still be left out of the index, and that calls for a different diagnosis. Here we're only dealing with whether the crawler can get there in the first place.

Find out what the crawler is seeing

Don't rely on what you see in your browser. You're logged in, you've got JavaScript running, and no firewall is challenging you. Use these checks instead.

The URL Inspection live test. In Search Console, test a page you're worried about. It tells you whether Google could crawl it, and it shows the HTML and screenshot Google received. A page that looks fine for you and blank in this test has a real problem.

The Crawl Stats report. In Search Console's settings, this shows requests over time, response codes and host status. A sudden drop in requests, or a jump in server errors, usually lines up with a release.

A site crawl. Crawl from your homepage with a tool such as Screaming Frog. Then compare the URLs it found with your sitemap. Sitemap URLs the crawl never reached have no internal links leading to them.

Server logs, when you have them. They show what Googlebot actually requested, which is the closest thing to the truth. Verify the bot, because plenty of scrapers claim to be Googlebot. Google publishes how to verify its crawlers.

Problem 1: A robots.txt rule that's too broad

SaaS teams often add rules to keep the app out of Google, and the rules catch more than intended. Robots.txt rules match by prefix, so this:

User-agent: *
Disallow: /app

blocks /app/dashboard, and also /app-integrations and /apps-marketplace. Add the trailing slash, as in Disallow: /app/, if you only mean that folder.

Also check for blanket rules that blocked staging and came along in the launch, such as Disallow: /. And check that your rules don't block the JavaScript and CSS needed to render pages, since Google can't then see the page properly. Search Console's robots.txt report shows which version Google has fetched. For the full syntax, see Google's robots.txt documentation.

Problem 2: Firewalls and bot protection

Cloudflare, Vercel and similar providers make it easy to switch on bot protection. That's a sensible thing to do when you're under attack, and it's easy to forget. A challenge page returns something to the crawler, but not your content.

It's particularly relevant now, as these rules can also block AI crawlers you may want to reach you. Check your settings for bot rules, rate limits and country blocks. Googlebot crawls largely from the US, so blocking traffic by country can have unexpected effects. Then look at your provider's logs for blocked requests from verified bots. See our guide to whether you should block AI crawlers for how to decide which ones to allow.

Watch for 429 status codes as well. They tell crawlers to slow down, and Google reduces its crawling in response. If your API routes or server-rendered pages rate limit too eagerly, you may be throttling Google without knowing.

Problem 3: Links crawlers can't follow

Google says it can generally crawl links only if they are <a> elements with an href attribute. On modern SaaS sites, plenty of things look like links and aren't.

• A card in your integrations directory with an onClick handler
• Pagination built as a "load more" button with no real URLs behind it
• Tabs that load content on click
• Filter dropdowns that change the content without changing the URL
• Search-only directories, where a page is reachable only by typing into a box

Say you run a 35-person customer support SaaS with 150 integration pages. If the directory shows a dozen integrations and a "load more" button, the other pages may be reachable only through your sitemap, if at all. Fix this by giving those pages real links: paginated pages with their own URLs, category pages or a simple A to Z index. Google has specific advice on pagination and incremental loading, which applies well beyond e-commerce.

Our JavaScript SEO guide covers the broader rendering issue.

Problem 4: Orphan and deeply buried pages

An orphan page has no internal links pointing at it. These tend to come from campaign landing pages, content moved during a redesign, and posts that drop off the blog's first pages. They may stay indexed for a while through the sitemap, but they send a weak signal, and they're easily lost.

A related problem is depth. If a blog post can only be found on page 40 of a chronological archive, it may be crawled rarely. To improve that, link older content from topic hubs, related articles and other evergreen pages. There's no magic number of clicks, but anything you care about should be easy to reach. Website architecture for SaaS deals with the wider design.

Problem 5: Server errors and slow responses

Repeated 5xx errors tell Google that your site's struggling, so it backs off. On a SaaS site, the marketing site can suffer from problems elsewhere, such as a shared database, a slow CMS API or a build system that fails intermittently.

Check Crawl Stats for error spikes and average response time, then match them against deployments. If your directory pages are server-rendered, look at caching: a page that takes several seconds to generate is a problem for everyone, including crawlers. See our guide on crawl budget for how capacity affects crawling.

For planned downtime, return a 503 status with a Retry-After header. That tells crawlers the problem's temporary, whereas a 200 page saying "we're down" could get indexed as if it were your content.

Problem 6: Redirect chains and loops

Redirects from one redesign pointing at redirects from an earlier one will build up over time. Long chains waste crawling and can end up not being followed at all. Your crawl report will list them. Update each redirect to point directly at the final page, and change your internal links to go straight there. We cover the options in 301 vs 302 vs 307 vs 308.

Problem 7: Too many low-value URLs

Sometimes the issue is too many pages to crawl, instead of pages Google can't reach. Faceted filters, calendar pages and customer workspaces can absorb a lot of Google's attention on your site. This mostly matters when the junk vastly outnumbers real pages, and the crawl budget guide above explains what to do.

How to prioritize the fixes

Use a simple order:

1. Anything blocking money pages: Fix it the same day. Pricing, features and comparison pages can't afford to wait.
2. Rules that affect a whole section: A single robots.txt change or a firewall rule often takes minutes.
3. Template problems: Links in a directory or navigation require an engineer, but only need fixing once.
4. Individual orphan pages and chains: Tidy these up as routine maintenance.

Going back to the support SaaS, the first job would be checking that bot protection isn't challenging Google, the second would be fixing the integrations directory, and the third would be linking its orphaned landing pages.

Then stop the problem coming back. Add three checks to your release process: robots.txt hasn't changed unexpectedly, important pages return a 200 for a Googlebot user agent, and the main navigation still contains real links. These are cheap to automate, and they catch the errors that are most expensive to find late. Add them to your technical SEO checklist.

If you'd rather someone else kept an eye on this, see the system we run.