How Googlebot Crawls a SaaS Site

Googlebot is the name of Google's web crawler, the program that fetches pages so Google can consider them for search. On a SaaS site, it arrives through links and sitemaps, fetches your pages as a smartphone by default, renders them in a headless browser when needed, and comes back at a rate based on how fast your server responds and how valuable your pages seem. Knowing how it behaves explains most of the odd things that you see in Search Console.

Here's what it does, what it can't do, and what that means for a site with a product, docs and a blog all under one domain.

How it finds your URLs

Googlebot doesn't start from your homepage and methodically work through your site. Google's overview of how search works describes a constantly updated list of known URLs, fed from a few sources:

• Links on pages it already knows, both on your site and on others
• Sitemaps you've submitted
• Previously crawled URLs, which it rechecks from time to time

Google doesn't forget a URL simply because you've removed the link to it. If an old page was once known, it may be retried for a long time, which is why 404s can turn up in your reports for pages that you deleted years ago. That isn't a problem in itself, and it doesn't need fixing unless a page should still exist.

What happens on each visit

Say you run a 30-person SaaS for scheduling field technicians. Here's what happens when Googlebot visits your /features/route-optimization page.

It checks robots.txt. Before crawling, Google checks whether your robots.txt file allows it. It caches that file, so a change may not take effect immediately. If robots.txt returns a server error, Google may pause crawling your site for a time, which is one reason to make sure that file's reliably served.

It fetches the HTML. It requests the page, using the smartphone crawler in most cases. It reads the status code, the headers and the page itself. Google documents a limit on how much of a file it crawls: for Search, the first 2MB of a supported file type. That's plenty for ordinary marketing pages, though a page that inlines very large scripts or data could push its content past the cutoff.

It decides what to do next. A 200 goes on to rendering and indexing. A 301 sends it to the new URL. A 404 tells it the page is gone. A 500 or 429 tells it to slow down.

It renders the page if needed. A headless version of Chromium runs the JavaScript, with the rendered result then read for content and links. Googlebot's Chromium is kept up to date, so it can handle modern JavaScript. See JavaScript SEO for React and Next.js for where this goes wrong.

It collects links and adds them to its list. That's how your new integration page gets found, assuming that something links to it.

What Googlebot doesn't do

Plenty of SEO mistakes come from assuming that it behaves like a person.

It doesn't log in. Anything behind authentication is invisible to it. If your docs or changelog need a session, they won't be crawled.

It doesn't click buttons or fill in forms. Content that's revealed only by a click, or a directory you can only reach by using a search box, may never be found. Real <a href> links are the reliable path.

It doesn't keep state between pages. Google's JavaScript troubleshooting guide notes that data such as cookies is cleared across page loads. So a page which shows a welcome modal to first-time visitors will show it every time. If a modal or consent wall covers your content, or only loads the content after someone agrees, this can cause problems.

It doesn't scroll like a person. Infinite scroll with no underlying URLs can hide anything past the first batch.

It doesn't accept every permission. Requests for a camera, location or similar get declined, so features that need them won't render.

It doesn't always follow what you've asked. Canonicals and sitemap entries are hints. It can crawl a URL you'd rather it didn't, unless robots.txt blocks it.

How to see it in your own logs

Server logs let you see what it actually does, as opposed to what you hope it does. Filter for the user agent containing "Googlebot" and look for:

• Which sections get the most requests, and which get few or none
• How often your money pages are visited
• How many requests go to URLs that you wish didn't exist, such as parameters or shared customer pages
• What status codes it's receiving

Anyone can fake a user agent, and many scrapers pretend to be Googlebot. So check with Google's verification method, a reverse DNS lookup or a comparison with its published IP ranges, before making decisions from your logs.

If you're on a platform such as Vercel or Cloudflare, logs may not be retained for long by default, so you may need to set up a log drain. If that's too much, the Crawl Stats report in Search Console offers a useful summary, including the split by purpose, file type and response code.

Google has more than one crawler

Seeing other names in your logs is normal. Google's list of crawlers includes separate agents for images, video and other products. Google-Extended is a robots.txt token that lets you control whether your content is used for certain AI training and grounding purposes, and Google's documentation on AI features explains how the controls differ. It's separate from what appears in Search. To be excluded from Google's AI features in Search, you'd use the normal Search controls, and those affect your regular results as well.

Rules for one agent don't automatically transfer to the others. If your robots.txt has a section for Googlebot, then Googlebot follows that section and ignores the * section, which sometimes catches people out.

How to shape where it spends time

You can't tell Googlebot how often to visit, but you can influence what it does.

Make important pages easy to reach. Link to them from your navigation and hubs. In the scheduling SaaS example, the route optimization page should be linked from your menu, your solutions pages and any relevant blog posts. See our guide to website architecture.

Keep your responses fast and healthy. Server errors and slow responses cause it to reduce crawling. Caching your marketing pages helps with this.

Reduce the junk. Stop generating and linking to URLs that have no value. See does a SaaS site need to care about crawl budget.

Keep your sitemap accurate. List canonical URLs only, and include a truthful lastmod date. Google has said that it can use lastmod when it's consistently accurate. Our XML sitemap guide covers this.

Use correct status codes. A 404 or 410 on a page that's really gone is better than a soft 404 with a 200 status. A 503 during maintenance is better than serving errors or a maintenance page as if it were content.

Don't try to control the rate unless you have to. Googlebot adjusts itself. If it's truly overloading your server, Google documents how to reduce the crawl rate, including temporarily returning 500, 503 or 429 codes. Use these only in an emergency, as they also reduce crawling of the pages you care about.

Common situations on SaaS sites

"Google hasn't crawled my new page." Check whether anything links to it. Requesting indexing helps with a single page, but a page with no links to it remains at a disadvantage. See how to get a SaaS site indexed.

"Google keeps crawling old URLs." That's normal. If they return a correct 404 or redirect, you can usually ignore them.

"Crawling dropped suddenly." Check for server errors, firewall changes and robots.txt edits around that date. Often, a security rule added for a different reason has started challenging bots.

"Googlebot visits the app subdomain." Put a login in front of anything sensitive, and don't rely on robots.txt for privacy. It's a public file, and it only asks crawlers to stay out.

If you want your crawl data read regularly and turned into fixes, see the system we use.