The Crawl Stats report shows Google's crawling history for a site: how many requests it made, how much it downloaded, how quickly the server responded, and any availability problems it met. You open it from Property settings in Search Console. Google says the report is aimed at advanced users and that sites with fewer than a thousand pages generally do not need it.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
What the report shows
The top of the report charts three totals over time, and the breakdowns underneath explain what sits behind them.
- Total crawl requests, including failed attempts and fetches skipped because robots.txt was unavailable
- Total download size in bytes
- Average response time for resources fetched
- Host status, based on robots.txt fetching, DNS resolution and server connectivity
- Crawl responses, such as OK, redirects, 404s and server errors
- File type, such as HTML, images, JavaScript, CSS and PDF
- Crawl purpose, split into discovery of new URLs and refresh of known ones
- Googlebot type, such as smartphone, desktop, image and page resource load
Limits to know about
Google makes the report available only for root-level properties: a Domain property, or a URL-prefix property at the root of the site. Requests to other domains, including resources you host elsewhere, are not counted.
Example URLs are samples, and Google acknowledges that some requests may not be counted, so the numbers can differ slightly from server logs. Use logs when you need every request.
How to read it
Look for changes in the pattern rather than for a target number. A sudden rise in requests may mean a crawl trap or a large publish. A drop alongside rising response times or server errors may mean Google is backing off because the server is struggling. Google's crawl budget guide says the capacity limit falls when a site slows down or returns 5xx or 429 responses.
Example
A spike traced to a filter parameter
Hypothetical marketplace SaaS Stallbright saw total crawl requests climb sharply over a month without any new content. The example URLs under the discovery purpose were full of filter combinations from a new sort control.
The team blocked those parameter URLs from crawling and stopped linking to them, then watched the pattern settle over the following weeks.
Common mistakes
Reading more crawling as a win
A rise in requests can be waste, such as parameter URLs or redirects. Check the breakdowns before celebrating.
Looking for it on a subfolder property
The report is available only for root-level properties. Add a Domain property or a root URL-prefix property to see it.
Frequently Asked Questions
Generally no. Google says sites with fewer than a thousand pages generally do not need it, although host status can still be useful when diagnosing an outage.
Google says some requests may not be counted, and logs can include other bots using a Googlebot user agent. Verify the requests before comparing totals.
Keep reading
- Crawl Budget: Crawl budget is the number of URLs a search engine is able and willing to crawl on a site in a given period.
- Google Search Console: Google’s free tool for site owners that shows which searches bring up a site, how often it is clicked, and whether Google can crawl and index its pages.
- Log File Analysis: Log file analysis means reading server access logs to see exactly which URLs search engine and AI crawlers requested, when, and what they got back.
- Googlebot: Googlebot is the web crawler Google uses to fetch pages for Search. It discovers URLs, downloads them and passes them on to be rendered and indexed.
- Crawl trap: A crawl trap is a part of a site that generates endless URLs, so crawlers waste time there instead of on your real pages.
- Critical CSS: Critical CSS is the minimal set of styles needed to render the visible top of a page, inlined in the HTML so the first paint does not wait on stylesheets.
- Dynamic rendering: Dynamic rendering serves pre-rendered HTML to crawlers and client-side JavaScript to users. Google now calls it a workaround and not a long-term solution.
- Edge rendering: Edge rendering runs server-side rendering code on CDN servers close to the visitor, so HTML is generated nearby instead of at one central origin.
