A crawl error is any failed attempt by a search engine bot to fetch a page or the site as a whole. Some affect one URL, such as a 404 or a redirect loop. Others affect the whole host, such as DNS failures or an unreachable server. Many are harmless, so the skill lies in separating errors on pages you care about from noise on URLs that should not exist.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
Site level and URL level errors
Site level problems stop a crawler reaching anything, so they come first. Search Console's Crawl Stats report rates host status using robots.txt fetching, DNS resolution and server connectivity.
URL level problems affect individual addresses. The Page indexing report lists them by reason, including Server error (5xx), Redirect error, Not found (404) and URL blocked due to other 4xx issue.
How Google responds to each type
Different errors have different consequences, so check the status code before deciding what to do.
- 5xx and 429: Googlebot temporarily slows crawling, and URLs that keep failing are eventually removed from the index
- Most 4xx codes: content is ignored, and already indexed URLs are removed
- 3xx problems: Googlebot follows up to 10 hops by default, and loops or overly long chains are reported as redirect errors
- Robots.txt fetch problems: reported as part of host status
How to triage them
Start with whether the URL should work at all. A 404 on an old test URL with no links or traffic is normal. A 5xx on a pricing page needs attention today. Sort by page value and by the number of URLs affected, since one bad template or rule often explains hundreds of errors.
Server logs help here, because they show every request a crawler made. Search Console examples are samples rather than complete lists.
Example
Sorting a scary report
Hypothetical billing SaaS Ledgerine saw thousands of 404s in Search Console after retiring a docs version. Most were old URLs with no links or traffic, so they were left alone.
A much smaller group of 5xx errors looked less alarming but came from the integrations template, which timed out under load. Fixing that template was the only urgent job in the report.
Common mistakes
Trying to reach zero errors
Removed pages should return an error. Redirecting every old URL to the homepage to clean up a report can create soft 404s instead of solving anything.
Ignoring short bursts of 5xx errors
These tell Googlebot to slow down, and recurring bursts can be a sign the server is struggling for real users too.
Frequently Asked Questions
A 404 on a page that should not exist is normal. The cost comes when valuable pages or pages with useful links return one by mistake.
Look at the Page indexing report for errors on individual URLs, and the Crawl Stats report in Settings for host status and response breakdowns.
Keep reading
- Crawl Stats Report: The Crawl Stats report in Google Search Console shows how Google crawls a site, including request volume, download size, response time, host status and response types.
- Page Indexing Report: The Page indexing report in Google Search Console shows which of your URLs Google has indexed and gives a reason for each one it has not.
- HTTP Status Codes: HTTP status codes are three-digit server responses that tell browsers and crawlers whether a request succeeded, moved, or failed.
- 5xx server errors: 5xx errors are HTTP status codes meaning the server failed to handle a valid request. Repeated 5xx responses slow crawling and can drop pages from Google.
- Bing Webmaster Tools: Bing Webmaster Tools is Microsoft's free console for site owners. It shows how Bing crawls and indexes a site and offers SEO reports and URL submission.
- Content delivery network: A CDN is a network of servers worldwide that caches and serves a site's files close to each visitor, cutting load times and shielding the origin server.
- Crawlability: Crawlability is how easily search engine and AI crawlers can discover and fetch the pages on a site.
- Googlebot: Googlebot is the web crawler Google uses to fetch pages for Search. It discovers URLs, downloads them and passes them on to be rendered and indexed.
