5xx status codes tell the client that the server, not the request, is at fault. Common ones are 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable and 504 Gateway Timeout. When Googlebot meets them, it slows its crawl rate to avoid overloading the site, and if errors persist for a URL, Google eventually drops it from the index.
How Google responds
Google documents that 5xx and 429 responses make its crawlers temporarily slow down. Already-indexed URLs are kept for a while, but if they keep returning errors they are removed. If robots.txt itself returns a 5xx, Google may stop crawling the site until it gets a valid response, which can make a short outage much more costly.
Common causes on SaaS sites
Marketing sites built on server rendering can fail when an upstream API or CMS is slow. Serverless functions time out. A CDN returns 502 or 504 when the origin is unreachable. Traffic spikes, including aggressive bots, can exhaust resources.
- CMS or API outages during server rendering
- Function timeouts on cold starts or heavy pages
- Origin misconfiguration behind a CDN
- Deploys that briefly serve errors
Monitoring and prevention
Check the Crawl stats report in Search Console for server error spikes and the Page indexing report for server error (5xx) entries. Server and CDN logs show which URLs fail and when. Caching rendered pages at the edge, using static generation where possible, and setting sensible fallbacks for failed API calls all reduce the chance that crawlers see errors.
Example
A CMS outage that crawlers noticed
Fernquill, a recruiting SaaS, renders blog pages on each request from a headless CMS. During a two-hour CMS outage, every blog URL returns 500, and Search Console's Crawl stats show the spike. The team switches the blog to static generation with on-demand revalidation, so a CMS outage now means stale content rather than errors.
Common mistakes
Returning 200 for error pages
An error page with a 200 status can get indexed. Make sure failures return a real 5xx code.
Letting robots.txt depend on the app
If robots.txt is generated by the same server that fails, outages also break crawl permissions. Serve it statically.
Questions
Google does not give a fixed time. Brief errors are usually tolerated; errors that persist across many crawl attempts lead to removal.
