A crawl trap is a URL pattern that produces a near-infinite set of pages. Common causes are calendars with a next-month link forever, filters that combine in every order, session IDs in URLs and relative links that append a path each time.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
Spotting traps
Look in log files and crawl reports for huge numbers of URLs sharing one pattern, or for crawl paths that keep getting longer.
Closing them
Stop generating the links, add noindex or canonical tags where pages must exist, or disallow the pattern in robots.txt. Fixing the source is better than blocking the symptom.
Common mistakes
Blocking in robots.txt only
Blocked URLs can still be discovered and listed from links. Remove the links too.
Frequently Asked Questions
Yes. Faceted filters and calendar widgets can create thousands of URLs on a small site.
Keep reading
- Crawl Budget: Crawl budget is the number of URLs a search engine is able and willing to crawl on a site in a given period.
- Faceted navigation: Faceted navigation lets users filter a listing by attributes. Each filter combination can create a new URL, which can flood crawlers with near-duplicates.
- URL parameters: URL parameters are key-value pairs after a question mark in a URL. They can filter, sort or track, and can create many duplicate URLs for crawlers.
- Log File Analysis: Log file analysis means reading server access logs to see exactly which URLs search engine and AI crawlers requested, when, and what they got back.
- Crawl Stats Report: The Crawl Stats report in Google Search Console shows how Google crawls a site, including request volume, download size, response time, host status and response types.
- Critical CSS: Critical CSS is the minimal set of styles needed to render the visible top of a page, inlined in the HTML so the first paint does not wait on stylesheets.
- Dynamic rendering: Dynamic rendering serves pre-rendered HTML to crawlers and client-side JavaScript to users. Google now calls it a workaround and not a long-term solution.
- Edge rendering: Edge rendering runs server-side rendering code on CDN servers close to the visitor, so HTML is generated nearby instead of at one central origin.
