Crawlability describes whether a crawler can find a page and successfully download it. A crawlable page is linked from somewhere a crawler can see, is not blocked by robots.txt, returns a normal status code, and does not depend on interactions a bot cannot perform. If a page is not crawlable, nothing else about it matters for search.
How crawlers discover pages
Crawlers find new URLs mainly by following links from pages they already know, and by reading sitemaps. Google's documentation is specific about links: it reliably follows standard a elements with an href attribute. Links that only exist as JavaScript click handlers, or buttons that change the view without a URL, may never be discovered.
<!-- Crawlable -->
<a href="/integrations/hubspot">HubSpot integration</a>
<!-- Not reliably crawlable -->
<span onclick="router.push('/integrations/hubspot')">HubSpot integration</span>
<a href="#" onclick="openIntegration('hubspot')">HubSpot integration</a>Common crawlability blockers
Most blockers fall into a handful of patterns.
- robots.txt rules that block important folders or rendering resources
- Content behind logins, forms or search boxes with no links to it
- Links generated only on user interaction, like infinite scroll without paginated URLs
- Server errors, timeouts, or bot protection returning 403 or challenge pages
- Redirect loops and very long chains
- Pages with no internal links at all (orphans)
Crawlable versus indexable
Crawlability is about access, indexability is about permission and eligibility. A page can be crawlable but carry a noindex tag, or be crawlable but canonicalized to another URL. A page blocked in robots.txt is not crawlable, but its URL may still be indexed from links. Diagnose them separately.
Checking crawlability
A desktop crawler starting from your homepage shows what a bot can reach through links alone. Compare that list with your sitemap and your CMS; pages missing from the crawl are either orphaned or blocked. Search Console's URL Inspection tool tests whether Googlebot can fetch a specific URL. Bot protection services deserve a specific check, because they sometimes challenge or block legitimate crawlers, including AI crawlers you may want to allow.
Example
An integrations directory with no links
A hypothetical support platform, Helpharbor, builds a searchable integrations directory. Each integration has its own page, but the directory index only shows results after a visitor types in a search box. The pages are in the sitemap, so Google finds some, but they have almost no internal links.
Adding a crawlable, paginated list of all integrations, grouped by category with plain links, gives every page a path from the homepage.
Common mistakes
Infinite scroll with no paginated URLs
Crawlers do not scroll. Provide paginated URLs with links, so every item is reachable.
Bot protection that blocks search engines
Aggressive firewall or anti-bot rules can serve challenges to Googlebot or other crawlers. Verify real crawler IPs and allow them.
Questions
It helps discovery, but pages should also be linked internally. Links tell crawlers which pages matter and help them be recrawled.
Google renders JavaScript and can find links added to the DOM, as long as they are a elements with href values. Many other crawlers do not render JavaScript, so server-rendered links are safer.
