Research · 268 SaaS sites
Do SaaS companies block AI crawlers?
Mostly, no. 15 of the 251 sites whose robots.txt we could read block at least one AI crawler, and only 1 blocks a crawler that fetches pages to answer a question.
Crawled 12 October 2026. By Comppound.
Sites blocking each crawler
251 sites- Bytespider · ByteDance11
- CCBot · Common Crawl6
- GPTBot · OpenAI3
- meta-externalagent · Meta3
- ClaudeBot · Anthropic2
- Google-Extended · Google2
6%
block at least one AI crawler (15 of 251 sites)
1
site blocks a crawler that fetches pages to cite
64%
publish an llms.txt (171 of 268 sites)
24%
name an AI crawler in robots.txt (59 sites)
A hand-picked list of 268 well-known SaaS companies, not a random sample. Method and limits are below.
Blocking is rare, and aimed at training crawlers
15 of 251 sites (6%) block at least one of the ten AI crawlers we checked. The most-blocked is Bytespider (ByteDance), blocked by 11 sites.
The crawlers that fetch a page in order to answer a question are left alone almost everywhere: figma.com is the only site that blocks one.
Who blocks what
15 sites- descript.comBytespider
- together.aiGoogle-Extended
- amplitude.comBytespider
- calendly.comCCBot
- canva.comGPTBotClaudeBotApplebot-ExtendedCCBotBytespidermeta-externalagent
- figma.comGPTBotOAI-SearchBotChatGPT-UserClaudeBotPerplexityBotGoogle-ExtendedCCBotBytespider
- loom.comGPTBot
- attio.comCCBotBytespider
- zoominfo.comCCBotBytespidermeta-externalagent
- github.comBytespider
- postman.comBytespider
- kajabi.comCCBotBytespider
- freshbooks.comBytespider
- proofpoint.commeta-externalagent
- housecallpro.comBytespider
Most sites that mention AI crawlers do it to let them in
59 of 251 sites (24%) name at least one AI crawler in robots.txt. 44 of those 59 do not block any of the ten from the site: the crawler is named to give it its own rules, usually the same access as everyone else with a few private folders excluded.
The other 192 sites say nothing about AI crawlers at all, which leaves them covered by the general rules.
Sites naming each crawler
Allowing or blocking- GPTBot · OpenAI49 (20%)
- ClaudeBot · Anthropic47 (19%)
- PerplexityBot · Perplexity47 (19%)
- Google-Extended · Google41 (16%)
- OAI-SearchBot · OpenAI39 (16%)
- ChatGPT-User · OpenAI39 (16%)
- CCBot · Common Crawl32 (13%)
- Applebot-Extended · Apple25 (10%)
- Bytespider · ByteDance24 (10%)
- meta-externalagent · Meta24 (10%)
64% publish an llms.txt
171 of 268 sites serve a real llms.txt: a plain-text file at the root of the site that points AI systems to the pages that matter. Adoption runs from 47% in Collaboration & productivity to 79% in Support & CX.
This counts whether the file exists. It says nothing about whether AI systems use it. For comparison, 236 of 251 readable robots.txt files (94%) include a sitemap line.
Share with an llms.txt, by category
268 sites- Support & CX79%
- Developer tools75%
- E-commerce & CMS75%
- CRM & sales72%
- HR & people70%
- Analytics & data67%
- Security & IT65%
- Fintech & finance58%
- Marketing58%
- AI products55%
- Vertical SaaS50%
- Collaboration & productivity47%
What to do with this
Access is rarely the problem
If your product is missing from AI answers, a blocked crawler is the first thing to rule out and the least likely cause. Check it once, fix it if it is wrong, and move on.
What is left is the pages themselves: whether they answer the question a buyer asks, in a passage an AI system can lift. That is the work described in AI Search and the answer capsules playbook.
Method
How we ran it
Everything needed to check the numbers or run the study again.
The sample
268 SaaS companies across 12 categories, chosen because they are well known. It is not a random sample, and it leans towards larger, US-based companies, so read the percentages as a description of this list rather than of all SaaS.
The crawl
On 12 October 2026 we requested /robots.txt and /llms.txt from each site once, identifying ourselves as ComppoundResearchBot/1.0.
What counts as blocked
A crawler counts as blocked when the rules that apply to it disallow the whole site (Disallow: /) with no matching Allow. A crawler named in robots.txt follows its own rules; one that is not named follows the rules for all crawlers.
What counts as an llms.txt
Only a response that is a text file in the format, with a markdown heading or link. An HTML page or an error message returned at that address does not count.
Limits
- 17 of 268 sites had no readable robots.txt for us: 3 have no file, and the rest refused or rate-limited our request. They are left out of the robots.txt figures. They stay in the llms.txt figure, where a refused request counts as no file, so that figure is a floor.
- We only test access to the whole site. A crawler that is allowed here can still be shut out of particular folders, and partial rules are not counted as blocking.
- robots.txt is a request, not a wall. A site can also block crawlers at its firewall, which this study cannot see, and a crawler can ignore the file.
- This is one day. Companies change these files, and the figures will drift.
Crawlers checked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent. The dataset is free to reuse; a link back to this page is appreciated.
Frequently asked questions
Almost none of the companies in this sample do. A crawler that cannot read a page cannot cite it, so blocking the crawlers behind ChatGPT's search, Perplexity and similar products removes you from their answers. Blocking training-only crawlers is a separate decision about how your content is used, and 15 of 251 sites here made it for at least one crawler.
A training crawler, such as GPTBot or CCBot, gathers pages that may be used to train a model. An answer crawler, such as OAI-SearchBot, ChatGPT-User or PerplexityBot, fetches a page at the moment a product needs it to answer a question, and is the kind that produces a citation. robots.txt lets you treat them differently.
This study cannot say. It counts how many sites publish the file, not whether the file changes what an AI system does. An llms.txt is cheap to add and tells an AI system which pages matter, but treat it as housekeeping rather than a ranking factor.
Use the free AI crawler access checker at comppound.com/tools/ai-crawlers. It reads your robots.txt with the same rules used for this study and reports each crawler separately.
Check your own site in a few seconds
The free checker reads your robots.txt with the same rules as this study.
