Comppound

Research

Research we ran ourselves

Studies on how SaaS companies show up in search and in AI answers. Each one publishes its method, its limits and the full dataset, so you can check the numbers.

268 SaaS sites12 Oct 2026

Do SaaS companies block AI crawlers?

Mostly, no. We read the robots.txt and llms.txt of 268 well-known SaaS sites to find out who lets AI crawlers in.

  • 15 of 251 sites block at least one AI crawler
  • 1 site blocks a crawler that fetches pages to cite
  • 171 of 268 publish an llms.txt
Read the study

Sites blocking each crawler

251 sites
  • Bytespider11 sites
  • CCBot6 sites
  • GPTBot3 sites
  • meta-externalagent3 sites
  • ClaudeBot2 sites
The five most-blocked of the ten crawlers checked

More studies are added here as we run them.

Want to know where your own site stands?

Ask for a free AI visibility report, prepared by a person, on whether AI products name you for your buyers' questions.

Get the free report

Check your crawler access in a few seconds

The free checker reads your robots.txt with the same rules as our study.

Check crawler accessRead the playbooks