An assistant cannot cite a page its crawler is not allowed to read. Many SaaS sites block some AI crawlers without meaning to, through a security default, a CDN setting or a robots.txt rule copied from somewhere else.
This lesson is the cheapest fix in the whole track: check access, decide on purpose and make sure the pages are readable.
Short on time? Comppound does this work for SaaS companies. AI-native, and a person reviews every page.
Have us do itTraining crawlers and answer crawlers
AI companies run different crawlers for different jobs. Some collect pages for training models. Others fetch pages to answer a user's question in real time or to build a search index. OpenAI, for example, documents separate user agents for training and for search. You can allow one kind and block the other.
If you want to be cited in answers, the search and user-triggered crawlers are the ones that matter most.
Check robots.txt and your firewall
Robots.txt is where you allow or block crawlers by user agent. But a CDN or bot-protection service can block a crawler before it ever reads robots.txt. Check both: the file, and whether your security settings challenge or block known AI bots.
Comppound can do this for you
Want to be named in AI answers?
Comppound checks what ChatGPT, Perplexity, Gemini and Google's AI features say about your category, opens your site to their crawlers and builds the pages and mentions they cite. AI-native, with every page reviewed by a person.
- AI visibility baseline for your buyer questions
- Crawler access and quotable pages
- Monthly tracking of where you are named
Make the content readable without JavaScript
Many AI crawlers fetch the raw HTML and do not run scripts. If your pricing, comparison or docs pages build their content in the browser, those crawlers see nothing. Render marketing pages on the server or at build time.
What llms.txt does and does not do
llms.txt is a proposed plain-text file at the root of a site that lists the pages most useful to a language model, with short descriptions. It is cheap to add and some tools read it. No major assistant has said it uses the file to decide what to cite, so treat it as a helpful extra rather than a ranking lever.
Worked example · an illustration, not a client result
Tallyroom's access check
The robots.txt was fine. The CDN's bot protection was the problem, and nobody on the team had known it was on.
| Check | Finding | Action |
|---|---|---|
| robots.txt | All AI crawlers allowed | None |
| CDN bot protection | Challenged unknown bots, including some AI crawlers | Allow verified AI crawlers |
| Pricing page HTML | Prices loaded by script | Render prices on the server |
| llms.txt | Missing | Add one listing key pages |
Do it yourself
Check and decide in thirty minutes
- 1Run your domain through the AI crawler access checker.
- 2Decide, on purpose, which training crawlers and which answer crawlers you allow.
- 3Ask whoever runs your CDN whether bot protection blocks AI crawlers.
- 4View the source of pricing and one comparison page, and confirm the text is in the HTML.
- 5Add an llms.txt listing your ten most useful pages.
Free tool for this: AI crawler access checker
Common mistakes
Blocking everything to be safe
Blocking all AI crawlers also blocks the ones that would cite you. Separate training from answering and decide each.
Fixing robots.txt and ignoring the CDN
A firewall rule can override everything robots.txt allows.
Expecting llms.txt to work alone
A file listing pages cannot help if the pages themselves are unreadable or thin.
Rather not do it yourself?
Want to be named in AI answers?
Comppound checks what ChatGPT, Perplexity, Gemini and Google's AI features say about your category, opens your site to their crawlers and builds the pages and mentions they cite. AI-native, with every page reviewed by a person.
- AI visibility baseline for your buyer questions
- Crawler access and quotable pages
- Monthly tracking of where you are named
Questions
Allowing a training crawler lets your public pages be used for training. Allowing an answer crawler lets them be read and cited. You can choose each separately.
Google says Google-Extended controls use for its AI models, while AI features in Search rely on normal Googlebot crawling. Check Google's current documentation before deciding.
Key terms
- AI CrawlersAI crawlers are bots run by AI companies to fetch web pages, either to train models, to build a search index, or to answer a user's live request.
- robots.txtrobots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request.
- llms.txtllms.txt is a proposed Markdown file at a site's root that gives AI tools a short summary of the site and links to its most useful pages.
- GPTBotGPTBot is OpenAI's web crawler for collecting content that may be used to train its AI models. It is controlled separately from ChatGPT's search crawler.
- OAI-SearchBotOAI-SearchBot is OpenAI's crawler for ChatGPT search. Allowing it in robots.txt is what lets your pages be surfaced and linked in ChatGPT's search answers.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether Google may use your content to train Gemini models and to ground answers in the Gemini app.
- User-Triggered FetchersUser-triggered fetchers are AI agents that load a page because a person asked about it, rather than crawling the web on their own schedule.
- Server-Side RenderingServer-side rendering builds a page's HTML on the server for each request, so browsers and crawlers receive complete content without running JavaScript.
