Playbook · AI search
Letting AI crawlers in
Before working on what your pages say, make sure the systems you want to cite them can read them. This takes an afternoon and removes the one cause of invisibility that has nothing to do with content.
In our study of well-known SaaS sites, very few blocked AI crawlers. It is still worth ruling out, because when it is the cause, nothing else you do will help.
The steps
6 steps- 1Read your robots.txt
- 2Decide on training and answer crawlers separately
- 3Check the firewall as well
- 4Make sure the content is in the HTML
- 5Add an llms.txt
- 6Add structured data where it fits
Use this when
- You are about to start AI search work and have never checked.
- Your site sits behind a firewall or bot-protection service.
- Your site was rebuilt or moved recently.
The steps
6 steps, in order
The technical checklist before any AI search work: robots.txt, llms.txt, rendering and structured data.
- 1
Read your robots.txt
Open yoursite.com/robots.txt and look for rules that name AI crawlers, or a blanket Disallow: / under User-agent: *. The free crawler access checker does this for you and reports each crawler separately.
- 2
Decide on training and answer crawlers separately
Some crawlers gather pages to train models, such as GPTBot and CCBot. Others fetch a page at the moment a product needs it to answer a question, such as OAI-SearchBot, ChatGPT-User and PerplexityBot.
You can allow the second group and block the first. Blocking the second group removes you from those products' answers.
- Answer crawlers allowed on public pages
- A deliberate decision on training crawlers
- Private areas disallowed for everyone
- A Sitemap line in robots.txt
- 3
Check the firewall as well
robots.txt is a request. Bot-protection services can block crawlers before they ever read it, and some do so by default. Check the service's settings for AI crawlers, and look in your server logs for their requests.
- 4
Make sure the content is in the HTML
Many crawlers do not run scripts. View the page source, not the rendered page, and confirm the main text is there. Content that appears only after a script runs may never be seen.
- 5
Add an llms.txt
A plain-text file at yoursite.com/llms.txt that says what the site is and links to the pages that matter. It is cheap to add. Nobody can yet show that it changes what AI systems say, so treat it as housekeeping.
- 6
Add structured data where it fits
Mark up FAQs, articles, products and your organisation with schema.org structured data. It states plainly what a page is about, for search engines and AI systems alike.
What goes wrong
The mistakes to avoid
Blocking everything on principle
A blanket block on AI crawlers also blocks the ones that would cite you. If the concern is training, block the training crawlers by name.
Assuming robots.txt is the whole picture
A site can allow every crawler in robots.txt and still turn them away at the firewall.
Stopping here
Access is a precondition. Being cited depends on what the pages say once they can be read.
How to tell it worked
The crawler access checker shows the crawlers you chose as allowed.
Server logs show AI crawlers fetching pages and receiving a 200 response.
The page source contains the main text without scripts running.
More playbooks
Want this run for you?
These are the steps we follow on every account. Book a call to see them on your site.
