Review robots.txt rules for AI user agents, confirm your pages return real content in the raw HTML, and read your server logs.
For marketers and developers who suspect AI tools cannot see their content. You will confirm what each bot can fetch and fix what blocks it.
If an AI tool says nothing about your product, or describes it wrongly, the first thing to check is not your writing. It is whether the tool could read your pages at all. Access problems are common, easy to miss and quick to fix once you find them.
1Read your robots.txt, rule by rule
Open yoursite.com/robots.txt. Write down every rule that names an AI user agent, and any blanket rule that applies to all bots. Then decide what you intend for each type of bot, because they do different jobs.
| Kind | What it does | Examples (check each vendor's docs) |
|---|---|---|
| Training crawler | Collects content to train models | GPTBot, Google-Extended, ClaudeBot |
| Search crawler | Builds an index for AI search | OAI-SearchBot, PerplexityBot |
| User-triggered fetch | Reads a page when someone asks about it | ChatGPT-User |
2See what a simple crawler sees
Many bots read only the first response and do not run JavaScript. Fetch the page the way they do and look for the content you care about.
curl -s https://example.com/pricing | grep -i "<h1"
curl -s https://example.com/pricing | grep -i "per seat"If neither command finds your heading or your plan details, the content is added by JavaScript after the page loads. A crawler that does not run scripts will see an empty page. Ask your developers to render content pages on the server or at build time.
3Check the layers in front of your site
A firewall, a CDN's bot protection or a hosting rule can block legitimate crawlers even though robots.txt allows them. The crawler just receives an error or a challenge page.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "OAI-SearchBot" https://example.com/pricing- A 200 means the page was served
- A 403 or a challenge page means something is blocking the bot
- A 429 means rate limiting is hitting it
- A 5xx points to a server problem under load
4Confirm in your logs
Server or CDN logs show who actually requested what. Search them for each bot you allowed, then look at the status codes it received. A bot that appears but only gets errors is as blind as one that never arrives.
grep -i "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c5Fix, then recheck on a schedule
Worked example
Why nobody could cite a pricing page (illustrative)
A made-up analytics tool finds that AI assistants never mention its pricing. robots.txt allows every bot. But a curl fetch of the pricing page shows no plan names, because the plans are loaded by a script. The firewall also returns a challenge page to unknown bots.
The team asks developers to render the plans on the server, adds an allow rule for the search and user-triggered bots they want, and rechecks the logs a week later. The bots now receive a 200 with the plan details in the HTML.
Platform migrations, new security tools and hosting changes can reintroduce blocks without anyone noticing. Put a recheck in the calendar after each one.
Say it in 30 seconds
The 30-second version
Read your robots.txt rule by rule, separating training bots from search bots. Fetch a key page without JavaScript to see what a simple crawler gets, check your firewall isn't blocking legitimate bots, and confirm in your logs that the ones you allowed arrive and get a 200.
Worksheet
AI crawler access audit
The full guide and a worksheet to fill in, as a PDF you can print or share with your team. The worksheet covers:
- Open yoursite.com/robots.txt and list every rule for an AI user agent
- Decide for each: allow or block, and write the reason
- Separate training crawlers from search-facing bots and user-triggered fetchers
- Fetch five key pages with curl and confirm the main heading is in the raw HTML
- and 6 more
One email with the PDF. No list, no follow-ups.
