AI crawlers are automated agents that AI companies use to fetch web content. Most companies now run more than one, each with a different job: collecting training data, building an index for AI search, or fetching a page because a user asked about it. Each identifies itself with a user agent and, for most, can be controlled in robots.txt. Getting these rules right decides whether your pages can appear in AI search answers at all.
The main crawlers by purpose
Names and behavior change, so check each vendor's documentation before writing rules. As of late 2026, the commonly documented agents include:
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results), ChatGPT-User (fetches made on behalf of a user).
- Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-requested fetches).
- Perplexity: PerplexityBot (search index), Perplexity-User (user-requested fetches).
- Google: Googlebot powers Search, including AI Overviews and AI Mode. Google-Extended is a robots.txt token, not a separate crawler, that controls use of content for Gemini models outside Search.
- Apple: Applebot-Extended is a token that controls use of content for Apple's model training.
- Common Crawl: CCBot builds an open web archive that many models have been trained on.
Training versus search
The split between training and search crawlers lets you make separate choices. A SaaS company that wants to be recommended in ChatGPT search should allow OAI-SearchBot even if it decides to block GPTBot. Blocking a search crawler is the more consequential choice, because it can keep your pages out of that product's live answers.
User-initiated agents are a special case. Some vendors state that robots.txt rules may not apply to fetches a user explicitly requested, because they act more like a browser than a crawler. Read the current documentation for each one.
A sample robots.txt
This example allows AI search crawlers and blocks two training crawlers. It is an illustration of syntax, not a recommendation for every site.
# Allow AI search and live-answer crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /What to check
Many sites block AI crawlers without knowing it, through a CDN or firewall bot setting rather than robots.txt. Check your CDN's bot management rules, then confirm in server logs that the crawlers you allow are actually getting 200 responses. Also check that key content is in the initial HTML: several AI crawlers are not documented as rendering JavaScript, so content that only appears after scripts run may not be seen.
Example
A fictional CRM's accidental block
Dealroom, a made-up CRM, wonders why Perplexity never cites its comparison pages. Its robots.txt allows everything, but log analysis shows PerplexityBot getting 403 responses. A CDN rule that challenges unknown bots had been switched on during a traffic spike. The team adds an exception for the verified crawlers it wants and confirms 200 responses in the logs the following week.
Common mistakes
Blocking everything with “AI” in the name
Blanket blocks often catch search crawlers along with training crawlers, which can remove you from AI answers you want to be in.
Trusting the user agent string alone
User agents can be spoofed. Vendors publish IP ranges for verification; use them in firewall rules.
Forgetting the CDN layer
robots.txt can say allow while your CDN returns a challenge page. Check the logs.
Questions
OpenAI documents GPTBot and OAI-SearchBot as separate, so blocking GPTBot alone should not stop OAI-SearchBot. Confirm against OpenAI's current bot documentation.
No. AI Overviews and AI Mode are part of Google Search and use Googlebot.
Googlebot does. Most other AI crawlers do not document JavaScript rendering, so server-rendered HTML is the safer assumption.
