AI crawler access control means deciding which AI bots can read your site and for what purpose. Some bots collect training data, some fetch pages to answer a live question, and some power a search index. They often use different user agents, so one blanket rule can block more than you intend.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
Separate the purposes
Read each vendor's documentation to see which user agent does what. Blocking a training crawler is a different decision from blocking the bot that fetches a page when a user asks about you.
Set rules and verify
Add rules per user agent in robots.txt, then check your server logs to confirm the bots behave as the vendors say. robots.txt is a request, not a lock, so sensitive content needs real access control.
User-agent: GPTBot
Disallow: /private/
User-agent: OAI-SearchBot
Allow: /Common mistakes
Blocking everything by reflex
Blocking search-facing bots can remove you from AI answers you would want to appear in.
Never reviewing the rules
Vendors add and rename user agents. Recheck on a schedule.
Frequently Asked Questions
It tells compliant crawlers not to fetch. It does not remove content already collected and does not bind bots that ignore it.
Keep reading
- robots.txt: robots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request.
- AI Crawlers: AI crawlers are bots run by AI companies to fetch web pages, either to train models, to build a search index, or to answer a user's live request.
- GPTBot: GPTBot is OpenAI's web crawler for collecting content that may be used to train its AI models. It is controlled separately from ChatGPT's search crawler.
- OAI-SearchBot: OAI-SearchBot is OpenAI's crawler for ChatGPT search. Allowing it in robots.txt is what lets your pages be surfaced and linked in ChatGPT's search answers.
- Agentic Browsing: Agentic browsing is when an AI model operates a web browser for a user, reading pages, clicking, scrolling and filling in forms to complete a task.
- AI Agents: AI agents are systems where a language model plans and takes actions with tools, such as searching, browsing, filling forms or calling APIs, to finish a task.
- AI Answer Sentiment: AI answer sentiment is how positively, neutrally or negatively AI assistants describe your brand, and which strengths and weaknesses they repeat.
- AI Answer Variability: AI answer variability is the way the same prompt can produce different answers, sources and brand mentions across runs, users, places and time.
