GPTBot is the user agent OpenAI uses to crawl public web pages for possible use in training its generative AI models. OpenAI introduced it in 2023 and documents it alongside OAI-SearchBot and ChatGPT-User. Blocking GPTBot tells OpenAI not to use your content for training; it is not meant to remove you from ChatGPT search, which uses OAI-SearchBot. GPTBot is one of the most commonly blocked crawlers on the web.
Controlling GPTBot
Use robots.txt to allow or disallow it, in whole or for specific folders. OpenAI publishes the IP ranges GPTBot uses for verification.
# Block training on the whole site
User-agent: GPTBot
Disallow: /
# Or block only certain sections
User-agent: GPTBot
Disallow: /customers/
Disallow: /internal-docs/The trade-off for SaaS companies
Training data shapes a model's built-in knowledge, which is what an assistant relies on when it answers without searching. Allowing training crawlers gives your public descriptions of your product a chance to be part of that knowledge in future models. Blocking it keeps your content out of future training by OpenAI, but does not remove anything already learned and does not stop other sites that quote you from being used.
Many SaaS companies allow training on marketing pages and docs, where broad awareness helps, and block sections with customer data or gated material. There is no single right answer; the decision depends on how you value the content itself versus its role in awareness.
Example
A split policy
Ambervale, a fictional analytics vendor, sells original benchmark reports as a lead magnet and does not want them used for training. It disallows GPTBot on its /reports/ folder, allows it elsewhere, and keeps OAI-SearchBot allowed sitewide so ChatGPT search can still surface and link the report landing pages.
Common mistakes
Blocking GPTBot to stop ChatGPT referrals
Search runs on a different crawler. Referrals come from ChatGPT search, which GPTBot rules do not control.
Expecting retroactive removal
Blocking affects future crawling. It does not remove data from models already trained.
Questions
No. It is an OpenAI crawler and has no connection to Google Search.
OpenAI provides some data controls and opt-out processes, but robots.txt is the documented control for future crawling. Check OpenAI's current policies for anything beyond that.
