OAI-SearchBot is the user agent OpenAI uses to crawl websites so they can be shown and linked in ChatGPT's search features. OpenAI documents it separately from GPTBot, its training crawler, and says content crawled by OAI-SearchBot is not used to train its foundation models. Sites that block OAI-SearchBot may not appear in ChatGPT search answers, though they can still be shown as plain navigational links in some cases.
Allowing and verifying it
Control it with a robots.txt rule naming OAI-SearchBot. OpenAI publishes the IP ranges it uses, so you can verify that requests claiming to be OAI-SearchBot really come from OpenAI. OpenAI's documentation has noted that robots.txt changes can take around a day to be reflected in search.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/What to check beyond robots.txt
Robots.txt is only one layer. Firewalls, CDN bot management and rate limits can block or challenge the crawler without anyone noticing.
- Search server logs for the OAI-SearchBot user agent and check status codes.
- Confirm CDN bot rules allow it, using OpenAI's published IP ranges.
- Make sure important content is in the initial HTML response.
- Check that pages you want surfaced are not behind logins or interstitials.
Example
A firewall rule nobody remembered
Rookfield, a fictional scheduling company, allows OAI-SearchBot in robots.txt, yet ChatGPT never cites its feature pages. Logs show the crawler receiving 403 responses from a WAF rule that blocks unfamiliar bots. Adding an allow rule for OpenAI's published ranges fixes the responses, and Rookfield monitors logs to confirm normal crawling resumes.
Common mistakes
Confusing it with GPTBot
GPTBot is for training. OAI-SearchBot is for search. Blocking one does not block the other.
Trusting the user agent string alone
Anyone can claim to be OAI-SearchBot. Verify against OpenAI's IP list before giving special access.
Questions
No. It makes your pages eligible to be found. Whether they are used depends on relevance, quality and factors OpenAI does not publish.
OpenAI's documentation says no, and that training is governed by GPTBot. Check the current docs, since policies can change.
