OAI-SearchBot vs GPTBot: Which One to Allow in robots.txt

By Muntazir · Published 18 September 2026

Disallowing GPTBot in robots.txt does not remove a page from ChatGPT’s search answers. That crawler only governs whether OpenAI can use a site’s content to train future models. The crawler that decides whether a page shows up — and gets cited — in ChatGPT’s live search results carries a different name: OAI-SearchBot. OpenAI’s own crawler documentation states plainly that the two settings are independent: a site can allow OAI-SearchBot while disallowing GPTBot, or the reverse, and neither choice touches the other.

What does OAI-SearchBot actually do?

It crawls pages to build the index ChatGPT’s search feature draws its citations from. OpenAI’s documentation is direct about what happens when a site blocks it: the page “will not be shown in ChatGPT search answers,” though it can still turn up as a bare navigational link elsewhere in a response. That’s the whole job — nothing about training, nothing about the model’s underlying knowledge, just whether a page is eligible to be surfaced and cited the next time someone asks ChatGPT a question that page could answer.

GPTBot and OAI-SearchBot aren’t two names for the same thing

A single rule meant to shut ChatGPT out entirely, written under one comment in robots.txt, often disallows both crawlers at once, on the assumption that they do the same job. They don’t:

A site that disallows only GPTBot, expecting to disappear from ChatGPT, keeps showing up in citations, because the crawler that actually feeds those citations was never touched. A site that disallows both under one blanket rule opts out of training and citation in the same stroke — which may be intended, or may just be what the broader rule happened to catch.

Where does ChatGPT-User fit in, and does it follow robots.txt?

ChatGPT-User is neither of the above. It isn’t a scheduled crawler building anything ahead of time — it fires once, mid-conversation, the moment a person’s question causes ChatGPT to reach for a specific page right then. OpenAI’s documentation is careful to note that “robots.txt rules may not apply” to this fetch, because a person, not an automated job, requested it. Disallowing GPTBot and OAI-SearchBot together still leaves this third path open: a live, on-demand fetch that behaves more like a person clicking a link than a crawler indexing a site.

That distinction matters more for OAI-SearchBot than it first appears. Blocking it removes a page from the pre-built index most ordinary ChatGPT searches draw from. It does nothing to stop ChatGPT-User from fetching and citing that same page on the spot, if one specific conversation happens to point there directly. Disallowing the index crawler lowers the odds of ambient, everyday citation. It doesn’t seal the page off.

What robots.txt can’t tell you about OAI-SearchBot specifically

Reading robots.txt only shows what a site intends to permit — not what actually happens when a request arrives. This site’s own audit engine, documented in full here, checks both, separately, for a named set of crawlers: it reads robots.txt for what’s declared, and it also sends a live request under a crawler’s real identity to see whether a CDN or WAF quietly blocks it regardless of what the text file says. That second check currently covers five agents — Googlebot, GPTBot, ClaudeBot, PerplexityBot and Bingbot — because those are the ones site owners ask about most and the ones a bot-management rule is most likely to name explicitly.

OAI-SearchBot isn’t one of the five yet, even though robots.txt access for it is tracked among eighteen named AI crawlers a report checks. That’s a real gap, not a rounding error: a report can currently say a site’s robots.txt allows OAI-SearchBot, but it can’t yet confirm the server actually lets a request identifying itself that way through — the exact failure mode that already shows up for GPTBot and ClaudeBot, where a CDN’s AI-crawler-blocking switch silently 403s a bot that robots.txt was written to allow. There’s no reason to assume OAI-SearchBot is exempt from that same failure mode; there’s just no direct check for it yet.

Checking what a site actually allows

Two separate questions decide whether a page can be cited in ChatGPT’s live answers: does robots.txt permit OAI-SearchBot, and — a question no robots.txt reading can answer — does the server actually respond to it the way the file says it should. The gptbot and AI crawler rundown covers the broader set of eighteen tracked agents if the goal is a full audit rather than this one comparison. A robots.txt checker answers the first question directly and for free. The second one, for most named crawlers, needs an actual probe — and for this particular crawler, that probe doesn’t exist yet.