GPTBot and the AI Crawlers: What to Allow, What to Block

By Sameer Bhat · Published 4 September 2026

GPTBot only trains future versions of ChatGPT — blocking it does not remove your pages from ChatGPT’s live answers today. The crawlers that actually gate that are OAI-SearchBot and ChatGPT-User, and a “block ChatGPT” rule broad enough to catch GPTBot commonly catches those two as well, without anyone intending both effects at once. Before adding a blanket rule, know what each crawler actually does.

Which AI crawlers exist?

Our audit engine’s ai-crawlers check tracks eighteen named user agents when it reads a site’s robots.txt: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Applebot-Extended, Amazonbot, Bytespider, Meta-ExternalAgent, cohere-ai, Timpibot and omgili — the crawlers site owners actually ask about, not every bot that has ever fetched a page.

They split into two jobs, and the split matters more than the brand name attached to each one.

Training crawlers fetch pages to build or improve a model over time. GPTBot, CCBot, Bytespider and anthropic-ai fall here. Blocking one of these affects what a future version of that model might know — nothing about the current one, and nothing about what’s already been collected.

Retrieval crawlers exist to power what a user actually sees. Some, like OAI-SearchBot, index pages ahead of time so they’re available to cite; others, like ChatGPT-User, fetch a specific page the moment a conversation needs it. OpenAI’s own publisher documentation confirms the split: GPTBot trains, OAI-SearchBot powers ChatGPT’s search results, ChatGPT-User is the live fetch. Blocking a retrieval crawler has an immediate, visible effect: your pages stop showing up in that product’s answers, today, not in some future training run.

Google-Extended is its own case. Google’s own documentation confirms it doesn’t gate Google Search at all — disallowing it controls whether the content helps train and ground future Gemini models, and Google states plainly that the token isn’t used as a ranking signal and doesn’t remove a site from AI Overviews, which draw on the live search index rather than training data.

What does each one actually do?

The mistake that causes the most damage: treating GPTBot and ChatGPT-User as the same thing, which a lot of “how to block ChatGPT” advice does. A robots.txt rule broad enough to catch all three OpenAI agents at once (Disallow: / under a generic OpenAI comment, say) blocks training and removes you from ChatGPT’s live answers in the same stroke, whether or not that was the intent.

The same split applies across vendors, though not identically. ClaudeBot and anthropic-ai train; Claude-User and Claude-SearchBot retrieve. Perplexity is different again: its own help centre states plainly that it doesn’t build foundation models, so nothing PerplexityBot crawls goes into pre-training — it indexes purely for Perplexity’s own search and answers, the same job OAI-SearchBot does for ChatGPT. Perplexity-User is the live fetch. There’s no training crawler to block in Perplexity’s case at all.

What does blocking actually cost you?

If your goal is to stay out of a retrieval crawler’s live answers, blocking it does that — cleanly and immediately, for a crawler that actually honours robots.txt. Not all of them reliably do: Cloudflare’s own post on the matter (August 2025) documented Perplexity using undeclared, rotating user agents to keep crawling sites that had disallowed PerplexityBot, after which Cloudflare de-listed it as verified. A Disallow rule assumes good-faith compliance; it is a request, not a lock.

If your goal is to keep your content out of training, blocking the training crawler only stops future fetches by that one crawler. It has no effect on:

• Content already collected before you added the rule.
• Content the model provider licenses directly from a publisher or dataset, which never touches your robots.txt at all.
• Any of the other training crawlers you didn’t also block — CCBot, Bytespider, and the rest.

So “block GPTBot to keep my content out of AI” is narrower than it sounds. It’s a real, effective decision about one company’s future crawl — not a way to opt out of AI training in general.

What robots.txt does — and doesn’t — prove

Here’s the file this site serves, which allows all eighteen tracked crawlers explicitly rather than relying on a catch-all:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

...(fourteen more, one block per crawler)

User-agent: *
Disallow: /api/

An explicit Allow line per crawler, rather than leaning on the fact that no rule means “permitted,” is a deliberate choice: if a future CDN change ever adds a blanket AI-bot block, it collides visibly with a named line instead of silently changing what “no rule” used to mean.

But robots.txt only tells you what a crawler is permitted to do — a voluntary convention, not an enforcement mechanism. It also can’t show you the opposite mistake: a site that means to allow a crawler but has quietly locked it out anyway. Our ai-crawlers check probes GPTBot, ClaudeBot and PerplexityBot directly at the server level, with their real user agents, and compares the result against what robots.txt says. When the file allows a crawler but the server 403s it — usually because a CDN’s bot layer (Cloudflare’s “block AI scrapers” toggle is a common culprit) added a rule the file was never updated to reflect — the finding is labelled “allowed by robots.txt but refused by the server,” and the fix lives in the CDN’s dashboard, not the text file. Nobody chose to block anything here, and there’s no way to know it happened without probing the server directly.

A robots.txt that reflects your actual choice

There’s no universal right answer — the decision is per-crawler, depending on whether you want training inclusion, live citation, both, or neither for each product. What matters is that the rule you ship matches the choice you meant to make:

1. Decide per product, not per company. OpenAI alone has three agents doing three jobs — blocking “OpenAI” as one idea usually means writing three rules, or one broader than intended.
2. Check what robots.txt says, and separately, what the server does. They can disagree, and only one is enforced against you.
3. Remember compliance is voluntary. Most named crawlers honour a Disallow; not all reliably do, as the Perplexity case above shows.

Not sure what your robots.txt actually permits, or whether your server backs it up? Run the free scan to check it against all eighteen tracked AI crawlers, try the robots.txt checker or llms.txt generator directly, or read the audit methodology for exactly what each check measures.