ClaudeBot: the crawler behind Claude, and what it does with your site

By Muntazir · Published 13 September 2026

A site owner blocking ChatGPT from training on a site adds a matching line for ClaudeBot, expecting the same effect on Claude. It doesn’t do what the GPTBot rule does — the mechanics of that one are covered separately. ClaudeBot never touches a live Claude answer in the first place — it isn’t in the path a real conversation takes, so disallowing it changes nothing about whether Claude cites the site tomorrow.

What ClaudeBot actually does

Anthropic’s own support documentation is specific about the job: ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” That’s the whole function. It doesn’t power search, it doesn’t fetch pages on a user’s behalf, and it has no role in generating the answer a Claude user reads. It builds a dataset for a future model, on a timeline nobody outside Anthropic can see.

Blocking it is straightforward. User-agent: ClaudeBot followed by Disallow: / in robots.txt opts a site out, and Anthropic’s documentation confirms its bots honor that directive.

ClaudeBot is not the only Claude crawler

Anthropic names two more agents, and each does a different job. Claude-User fetches a specific page the moment a person’s question sends Claude there — a live, on-demand request, not a training pass. Claude-SearchBot indexes content ahead of time, the way a search engine’s crawler does, so Claude’s own search results have something to draw on when a query needs it.

The three don’t share a switch. A site that disallows ClaudeBot alone still gets visited by Claude-User and Claude-SearchBot, because each carries its own line in robots.txt and its own job. Writing one blanket rule under a vague “block Claude” comment, the way a lot of ad-hoc advice suggests, either misses two of the three agents or catches all three at once — training, live citation and indexing — when the intent was usually just the first.

What blocking it actually changes

Disallowing ClaudeBot stops one thing: future pages being collected for training. It does not remove a site from Claude’s live answers, because that path never ran through this bot. It does not touch content Anthropic already collected before the rule went up, and Anthropic’s documentation doesn’t say whether a page already gathered gets purged from a future training run once a site opts out after the fact — that’s a real gap in what’s publicly known, not something this piece can resolve.

Google-Extended isn’t a crawler at all

The second term worth understanding here works differently from every crawler above it. Google-Extended doesn’t send its own requests — Google’s developer documentation is explicit that it isn’t a ranking signal and has no effect on inclusion in Google Search. Googlebot does the actual fetching it always did; Google-Extended is a downstream permission that governs whether that already-collected content trains or grounds Gemini and Vertex AI. Disallowing it doesn’t hide a site from Google Search, and it doesn’t remove it from AI Overviews, which draw on the live search index rather than a training set.

That makes it the opposite case from ClaudeBot: one is a distinct bot with its own user agent and its own robots.txt line; the other is a permission flag riding on a crawler that was already there.

What robots.txt can’t tell you

Robots.txt records what a site owner intends to allow. It says nothing about what the server actually does when a request shows up. The audit engine behind this site checks both, separately, for exactly that reason — the full breakdown of what gets checked and how is in the audit methodology: it reads robots.txt for what’s permitted, and it also sends a real request identifying itself as ClaudeBot — among a small handful of named crawlers checked this way — to see what the server does when that name shows up. The two answers sometimes disagree. A CDN’s bot-management layer can quietly 403 a crawler that robots.txt allows, and a site owner reading only the text file has no way to know it’s happening. A robots.txt checker reads the file itself for free, though it takes a live probe like the one above to catch a server that’s quietly out of step with it.

Google-Extended never gets that second check, because there’s no separate identity to send a request as — it isn’t a crawler, so there’s nothing at the server level to probe. The only lever it has is the one line in robots.txt.

Knowing which bot does what settles half the question. Knowing whether the server backs up what robots.txt claims settles the other half, and that part isn’t visible from the file alone.