CCBot and Common Crawl: The Archive Behind AI Training

By Muntazir · Published 24 October 2026

On 4 November 2025, The Atlantic published an investigation alleging that Common Crawl had misled publishers about removing their content from its archive — pointing to a July 2023 takedown request from The New York Times whose articles, according to detailed reporting on the exchange, were still present in the archive. Common Crawl’s executive director, Rich Skrenta, responded the same day. Asked why the content hadn’t simply been deleted, his answer was blunt: the archive format “is meant to be immutable. You can’t delete anything from it.”

That sentence is the whole story of what CCBot — the crawler behind that archive — actually commits a site to, in a way the usual advice to check a robots.txt file doesn’t cover.

A crawl that feeds many labs, not one

Common Crawl’s own page for CCBot describes its purpose plainly: collecting data from publicly accessible pages to build a free, open repository for research. It says nothing about any specific AI company, because it doesn’t need to. The archive gets published for anyone to download, and a lot of anyones have. A 2024 Mozilla Foundation study of 47 language models released between 2019 and October 2023 found that at least 64% of them were trained on Common Crawl data — and for GPT-3 specifically, Mozilla put the figure at over 80% of its training tokens.

That is the structural difference between CCBot and the crawlers covered elsewhere on this site. GPTBot feeds OpenAI’s own training pipeline; ClaudeBot feeds Anthropic’s. Blocking either one is a decision about a single company. CCBot’s captures become a public file set that dozens of unrelated labs have pulled from without running a crawler of their own — which is also why both OpenAI and Anthropic each gave Common Crawl $250,000 in 2023, a funding disclosure reported alongside the foundation’s November 2025 response to the takedown allegations. The crawlers this site has already covered are, in effect, funding the one this post is about.

Why a block today doesn’t undo yesterday

A Disallow: / rule for CCBot works the same way any robots.txt rule does: it is a request the crawler is expected to honour going forward. What it cannot do is reach backward into a snapshot already taken. Common Crawl’s archive is built from monthly crawls, each saved as a set of WARC files — a format designed, by the foundation’s own account, never to be edited after it’s written. A page CCBot captured in 2022 is still sitting in the 2022 snapshot in 2026, independent of whatever the site’s robots.txt says today.

Common Crawl does have a process for publisher requests: a page on its site keeps what it now calls an Opt-Out Ledger, a running public list of every legal removal request it has received since 2025, with a commitment to keep honouring them. But by the foundation’s own description, that process works forward — filtering a domain out of future crawls and out of what its own search tools will return — rather than rewriting files already published. One exchange cited in the reporting involved the Danish Rights Alliance, which requested removal of its members’ content in July 2024; five months on, a Common Crawl attorney estimated that roughly half of that flagged content had been removed from what the tools could surface. Common Crawl disputed that the organisation had misled anyone; it did not dispute that the underlying files are permanent.

What a “no captures” result actually means

There’s a sharper point buried in Common Crawl’s own response to that investigation: it said its index and search tools exist to help people find data, not to certify legal removal status — so a domain returning zero captures can mean either “never crawled” or “filtered out of what the public tools will show.” The two look identical from the outside.

That ambiguity matters for anyone checking their own standing, including the deeper tier of this site’s own audit, which queries Common Crawl’s public index directly for a domain across its two most recent snapshots and counts how many captures and distinct URLs come back — a different question from what robots.txt simply permits. Zero captures gets treated as unremarkable for a small or recent site. For a domain large and established enough that it should already be in the archive, the same zero result is flagged as worth investigating — most often explained by a CCBot block in robots.txt, though Common Crawl’s own account of its index tools means a past removal request can’t be ruled out as a cause either. This check doesn’t run on a free scan; it’s part of the deeper one, alongside a separate step that checks whether AI models actually name the brand when asked about its category — when both come back empty together, the pairing at least rules out “the models know the site but chose not to mention it.”

Who actually blocks it

Not many sites, as it happens, and barely more than block the training crawler whose own posts already cover this ground. A daily scan of the top 1,000 websites’ robots.txt files by Known Agents put CCBot’s disallow rate at 24% as of 23 October 2026 — close to, but a touch ahead of, the 22% it recorded for GPTBot the same day. Both numbers describe a clear majority of large sites choosing, for now, to leave the door open.

Leaving that door open is less a decision to let one AI company train on a site’s content, and more a decision to let an archive nobody at the site controls, pulled from by labs the site owner will likely never hear of by name, keep what it already has and add to it indefinitely. Whether that trade is worth making probably depends less on CCBot itself than on how a publisher feels about data that was never designed to be taken back.