Cloudflare AI Crawl Control: The Settings That Hide a Site

By Muntazir · Published 1 November 2026

A review of one site’s Cloudflare dashboard, done after two unrelated AI-crawler incidents five weeks apart in 2026, counted six separate settings that can each, on their own, stop an AI crawler from reaching the site. None of the six live in robots.txt, and none of them check what the other five are set to.

The September incident came from a feature that writes rules into robots.txt automatically, to keep the file matching a set of dashboard preferences. The moment one of those preferences changed, the file changed with it — new disallow lines, added without anyone opening a text editor, live and indexed before anyone looked. The October incident, on the same site, came from a completely different setting: a block switch that returns a 403 at the edge regardless of what robots.txt says, so the file stayed accurate and the crawlers still didn’t get through. Two settings, two unrelated causes, the same result: a site that believed it was reachable and was not.

What does AI Crawl Control actually control?

AI Crawl Control gives a site three dashboard preferences — Search, Agent and Training — covering what each category of AI crawler is allowed to do, plus a block switch for individual named crawlers underneath. Which named crawlers fall into which category, and what each one actually does with the content it’s allowed to take, is covered crawler by crawler elsewhere; this is about the switch that overrides whatever any of them is told, regardless of category. Cloudflare’s own announcement states that for new domains with ad-supported pages, Training and Agent crawlers will be blocked by default from 15 September 2026 onward, while Search crawlers stay allowed. A site that signed up before that date, or doesn’t carry ads, may have a different default — which is exactly why checking the live setting matters more than assuming one.

There’s also a fourth lever on some plans: Pay per crawl, which charges listed crawlers for access and returns HTTP 402 to anyone who hasn’t paid. A 402 looks nothing like a robots.txt disallow, and a site owner who has never opened that tab has no reason to expect it.

How does a robots.txt file get rewritten without anyone touching it?

Bot Preference Sync, launched in August 2026, keeps a site’s robots.txt in step with its Search, Agent and Training settings by writing rules directly into the file, ahead of whatever the site already published. Cloudflare’s own documentation shows exactly this: a block of generated Disallow lines sitting above the site’s own rules, refreshed whenever the dashboard settings change or Cloudflare updates its list of known bots.

That’s a reasonable design — robots.txt and the dashboard settings are supposed to agree. The failure mode is that the sync runs in one direction. If a setting gets switched to Block for any reason, by anyone with dashboard access, the file changes with it, silently, the next time a crawler asks for it.

Why did the same site hit two different settings in two months?

Because AI Crawl Control and Bot Preference Sync aren’t the only places a crawler can be stopped, and fixing one does nothing for the others. Security › Bots carries its own switch for blocking AI scrapers, separate from AI Crawl Control. Bot Fight Mode and Super Bot Fight Mode can challenge a crawler as a side effect of fighting something else entirely, which shows up as a 403 in Security Events rather than a dashboard crawler count. WAF rules and rate limiting can match a crawler’s user agent or ASN without anyone writing a rule that mentions AI at all. None of these read robots.txt, and none of them update it — they just answer the request before the file is ever consulted.

How can a check tell “blocked” apart from “blocked and nobody knows it”?

A robots.txt line is a statement of permission, not a measurement of what happens. Treating the two as the same thing is the actual failure here. An audit that reads robots.txt alone can confirm a crawler is permitted and still be completely wrong about whether it is reaching the site — the two questions have to be checked separately, by asking the server directly, with the crawler’s own user agent, from outside the CDN.

This product’s own audit engine does exactly that: it reads what robots.txt permits, separately requests the page as each tracked AI crawler, and compares the two answers. When the file says a crawler is welcome and the server refuses it anyway, that specific combination — allowed on paper, blocked in practice — gets flagged on its own, distinct from a crawler that robots.txt disallows outright. There’s no aggregate figure yet for how often that combination turns up across audited sites generally; the two settings described above are simply the clearest case found so far, and they were caught by hand, before any automated check went looking for the pattern.

What should a site on Cloudflare actually check?

Five things, in this order: the AI Crawl Control category settings (Search, Agent, Training, plus any per-crawler overrides); whether Bot Preference Sync is on, since that’s what turns a dashboard setting into a file change; Security › Bots and Bot Fight Mode, independently of AI Crawl Control; any WAF rule or rate limit that might match a crawler’s user agent without mentioning AI at all; and, if the plan offers it, whether Pay per crawl is enabled for crawlers the site doesn’t want billed. Reading robots.txt answers none of these — a free robots.txt checker only shows what was last written, by whichever of those features touched it last, not what the server does with the request that follows. The audit methodology page lays out exactly which of these is checked from outside the CDN and which isn’t.

Testing from outside helps, but imperfectly. Requesting a page with a crawler’s named user agent and checking for a 403 catches a CDN-level block quickly. It is still only an approximation: Cloudflare verifies some bots, including the major search engines, by IP address rather than by user agent string, so a spoofed request can behave differently from the real crawler. The dashboard’s own per-crawler allowed and blocked counts are the number that actually matters, and they take a day or two to populate after a setting changes — which is long enough that a site owner who checks immediately after flipping a toggle can still draw the wrong conclusion.

A robots.txt file states what a site intends. Whether that intention survives contact with the CDN sitting in front of it is a separate question, answered in a different part of the dashboard, sometimes by a different person, sometimes months apart.