Meta-ExternalAgent: Meta's AI Crawler, and What It Takes From a Site
On 7 September 2026, a site’s robots.txt quietly grew eight new lines nobody on the team had written. A Cloudflare feature called Bot Preference Sync had prepended Disallow: / rules for eight named crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Amazonbot, Bytespider, and meta-externalagent — sitting directly above the site’s own Allow: / line for every one of them. The change went live and got indexed before anyone noticed it had happened. Nobody had touched the file. A dashboard toggle, several layers away from the text editor anyone would normally open to change crawler access, had rewritten it instead.
Meta-ExternalAgent was one of the eight names caught in that rewrite, and it is also one of the easiest crawlers on that list to misidentify. Meta runs several bots with similar names and different jobs, and getting one confused with another is a quick way to block the wrong thing entirely.
What does Meta-ExternalAgent actually take?
Meta’s own developer documentation is direct about the purpose: Meta-ExternalAgent “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” That is the entire stated job. It is not described as powering a specific visible feature a user would recognise — it feeds model training and internal product indexing, on a timeline that, like every other model-training crawler, nobody outside Meta can see.
The documentation also states that Meta-ExternalAgent honours robots.txt, and gives the standard fix: a User-agent: meta-externalagent block with Disallow: /, which Meta says can take up to 24 hours to take effect because the crawler may read a cached copy of the file for that long. That part of the story is simple. The part that isn’t is figuring out which bot is actually asking.
Meta-ExternalAgent is not the bot behind link previews
The crawler most site owners have actually met, even without knowing its name, is facebookexternalhit. Its job, per the same Meta documentation, is to fetch a shared page’s title, description and thumbnail so a link posted into Facebook, Instagram or Messenger shows a preview card. It has nothing to do with AI training. Blocking it does not opt a site out of anything a model will later learn — it just makes links to that site show up bare wherever someone pastes them into a Meta app.
The two bots don’t just do different jobs; they get treated differently by the same documentation. Meta states that facebookexternalhit “might bypass robots.txt” when running a security or integrity check, such as scanning a link for malware before it renders a preview. Meta makes no such claim about Meta-ExternalAgent — the training crawler is the one presented as reliably honouring a Disallow rule, and the preview fetcher is the one that, by Meta’s own account, sometimes won’t.
That inversion is exactly backwards from what the names suggest. A site owner skimming a list of “Meta” bots and disallowing the one that sounds more consequential, or the one a blocklist copy-pasted from somewhere else happened to name, can end up blocking the reliable, rule-following training crawler while leaving the preview fetcher — the one that can ignore the rule anyway — completely untouched. The outcome nobody wanted either way: a broken preview card on a shared link, with the actual AI-training crawler still running.
Not many sites bother blocking it
As of 22 October 2026, Known Agents’ daily scan of the top 1,000 sites’ robots.txt files puts the share disallowing meta-externalagent at 19%. That is a small minority, and it says something about how recently this particular name entered the average webmaster’s vocabulary next to older, more familiar ones like GPTBot. A site that wants to opt out of this specific crawler is still, for now, doing something most of its peers haven’t gotten around to.
What robots.txt doesn’t show
The September incident is the reason a published robots.txt file is worth treating as a claim rather than a fact. The audit engine behind this site reads a site’s robots.txt and reports which of eighteen named AI crawlers it permits, meta-externalagent among them — but that check only tells you what the file says, not what a request actually gets. A separate, smaller set of crawlers gets a live server-level probe as part of a full audit; Meta-ExternalAgent isn’t currently one of them. The one place it does get a real, live request is the free AI crawler checker, which sends a request carrying Meta-ExternalAgent’s actual user agent and reports what comes back — including recognising Cloudflare’s “Your request was blocked” response for what it is, rather than treating it as a normal fetch failure.
That gap between what a file claims and what a server does runs in both directions. A site can have a clean, permissive robots.txt and still 403 Meta-ExternalAgent at the CDN layer, invisible to anyone reading the text file. Or, as the September case showed, the file itself can say something the site owner never wrote, added by a feature living in a different part of the dashboard than the one anyone would think to check for crawler access.
What to actually do about it
Decide separately whether Meta-ExternalAgent and facebookexternalhit should be allowed — they are different bots doing different jobs, and a rule written for one does nothing to the other. If the goal is opting out of training, meta-externalagent is the token that matters, and it’s the one Meta documents as reliably honouring a disallow rule. If the goal is keeping link previews working, leave facebookexternalhit alone and look elsewhere for the fix if previews break — the crawler that generates them isn’t guaranteed to even check the rule.
Then check the file that’s actually live, not the one in the repository. A rule that was never written by anyone on a team can still end up in a robots.txt a crawler receives, the same way eight Disallow lines did on one site in September without a single commit behind them.
Frequently asked questions
What does Meta-ExternalAgent actually do with the pages it reads?
Meta's own developer documentation says it crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. That is the whole stated purpose — nothing about link previews, and nothing about powering a specific feature inside Facebook or Instagram.
Is Meta-ExternalAgent the same crawler that generates link previews on Facebook?
No. That job belongs to a separate agent, facebookexternalhit, whose purpose is to fetch a page's title, description and thumbnail when someone shares a link into one of Meta's apps. Meta's documentation describes the two as distinct crawlers with distinct jobs, and blocking one has no effect on the other.
If I block Meta-ExternalAgent, will my links stop showing a preview card on Facebook or Instagram?
Not from that rule alone. Previews run through facebookexternalhit, not Meta-ExternalAgent. The mistake runs the other way in practice: a site owner trying to stop AI training blocks a bot carrying a name close enough to the wrong one, and breaks previews without touching training at all.
Does Meta-ExternalAgent honour a robots.txt Disallow rule?
Meta's documentation presents it as a crawler that respects robots.txt and documents the standard disallow syntax for blocking it, with changes taking up to 24 hours to apply. It makes no claim, in either direction, about what happens to pages already collected before that rule went up.
Can a robots.txt file end up blocking a crawler without anyone editing it?
Yes. A CDN or bot-management feature can rewrite what a site's robots.txt actually serves, adding rules above or below whatever the site itself published. The file a site owner wrote and the file a crawler receives are not guaranteed to be the same file, and nothing in the dashboard necessarily flags the difference.