Amazonbot: What Amazon's Crawler Takes, and Whether to Let It In

By Muntazir · Published 19 October 2026

On 3 October 2026, a site that had allowed Amazonbot in its robots.txt for months stopped getting through to it. Nothing in the text file had changed. Cloudflare’s AI Crawl Control had switched on a “Block Crawler” toggle for it, alongside GPTBot and ClaudeBot, and the edge started answering every one of those requests with a 403 — “Your request was blocked.” Over the previous seven days, GPTBot had been allowed zero times out of fifteen attempts. ClaudeBot: zero out of fourteen. The robots.txt a visitor could read in a browser said none of this was happening.

That gap between the file and the edge is the whole problem with deciding whether to let Amazonbot in. The decision itself is simple enough once you know what the bot actually does — which is narrower, and stranger, than most “block the AI crawlers” advice assumes.

What does Amazonbot actually take?

Amazon’s own developer documentation is specific, and shorter than the question deserves: Amazonbot “is used to improve our products and services,” and what it collects “may be used to train Amazon AI models.” That’s the whole stated purpose. The page does not mention Alexa. It does not mention Rufus, Amazon’s shopping assistant, at all — not as something Amazonbot feeds, and not under any other name either, across every page Amazon publishes about its crawlers. Third-party crawler directories routinely describe Amazonbot as powering both Alexa and Rufus; Amazon’s own documentation makes neither claim.

Two things about how it behaves are worth knowing before touching robots.txt. First, it honours the Robots Exclusion Protocol and respects per-crawler Allow/Disallow rules, but it does not support the crawl-delay directive — a site hoping to slow it down rather than block it outright has no lever for that. Second, it reads a fresh copy of robots.txt or falls back to one cached for up to thirty days, and Amazon states that a rule change takes roughly twenty-four hours to take effect — so a Disallow added today is not instant.

There’s a second, more granular lever most site owners never use. Amazon’s documentation describes a page-level noarchive meta tag that specifically tells Amazonbot not to use that page for model training — distinct from noindex, which removes the page from Amazon’s results altogether. That’s a narrower tool than a robots.txt rule: a page can stay eligible to appear in Amazon’s products while opting out of training, something a blanket Disallow cannot do on its own.

Is Amazonbot the bot that feeds Alexa?

Not quite, and this is where the “block it to stay out of Alexa” assumption runs into trouble. Amazon names two further agents, each with its own documentation and its own job. Amzn-SearchBot is what makes a page eligible to appear in Amazon’s search experiences, Alexa included — allow it, and a page can surface there; block it, and it can’t. Amzn-User is the live fetch: the one that runs the moment someone asks Alexa a question that needs current information from the web, rather than something already indexed.

Both of those pages state, in close to identical wording, that the bot in question “does not crawl content for generative AI model training.” Only Amazonbot’s own page carries that admission. So the picture splits three ways rather than one: a page can be fully visible to Alexa through Amzn-SearchBot and Amzn-User while still opting out of whatever Amazonbot itself is collecting, and blocking Amazonbot alone does nothing to a business’s presence in Alexa’s answers, because that traffic was never running through it.

That split matters for what a crawler check can actually tell a site owner. The audit engine behind this site tracks eighteen named AI crawlers by their robots.txt entries, and the free AI crawler checker sends Amazonbot a live request to see whether the server actually honours what the file claims — the same treatment the GPTBot and ClaudeBot write-ups describe for those crawlers. Amzn-SearchBot and Amzn-User are not on that list yet. A site that wants to know specifically whether Alexa’s live-answer path can reach it has no automated way to check that today, from this tool or most others — only Amazon’s own, slower, unverifiable channel.

What robots.txt can’t show you

The October incident above is the reason that distinction matters in practice, not just on paper. Cloudflare’s AI Crawl Control sits in a different part of the dashboard from robots.txt entirely, and its per-crawler “Block Crawler” toggles can 403 a request at the edge regardless of what the text file says. A site owner who checks only the file — the thing every “how to allow AI crawlers” guide tells you to edit — has no way of seeing that the server is quietly refusing the crawler anyway. The zero-out-of-fifteen and zero-out-of-fourteen counts above came from Cloudflare’s own per-crawler logs, not from the file, and the file gave no hint anything was wrong. It took checking what actually arrived at the server, not what the robots.txt promised, to find it.

Amazonbot is exactly as exposed to this as GPTBot and ClaudeBot were. Allowing it in robots.txt is necessary but not sufficient; the only way to know whether that permission is actually honoured at the edge is to test the live response, separately, the way the incident above eventually was.

Should a site block it?

The honest answer is that it depends on what a business is actually trying to avoid, and the three-way split above means “block Amazonbot” and “stay out of Alexa” are not the same decision. Blocking Amazonbot in robots.txt — or using noarchive instead, where the goal is narrower — stops one named crawler from collecting pages that, by Amazon’s own account, may end up training its models. It does nothing to a business’s visibility inside Alexa’s actual answers, because that runs through two different agents that explicitly disclaim the training use case and that this product’s own crawler check does not yet probe on the server side. A site owner chasing the second concern by editing a rule aimed at the first is solving the wrong problem and will have no way to tell, from the file alone, that the real one is untouched.