Skip to content
English Deutsch

AI Crawler Audit

AI Crawler & Robots.txt Checker

Check what robots.txt allows and what your server actually returns to GPTBot, ClaudeBot, PerplexityBot and 10 more AI crawlers. Free, no signup.

Embed this Tool on Your Website

Add this full interactive diagnostic tool directly into your own website, blog, or documentation.

Free & Iframe Ready
<iframe src="https://aivisibilitycheck.io/tools/robots-txt-checker/?embed=true" width="100%" height="720" frameborder="0" style="border:1px solid #e5e7eb; border-radius:12px; max-width:100%; box-shadow:0 1px 3px rgba(0,0,0,0.05);"></iframe>
<div style="font-size:12px; text-align:right; margin-top:6px; color:#6b7280; font-family:system-ui,sans-serif;">
  Powered by <a href="https://aivisibilitycheck.io/tools/robots-txt-checker/" target="_blank" rel="noopener" style="color:#0284c7; text-decoration:none; font-weight:500;">AI Visibility Check</a>
</div>

technical guide & best practices

Why robots.txt Is Only Half of AI Crawler Access

robots.txt is a request, not a gate. It tells crawlers what they may fetch, but the server, CDN or firewall in front of the site decides what they actually receive. A site can allow GPTBot in robots.txt while its CDN answers GPTBot with a 403, and the owner sees a clean robots.txt and never finds out. This checker reads robots.txt with the same matching rules crawlers use (RFC 9309), then requests the page as each AI crawler and reports both answers side by side.

Key Optimization Standards

✓Name every AI crawler you want with an explicit Allow rule, so a later blanket block conflicts with it visibly instead of silently changing what no rule meant.
✓Separate search crawlers from training crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether AI search can cite you; GPTBot, ClaudeBot and CCBot only collect training data.
✓On Cloudflare, check AI Crawl Control (the Block Crawler toggles and the AI bot policies), Security › Bots and any WAF rules. Each can block crawlers that robots.txt allows.
✓Re-check after any CDN, firewall or bot-protection change. Those settings are where blocks come from, and they never touch robots.txt.

frequently asked questions

Frequently Asked Questions

My robots.txt allows GPTBot. Why does this tool say the server refused it?

Because something in front of your site answered GPTBot with an error before your robots.txt could matter. On Cloudflare the usual cause is AI Crawl Control's Block Crawler toggle or a Block AI bots setting; on other hosts, a WAF or bot-protection rule. The fix is in that dashboard, not in robots.txt.

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. GPTBot only collects training data. Whether ChatGPT search can cite a page depends on OAI-SearchBot, and live fetches during a conversation use ChatGPT-User. Each is controlled separately.

How accurate is testing with a crawler's user agent?

It shows how your site treats anything that identifies itself as that crawler, which is how most CDN blocks work. Some providers also verify the big crawlers by IP address, so the real bot can occasionally be treated differently. Your CDN's own logs are the final word.

Why are Google-Extended and Applebot-Extended only checked in robots.txt?

They are not separate crawlers. Google and Apple fetch pages with their normal bots and read these tokens in robots.txt to decide whether the content may be used for AI training.