AI Crawler Audit

AI Crawler & Robots.txt Checker

Audit permissions for AI search crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and inspect /llms.txt.

Embed this Tool on Your Website

Add this full interactive diagnostic tool directly into your own website, blog, or documentation.

Free & Iframe Ready

<iframe src="https://aivisibilitycheck.io/tools/robots-txt-checker/?embed=true" width="100%" height="720" frameborder="0" style="border:1px solid #e5e7eb; border-radius:12px; max-width:100%; box-shadow:0 1px 3px rgba(0,0,0,0.05);"></iframe>
<div style="font-size:12px; text-align:right; margin-top:6px; color:#6b7280; font-family:system-ui,sans-serif;">
  Powered by <a href="https://aivisibilitycheck.io/tools/robots-txt-checker/" target="_blank" rel="noopener" style="color:#0284c7; text-decoration:none; font-weight:500;">AI Visibility Check</a>
</div>

technical guide & best practices

How to Build High-Performance XML Sitemaps for Google & AI Search

XML sitemaps are the primary protocol search engines and LLM web crawlers use to discover canonical pages, detect update timestamps (lastmod), and navigate international multi-language language alternates (hreflang).

Key Optimization Standards

✓Include only canonical, 200 OK URLs (never include 301 redirects, 404s, or noindex pages).

✓Use valid W3C ISO 8601 timestamps for <lastmod> (e.g. 2026-08-25T12:00:00Z).

✓Declare xmlns:xhtml namespace for multi-language sites and include reciprocal <xhtml:link rel='alternate' hreflang='...' /> tags.

✓Always specify an x-default hreflang fallback for international users.

frequently asked questions

Frequently Asked Questions

What is the maximum size of an XML sitemap?

A single XML sitemap can contain up to 50,000 URLs and must not exceed 50MB uncompressed. If you have more URLs, use a Sitemap Index file.

How do xhtml:link hreflang tags work in sitemaps?

Instead of cluttering your HTML <head>, you can declare language alternates directly inside each <url> block using <xhtml:link rel='alternate' hreflang='en-GB' href='...' />.

Why should localhost and staging URLs be excluded?

Leaking internal development URLs (e.g. http://localhost:3000) causes crawl budget waste and can expose private staging endpoints to search indexers.