llms.txt Explained: What It Is and How to Write One

By Sameer Bhat · Published 31 August 2026

llms.txt is a plain-text file at your domain root that lists the pages an AI assistant should read to understand your site, each with a one-line description. It is a proposal, not a standard — no engine is required to fetch it, and none has confirmed that it does. This explains what to put in one, what it can’t do, and how to write one properly.

What is llms.txt?

The file lives at /llms.txt. It’s Markdown: an H1 with your site name, a blockquote summary, then H2 sections grouping links by topic — product pages, docs, blog posts. Each link gets a short description after it, so a model can decide whether to follow it without fetching the target first.

The idea, proposed by Jeremy Howard’s team at Answer.AI on 3 September 2024, borrows from robots.txt and sitemap.xml: a predictable location, a simple format, one job. Where it differs is audience. robots.txt talks to crawlers about permissions. sitemap.xml talks to search engines about what exists. llms.txt talks to a language model mid-conversation, trying to answer a question, about which of your pages are worth reading right now.

Do any AI engines actually read it?

Be skeptical here. As of this writing, no major AI provider — not OpenAI, not Anthropic, not Google, not Perplexity — has published documentation confirming their consumer assistant fetches /llms.txt on an arbitrary domain and uses it to guide an answer. Agent frameworks, MCP integrations and RAG pipelines built on top of these models do read it in practice, because those workflows already fetch project files directly and llms.txt gives them a shortcut. General-purpose consumer AI search is a different question, and the honest answer is: unproven, and the traffic numbers back that up — Otterly.AI’s 90-day study of real AI bot requests logged over 62,100 visits and only 84 of them fetched /llms.txt — about 0.1%, roughly a third the rate an average content page received.

That doesn’t make the file worthless. It costs nothing to add — a static text file — and if adoption grows, you already have one. Treat it the way you’d treat a new schema.org type in its first year: cheap to try, not something to bet a content strategy on.

What goes in one?

Keep it short and link to pages that are actually worth reading, not everything you own. Here is the file this site serves at /llms.txt:

# AI Visibility Check by CyberSpark

> Free and paid audits that check whether ChatGPT, Google, and other
> AI/search engines can find, understand, and recommend a business —
> and score the result out of 100. Missing data is always shown as
> unmeasured, never disguised as a pass or a zero.

## Product

- Free AI & search visibility audit: start here — no email required.
- Audit methodology: what the audit checks, how scoring works.

## Blog

- (one line per post, title + description)

Two things worth copying. First, the summary blockquote does real work — it’s the one sentence a model reads if it reads nothing else, so it should say what the site is and its one distinguishing fact, not marketing copy. Second, we generate this file from the live blog collection at build time instead of hand-maintaining a static list. A hand-written /llms.txt drifts the first time someone forgets to add a new post to it; a generated one can’t.

There’s a second reason short, self-contained descriptions matter here, and it’s the same reason they matter throughout the rest of your site. Our own audit engine scores pages on exactly this: whether paragraphs are self-contained enough to quote. The check looks for passages between 15 and 80 words that end in a full stop and don’t depend on surrounding context to make sense — the same shape a good llms.txt link description needs, just applied to your body content instead of a file. A page full of quotable, complete statements is easier for a model to lift a fact from, with or without an llms.txt pointing at it.

llms.txt vs llms-full.txt

llms.txt is an index: links and descriptions, nothing more. llms-full.txt is the same structure with each linked page’s content pasted in full, so a reader gets everything in one fetch instead of following ten links.

llms-full.txt makes sense for small, mostly-static sites — documentation for a single library, a product with a handful of pages. It stops making sense once your content is large enough that the full file would run to hundreds of thousands of words; at that point you’re asking a model to download your entire site to answer one question, which no retrieval system is going to do. For a blog that grows every week, an index that a model can selectively follow is the more realistic file.

How do I write one?

1. Start with the summary. One blockquote line: what the site does, in a sentence a stranger could repeat accurately.
2. List only the pages worth reading. Your homepage, your core product or methodology pages, your best content. Not your privacy policy, not your login page.
3. Write a real description for each link, not the page’s own title repeated. The description is the only context a model gets before deciding whether to fetch the page.
4. Group by topic with H2s once you have more than a handful of links — Product, Docs, Blog — so the structure itself carries information.
5. Generate it from your actual content where you can, the way our /llms.txt reads from the blog collection at build time. A file that goes stale the first time someone ships a page and forgets to update it is worse than no file.
6. Serve it as text/plain at the domain root. No auth, no redirect chain.

Does it replace robots.txt?

No, and conflating the two causes real problems. robots.txt is the permission layer: it tells a crawler which paths it may fetch at all, and it’s checked before any request goes out. llms.txt is a discovery layer: it assumes fetching is already allowed and tries to point a reader toward the pages worth their attention.

They can contradict each other, and when they do, robots.txt wins, because it’s the one crawlers actually enforce. Listing a page in llms.txt while robots.txt disallows the path a few directories up blocks it just as effectively as if the page didn’t exist. If you’re setting up llms.txt, check your robots.txt rules for AI crawlers at the same time — they’re supposed to work together, and a mismatch between them is easy to ship without noticing.

llms.txt is a low-cost bet with no confirmed payoff yet. Getting the fundamentals right — crawlable pages, quotable content, an accurate robots.txt — matters regardless of whether any engine ever reads your llms.txt. Run a free audit to check where your site actually stands, or read our audit methodology for how each check works. You can also try our llms.txt generator and robots.txt checker directly.