How to Write an llms.txt File, and Check It Actually Works

By Muntazir · Published 15 September 2026

A site publishes /llms.txt, writes a paragraph about it in a blog post, and moves on. Months later, a routine check flags the file as missing. That’s confusing, because the URL still returns HTTP 200. What it returns, though, is the homepage — the same page a visitor gets for any address on the domain that doesn’t match a real route. The file was never actually served. It just looked like it was, to anyone who didn’t check the body.

This happens because most sites don’t return a real 404 for an unmatched path. A single-page app’s router, or a CDN’s catch-all rule, sends every unrecognized URL to the same fallback page rather than an error. /llms.txt placed in the wrong directory, or referenced in a docs page before the deploy that was supposed to add it actually shipped, fails exactly the same way as a typo’d URL: 200, and the wrong content. A file that’s “there” in the sense that the address resolves and “not there” in the sense that no such document exists is a strange middle state, and it’s the default outcome of getting one small detail wrong rather than the exception.

What does the file actually need to contain?

The format, set out at llmstxt.org, is deliberately small. One required element: an H1 with the site or project’s name. Everything past that is optional but recommended — a blockquote summary underneath the title, then zero or more H2 sections, each one a list of Markdown links with an optional one-line note after a colon.

# Site Name

> One or two sentences describing what this site is.

## Docs

- [Getting started](https://example.com/docs/start): setup and first steps
- [API reference](https://example.com/docs/api): full endpoint list

## Optional

- [Changelog](https://example.com/changelog): recent releases

That’s the whole shape. No front matter, no XML, no schema to validate against beyond “is this Markdown, and does it have the sections in this order.” The “Optional” H2, by convention, holds the links a model can skip when it’s working with a short context window — a changelog or an older release note, rather than the docs it actually needs. The simplicity is the point, and it’s also why the file is easy to get subtly wrong without any tool complaining: there’s no strict parser rejecting a malformed one the way a broken XML sitemap would get rejected. A missing blank line between the blockquote and the first H2, or a link written as plain text instead of Markdown syntax, still renders as a page. It just isn’t a valid llms.txt any more, and nothing on the file itself says so.

Take a fictional example: Acme Ltd ships an /llms.txt with the title, a summary, and a “Docs” section — but the person who wrote it pasted the links as bare URLs instead of [text](url) pairs, because that’s how they’d written the equivalent section of the site’s own sitemap. The file is still there, still returns 200, still reads as plain text. It just no longer has a link list a parser can extract, because “file list” in the spec means Markdown links specifically, not any line that happens to contain a URL. Nothing rejects the file at write time. It sits there passing every check that only confirms the address resolves.

Where does it live, and what should the response look like?

At the root of the domain: https://example.com/llms.txt, served as plain text or Markdown, with a 200 status and an actual body — not redirected, not gated behind a query string, not nested under a subdirectory. A model or agent fetching it expects exactly what robots.txt and sitemap.xml already taught it to expect from a root-level file: something it can read directly, with no auth and no JavaScript required to render it.

A redirect is a more common mistake than it sounds. A site that moved from www.example.com to example.com, or added a trailing-slash rule for every other path, can end up 301-redirecting /llms.txt along with everything else. The redirect target usually still resolves, so a human clicking the link never notices. A fetcher that doesn’t follow redirects — and not every one does — gets a 301 and stops, having “found” a file it never actually read.

How do you check it actually works?

Not by opening the URL in a browser tab and seeing that something loads. A browser will happily render an HTML error page and make it look like a normal page loaded successfully. The check that matters is on the raw response:

• Fetch the URL directly — curl -i, or any HTTP client that shows you headers and the unrendered body, rather than a browser tab that renders whatever comes back.
• Look at the first few bytes. curl -i https://example.com/llms.txt on a working file starts with HTTP/2 200, a content-type header of text/plain or text/markdown, and a body opening with # and a title. On a soft 404 the status line still says 200, but the body opens with <!doctype html> or <html, and the content type reads text/html.
• Check the content type header specifically, not just the status. text/html on this exact path is the tell — the status code alone can’t distinguish a real file from a fallback page, because both return 200.
• Request a path that definitely doesn’t exist alongside it — something like /this-should-not-exist-12345. If that also comes back 200 with the same body as /llms.txt, the server answers 200 to everything, and the llms.txt check on its own proves nothing: the site would “pass” whether or not the real file was ever published. This is the same logic a well-built site audit uses before trusting any existence check on any well-known path — confirm the server can say no before trusting that it said yes.
• Follow every link inside the file. A file that lists five pages and two of them 404 is worse than not publishing one, because it hands a reader — human or model — two dead ends with confidence.

None of this requires special tooling. It’s the same discipline as checking that robots.txt or sitemap.xml actually resolve, just applied to a newer file most teams haven’t built a habit around yet. A generator gets the required format right by construction, which removes the “wrong shape” failure mode — it doesn’t remove the “never actually deployed” one, which only shows up by fetching the live URL after publishing.

Does having one change what gets cited yet?

Worth stating plainly: publishing a correct llms.txt is not the same claim as publishing one that does something measurable. A separate look at the proposal goes into why adoption is still unconfirmed on the consumer side — no major AI assistant has documented that it fetches an arbitrary domain’s llms.txt mid-conversation. That uncertainty doesn’t change the case for writing one correctly. It does mean the effort belongs in the “cheap, plausible, unproven” bucket, not the “verified ranking lever” bucket — one entry on a longer list of what actually affects AI search visibility, not a shortcut past the rest of it.

What actually breaks, in practice

Four failure modes account for nearly everything. The soft-404 case above, where the path resolves to something else entirely. A file with the right structure but bare URLs instead of Markdown link syntax, like Acme’s, which fails silently because nothing about the response looks wrong. A file that validates against the spec but links to pages that were since moved or deleted, which nothing catches unless something actually follows the links rather than just reading them. And a file that was correct on the day it was written but never touched again as the site added and retired pages, so it slowly stops describing the site it sits on. The format has no expiry warning built in — a stale llms.txt looks exactly as valid as a current one to anything that only checks the shape, and the shape is the only thing most people check.

The four have a common shape of their own: each one passes a glance and fails a fetch. That’s the argument for treating “does it exist” and “is it actually correct” as two separate questions, checked separately, rather than assuming the first answers the second.