llms.txt: what it is, and whether your site needs one

llms.txt is a proposed standard: a single Markdown file at the root of your domain that tells a language model what your site covers and where the important pages are. Ours is at remotepuzzle.com/llms.txt, and it took about twenty minutes to generate.

The proposal comes from Jeremy Howard and is documented at llmstxt.org. It is worth being precise about what it does, because it is routinely confused with two files that do something else entirely.

How llms.txt differs from robots.txt and sitemap.xml

These three files are often described as a set. They are not. Each answers a different question, and only one of them is a standard that search engines are obliged to respect.

That last row is the one to hold on to. robots.txt is a formal standard that the major crawlers honour. llms.txt is a convention that some tools read and many ignore. Adding one does not make you visible to AI search; it makes you slightly easier to summarise correctly if something is already reading you.

What actually goes in the file

The format is deliberately plain: an H1 with your name, a blockquote summarising what you do, then linked sections grouping your key pages with one line of context each. A model reading it should be able to answer "what is this company, and where would I look for X" without crawling the whole site.

The single biggest mistake is dumping every URL you have into it. That is what sitemap.xml is for. llms.txt is an editorial summary — it should be short enough that a person would read it.

• Lead with a one-sentence description of what you do, in the blockquote. Many tools read only this.
• Group links under headings that match how a buyer thinks: services, pricing, case studies, writing.
• Give every link a short description. The link on its own tells a model nothing.
• Generate it from the same data your site renders from, so it cannot drift out of date.
• Keep it under a few hundred lines. If it needs pagination, it has become a sitemap.

Does anything actually read it?

Honestly: less than the enthusiasm suggests. As of August 2026 no major AI search engine has publicly committed to using llms.txt as a ranking or retrieval input. Some developer tools and documentation crawlers read it, and adoption is growing, but anyone telling you it is a required part of AI search is selling something.

The case for adding one anyway is that it is close to free. It is one build-time script, it costs nothing to serve, and if adoption continues you are already there. That is a reasonable bet at twenty minutes of work. It is not a reasonable bet at twenty hours.

What matters far more

If your goal is being cited in AI answers, llms.txt is somewhere near the bottom of the list. Three things matter more, in this order.

1. Being crawlable at all. Check your own site: if your firewall returns 403 to GPTBot or ClaudeBot, nothing else you do matters. We wrote up how to check in should you block GPTBot.
2. Server-rendering your content. Many AI crawlers execute little or no JavaScript. Fetch your own page with curl and read what comes back — if the text is not in that response, assume it does not exist.
3. Writing in self-contained passages. A model quotes paragraphs, not pages. We cover the mechanics in generative engine optimisation.

llms.txt helps a model that has already found you describe you accurately. It does nothing for a model that cannot reach you.

How to add one

Write a script that reads your existing route and content data and emits the Markdown at build time. Ours runs as part of the build and regenerates sitemap.xml, robots.txt and llms.txt together from one source, which is the only way to keep three files describing the same site in agreement.

Then verify it the same way a crawler would: fetch the raw URL and read what comes back. If you would like a second pair of eyes on how your site looks to AI crawlers, that is exactly what our AI-search SEO work covers.

Frequently asked questions

References

1. The /llms.txt proposal — llmstxt.org
2. Introduction to robots.txt — Google Search Central
3. RFC 9309: Robots Exclusion Protocol — IETF