llms.txt explained: what it is, what it is not, and whether to bother

llms.txt is a proposed convention for a plain-text file at the root of your website that tells large language models which of your pages matter and what each one contains. It was proposed by Jeremy Howard in 2024 and documented at llmstxt.org.

The problem it addresses

When an AI assistant tries to answer a question about your business, it has to decide which of your pages to read. Your HTML is mostly not content: navigation, cookie banners, scripts, footers, tracking. Context windows are finite and reading is not free. Faced with a 400-page site, a model has no reliable signal about which twenty pages actually explain what you do.

llms.txt is an editorial answer to that: a short, curated, human-written map.

How it differs from files you already have

The distinction that matters: a sitemap is exhaustive and unranked; llms.txt is selective and prioritised. A sitemap listing 4,000 product URLs helps an indexer and overwhelms a model.

The format

Deliberately minimal — it is Markdown, and only the H1 is required.

# Acme Bakery

> Independent bakery in Leeds supplying sourdough and
> pastries to cafés across Yorkshire since 2009.

## Main
- [Home](https://acme.example/): What we bake and who we supply
- [Wholesale](https://acme.example/wholesale): Trade ordering, minimums and lead times
- [Our bakers](https://acme.example/team): The people behind the bread

## Guides
- [Storing sourdough](https://acme.example/guides/storage): How to keep a loaf for five days

## Optional
- [Privacy policy](https://acme.example/privacy)

The structure is: an H1 with the site name, an optional blockquote summary, optional prose, then H2 sections containing lists of links in the form [title](url): description.

The Optional section is meaningful

It is not a dumping ground — it has a defined meaning in the spec: those URLs may be skipped when a shorter context is needed. Put legal pages, tag archives and utility pages there.

The honest state of adoption

This deserves a straight answer, because there is a lot of overclaiming about it.

• No major search engine has committed to using llms.txt as a ranking or retrieval signal.
• Google has publicly indicated it is not using it.
• Several AI-facing developer tools and documentation platforms do read it, and a growing number of documentation sites publish one.
• Nobody can currently show you a controlled study demonstrating traffic gains from adding one.

So: it is a convention with real but narrow adoption, not an established standard.

Why publish one anyway

Three reasons that do not depend on adoption predictions:

1. The cost is nearly zero. One text file, no ongoing maintenance beyond occasional updates.
2. The downside is nil. It cannot harm your SEO; it is an inert file that nothing is obliged to read.
3. Writing it is a useful exercise. Choosing the twenty pages that explain your business, and describing each in one sentence, surfaces gaps in your content that nothing else does. Plenty of people discover mid-way through that they have no page explaining what they actually sell.

Writing a good one

• Descriptions are the whole point. - [Pricing](/pricing) tells a model nothing it could not guess. - [Pricing](/pricing): Per-seat monthly pricing, free tier limits and the enterprise contact route is genuinely useful.
• Be selective. Twenty well-described pages beat two hundred bare links.
• Group by meaning, not by URL structure where they differ.
• Write for a reader, not a parser. The format is human-readable on purpose.
• Keep it current. A file describing pages that 404 is worse than no file.

Where it goes

The root of your domain, served as text/plain: https://yoursite.com/llms.txt. Same place as robots.txt.

The thing llms.txt does not do

It helps a model understand your site once it visits. It does not tell anyone your site changed. Discovery is a separate problem, and one with an established answer — sitemaps for everyone, and IndexNow for the engines that support it.