Onbiom

Website guide

Do you need llms.txt? What the evidence shows

llms.txt is a small text file some site owners publish for AI tools. It is cheap to make, but the public evidence that the big AI services read it is thin. This guide says what is documented and what is not.

Sources read on 30 October 2026; these pages change, so check them before you rely on this guide. Where a vendor page was silent, we say so instead of guessing.

What is llms.txt?

It is a proposal by Jeremy Howard, published on llmstxt.org (version 2, modified 10 August 2026). The proposal describes an /llms.txt file that gives agents a short overview of a site and links to the pages worth reading, "to help agents use a website". Only the H1 title is required; a short summary, link lists and an optional section for secondary links come after it. The file can sit at the site root or at another path to cover pages beneath it.

It is a proposal from one author, not a standard backed by a standards body, and it is not robots.txt: it describes content, it does not grant or deny access.

Who says they read it?

  • Google says it does not use it for Search. Its AI optimization guide says that creating such files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them", and that it is fine to keep one for other services.
  • OpenAI, Anthropic and Perplexity: we found no statement that their crawlers or assistants read it. The OpenAI crawler documentation, the Anthropic crawler article and the Perplexity bots guide describe their bots and robots.txt controls. The Anthropic and Perplexity bot descriptions do not mention llms.txt at all, and the OpenAI page mentions it only to point readers to its own documentation index.

Publishing a file on your own developer docs is not the same as reading other sites' files. The llmstxt.org page notes that several large AI companies publish llms.txt for their developer documentation, but it does not say their crawlers consume anyone else's.

What about adoption and usage numbers?

None are quoted here. We did not open a primary source that measures how many sites publish the file or how often an AI service fetches it, so we do not repeat a second-hand figure.

Should you add one?

Optional, low cost. If you maintain documentation or a product with a few key pages, a short file is an hour of work and may help tools that choose to read it. Do not expect citations, rankings or traffic from it: no vendor page we read promises any. Fix customer-facing basics first: clear page text, correct contact details and a robots.txt that matches your intentions.

A minimal example

Save this as llms.txt at the root of your site. The names and links are placeholders; use only facts that are true for your business.

# Example Heating

> Boiler servicing and heat pump repair in Delft and Rijswijk.

## Services

- [Heat pump repair](https://example.com/heat-pump-repair): what we repair and how to request a visit
- [Boiler servicing](https://example.com/boiler-servicing): annual service, what is included

## Contact

- [Request a visit](https://example.com/contact): form and telephone number

## Optional

- [About us](https://example.com/about)

What does the Onbiom free check report?

The free website scan requests /llms.txt as a fixed public file and shows a finding called llms.txt (optional). It is present when the server answers HTTP 200 with a non-empty text/plain body, and not_observed otherwise. The wording says a missing file is not an indexing requirement and a present one does not guarantee AI citations or visibility. A normal site can pass the technical assessment without it.

Limits: the check only fetches the one path at the site root (not other paths the proposal allows), accepts at most 256 KB, and does not validate the format against the proposal. A file served as another content type, such as text/markdown, is reported as not observed. Nothing here shows whether any AI service reads your file. See the how Onbiom checks your site.

Check yours

Run the free website scan on your domain and read the llms.txt (optional) finding next to the other results. Treat it as the lowest-priority item on the list.

Sources and further reading

Last reviewed:

Run the free website scan Current plans