Scrapeless Wiki

What Is llms.txt? – Definition and How It Works

Informational P2 what is llms.txt

Learn what llms.txt proposes, how it differs from robots.txt and sitemaps, what the file contains, and what its practical status is today.

llms.txt is a proposed convention: a Markdown file at a site's root that presents its most useful content in a form a language model can consume directly, without navigating or rendering the site.

1. What It Proposes

A curated, machine-friendly summary of a site.

  • Location: /llms.txt at the domain root.
  • Format: Markdown — a title, a short description, and annotated links grouped by section.
  • Purpose: give a model the clean version of what matters, rather than making it parse rendered pages.

Shape

# Example Corp

> Web data infrastructure for teams building on public information.

## Docs
- [Quickstart](https://example.com/docs/quickstart.md): Install and first request
- [API reference](https://example.com/docs/api.md): Endpoints and parameters

## Guides
- [Rate limits](https://example.com/guides/limits.md): Budgets and headers

2. How It Differs From Existing Files

File Audience Says
robots.txt Crawlers Where you may and may not go
sitemap.xml Search engines Every URL, with metadata
llms.txt Language models Which content matters, and where the clean version is

The distinction is curation. A sitemap is exhaustive; this is a deliberate selection with human-written descriptions.

3. The Companion Convention

The proposal pairs the index with clean Markdown versions of individual pages, conventionally the page URL with a .md suffix.

That addresses a real problem: a rendered page wraps its content in navigation, banners, and scripts, and a model reading it spends most of its context on material that is not the content.

4. Why Sites Consider It

  • Control the framing. You choose which pages represent you and how they are described.
  • Reduce extraction error. Clean Markdown removes the boilerplate that pollutes retrieval.
  • Save context. A smaller, denser document is more likely to be read fully.
  • Serve fetchers that do not render. Content is present without JavaScript.

5. Its Practical Status

It is a community proposal rather than a standard, and adoption is uneven. Support by AI systems is not guaranteed, and publishing the file does not oblige anyone to read it.

The honest framing: it is inexpensive to publish, useful for tools that do consume it, and not a substitute for making the site itself readable to fetchers.

6. If You Publish One

  1. Keep it short — it is an index, not a corpus.
  2. Write real descriptions; they are the part that helps most.
  3. Link the clean Markdown version of each page where you provide one.
  4. Group by section so a reader can select rather than ingest everything.
  5. Keep it current, since a stale index is worse than none.

7. What It Does Not Do

  • It does not control access. That is robots.txt and enforcement.
  • It does not guarantee citation. Retrieval and quotability still decide that.
  • It does not replace structure on the site. Most fetchers read the pages themselves.
  • It is not a ranking signal.

8. Summary

llms.txt is a proposed root-level Markdown index that curates a site's most useful content for language models, usually paired with clean Markdown versions of individual pages.

It is cheap to publish and helpful where supported, but it is a convention rather than a standard — and it complements, rather than replaces, making the site itself readable to fetchers.