llms.txt adoption & conformance

llms.txt is an emerging, non-IETF convention (proposed by Jeremy Howard, 2024): a markdown file at /llms.txt describing a site for LLMs. We measure how many sites publish one and whether it conforms to the llmstxt.org structure.

Adoption by rank decile

Decile 1 = the most popular 100k. This tests RQ-4: is adoption driven by big brands, or spread across the long tail via CMS defaults?

Adoption is roughly flat across the top 500k (deciles 1–5 all ~10–13%; genuine, non-HTML files) — it does not concentrate in the head, which is consistent with passive CMS-plugin defaults rather than brand-led strategy (RQ-4).

This crawl's llms.txt cohort covers the Tranco top 500k (rank deciles 1–5); deciles 6–10 are not probed this crawl and join in a later crawl. The cohort is rank-stratified & liveness-gated (see methodology).

Genuine files vs HTML soft-404s

Many origins serve their homepage HTML for a missing /llms.txt or /ai.txt and still return 200. We count a file as genuine only when it's a 200 that is not HTML. Raw-200 adoption would be 29.5% — but that's mostly fakes.

~97% of apparent /ai.txt "adoption" is HTML, not a real file. Separating the two is itself a finding.

caveat These are counts, not adoption rates — each variant has a different denominator. This crawl probes tiered and stores one row per domain (the winning variant), so a variant's count is only over the domains where that path was actually probed and won: /llms.txt is probed on every live domain; the supplements (/llms-full.txt, /llms-ctx.txt, /llms-ctx-full.txt) are probed only when /llms.txt returned 200 but was non-conformant; and /ai.txt is probed on every live domain but only recorded when no conformant llms.txt variant was found first. A site that publishes a good /llms.txt and a /llms-full.txt is invisible in the supplement rows. Per-variant denominators need a re-crawl that records every variant independently; until then, do not divide these counts by the probe total.

Who generates llms.txt?

Of 6,007 genuine files, 38.2% carry a detectable generator signature — our own measurement of the CMS/plugin-driven nature of adoption (RQ-4). Bars colored by conformance rate.

Mixed signals

Of genuine llms.txt publishers, 18.2% (1,091 of 6,007) also name at least one AI crawler in robots.txt — publishing a welcome mat for LLMs while addressing their crawlers directly.

caveat Naming ≠ blocking. A named crawler could be an Allow or a Disallow — this v1 measures naming only. The rigorous allow/block verdict split (the Mixed Signals quadrant) lands with per-bot verdicts in a later crawl.

Emergent section vocabulary

The most common H2 section titles across the genuine corpus — the de-facto schema sites are converging on (no spec mandates these).

llms-full.txt: observed, but not measured as a rate

We observed 20 genuine /llms-full.txt files (after removing HTML fakes). We deliberately do not express that as an adoption percentage, because this crawl's probe order means the denominator is unknown — see the caveat above. no denominator

llms.txt Hall of Fame