Has anyone actually adopted llms.txt?

llms.txt is an emerging, non-IETF convention (proposed by Jeremy Howard, 2024): a markdown file at /llms.txt describing a site for LLMs. We measure how many sites publish one and whether it conforms to the llmstxt.org structure.

Balanced panel: 50,000 selected domains (5,000 from each equal 50k band), 28,783 live. The separate top-50k census improves Explorer coverage but is down-sampled for the representative top-500k estimate.

Do popular sites adopt llms.txt more than smaller ones?

Band 1 = ranks 1–50,000; band 10 = ranks 450,001–500,000. Each point uses the balanced 5,000-domain draw for that band. Adoption is conditioned on a strict-200 live homepage; liveness is shown beside it so the changing denominator is visible.

All ten top-500k bands are represented. D1 also has a separate full census, but only its random 5,000-domain draw enters this chart and the balanced headline. See the sampling and liveness methodology.

How many llms.txt files are really files?

Many origins answer a missing /llms.txt or /ai.txt with 200 anyway — their homepage HTML, their robots.txt, an error string, or a bare OK. We count a file only when the body actually contains a document (see how presence is decided). Raw-200 adoption would be 29.5% in the observed crawl corpus — but that diagnostic pool includes the oversized top-50k census and is not a representative prevalence estimate.

~97% of apparent /ai.txt "adoption" is not a real file. Separating the two is itself a finding.

caveat These are counts, not adoption rates — each variant has a different denominator. This crawl probes tiered and stores one row per domain (the winning variant), so a variant's count is only over the domains where that path was actually probed and won: /llms.txt is probed on every live domain; the supplements (/llms-full.txt, /llms-ctx.txt, /llms-ctx-full.txt) are probed only when /llms.txt returned 200 but was non-conformant; and /ai.txt is probed on every live domain but only recorded when no conformant llms.txt variant was found first. A site that publishes a good /llms.txt and a /llms-full.txt is invisible in the supplement rows. Per-variant denominators need a re-crawl that records every variant independently; until then, do not divide these counts by the probe total.

Who generates llms.txt?

Of 6,007 genuine files, 38.2% carry a detectable generator signature — our own measurement of the CMS/plugin-driven nature of adoption (RQ-4). Bars colored by conformance rate.

Do sites that publish llms.txt also block AI crawlers?

Of genuine llms.txt publishers, 18.2% (1,091 of 6,007) also name at least one AI crawler in robots.txt — publishing a welcome mat for LLMs while addressing their crawlers directly.

caveat Naming ≠ blocking. A named crawler could be an Allow or a Disallow — this v1 measures naming only. The rigorous allow/block verdict split (the Mixed Signals quadrant) lands with per-bot verdicts in a later crawl.

What sections does an llms.txt actually contain?

The most common H2 section titles across the genuine corpus — the de-facto schema sites are converging on (no spec mandates these).

Is llms-full.txt any different from llms.txt?

not measured /llms-full.txt was not independently probed in this crawl, so it has no denominator and no rate. Earlier crawls requested it only as a supplement to a non-conformant /llms.txt; see the variant-denominator caveat above.

Which sites publish the best llms.txt files?