Has anyone actually adopted llms.txt?

llms.txt is an emerging, non-IETF convention (proposed by Jeremy Howard, 2024): a markdown file at /llms.txt describing a site for LLMs. We measure how many sites publish one and whether it conforms to the llmstxt.org structure.

Balanced panel: 50,000 selected domains (5,000 from each equal 50k band), 30,386 live. The separate top-50k census improves Explorer coverage but is down-sampled for the representative top-500k estimate.

Do popular sites adopt llms.txt more than smaller ones?

Band 1 = ranks 1–50,000; band 10 = ranks 450,001–500,000. Each point uses the balanced 5,000-domain draw for that band. Adoption is conditioned on a strict-200 live homepage; liveness is shown beside it so the changing denominator is visible.

All ten top-500k bands are represented. D1 also has a separate full census, but only its random 5,000-domain draw enters this chart and the balanced headline. See the sampling and liveness methodology.

How many llms.txt files are really files?

Many origins answer a missing /llms.txt or /ai.txt with 200 anyway — their homepage HTML, their robots.txt, an error string, or a bare OK. We count a file only when the body actually contains a document (see how presence is decided). Raw-200 adoption would be 12.3% in the observed crawl corpus — but that diagnostic pool includes the oversized top-50k census and is not a representative prevalence estimate.

~97% of apparent /ai.txt "adoption" is not a real file. Separating the two is itself a finding.

caveat These are counts, not adoption rates — each variant has a different denominator. This crawl probes tiered and stores one row per domain (the winning variant), so a variant's count is only over the domains where that path was actually probed and won: /llms.txt is probed on every live domain; the supplements (/llms-full.txt, /llms-ctx.txt, /llms-ctx-full.txt) are probed only when /llms.txt returned 200 but was non-conformant; and /ai.txt is probed on every live domain but only recorded when no conformant llms.txt variant was found first. A site that publishes a good /llms.txt and a /llms-full.txt is invisible in the supplement rows. Per-variant denominators need a re-crawl that records every variant independently; until then, do not divide these counts by the probe total.

Who generates llms.txt?

Of 6,247 genuine files, 44.3% carry a detectable generator signature — our own measurement of the CMS/plugin-driven nature of adoption (RQ-4). Bars colored by conformance rate.

Do sites that publish llms.txt also block AI crawlers?

Of genuine llms.txt publishers, 22.2% (1,388 of 6,247) also name at least one AI crawler in robots.txt — publishing a welcome mat for LLMs while addressing their crawlers directly.

caveat Naming ≠ blocking. A named crawler could be an Allow or a Disallow — this v1 measures naming only. The rigorous allow/block verdict split (the Mixed Signals quadrant) lands with per-bot verdicts in a later crawl.

What sections does an llms.txt actually contain?

The most common H2 section titles across the genuine corpus — the de-facto schema sites are converging on (no spec mandates these).

Is llms-full.txt any different from llms.txt?

This crawl promotes /llms-full.txt from a supplement (probed only when /llms.txt was a non-conformant 200) to an independently probed path — requested on every live cohort domain regardless of the /llms.txt outcome (FR-18d). It is therefore the one non-base variant with a real denominator, and the only one we express as a rate.

read these together The not-a-file rate is the story, not a footnote. This path returns 200 for 12,210 domains but only 2,355 are real files — 80.7% return something else for a missing path (a web page, a robots.txt, a stub). Compare /llms.txt, where the rate is far lower. Quoting the raw 200 count as adoption would overstate this path by roughly 5.2x.

Is the full dump maintained alongside the index?

This is the question independent probing exists to answer: both lanes now cover the same domains, so the breakdown is measurable for the first time.

Of 6,247 /llms.txt publishers, only 29.5% also serve a full-context dump — so the dump is the exception, not the norm, even among sites that have adopted the convention.

An interesting insight: one platform dominates this path

insight Shopify accounts for 1,350 of the 2,355 genuine /llms-full.txt files (57.3%). That is a far higher single-vendor concentration than the base /llms.txt corpus shows, where no generator exceeds ~17%. Read carefully: it means this path's adoption curve is substantially one platform's default output rather than 2,355 independent decisions — the same passive-CMS-default dynamic the HTTP Almanac found for llms.txt (39.6% All in One SEO), but more concentrated.

One thing this does not say: Shopify is not the explanation for the 512 full-only domains. Its files overwhelmingly sit on domains that publish a base /llms.txt as well — the platform emits both files, not the dump alone, which is why it shows up in the both row above rather than the full-only row. The full-only group is instead dominated by files with no detectable generator, which we can count but not attribute.

An interesting insight: for most publishers, the “full” file isn’t fuller

The 1,845 domains publishing both files let us ask something the tiered crawl never could: is /llms-full.txt actually a fuller document, or the same content at a second path? Compared per domain on size and link count.

insight 70.2% of both-lane publishers serve a /llms-full.txt that mirrors their /llms.txt rather than expanding on it. We verified this against live files rather than inferring it from sizes: those pairs differ by exactly the filename, in a line that says so — “you’re reading /llms.txt, which mirrors that content” — and 33 serve the two paths byte-for-byte identical. So the file documents its own duplication.

Read the aggregate with its concentration. “70.2% are mirrors” is really “one platform’s output is a mirror, and that platform dominates this lane”: Shopify accounts for 1,341 of the 1,845 pairs and 94.6% of its pairs are mirrors. Outside it the pattern largely disappears. The table below is the honest form of the finding; the single lane-wide percentage is not.

What this does not establish: we detect the generator from a signature the file itself carries, and we checked a handful of these against their live files, not all 1,341. The (none detected) row is counted but not attributable — those files carry no generator signature, so we can report their mirror rate and nothing about who produced them.

Which sites publish the best llms.txt files?