robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-07-17 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 746,000 robots.txt fetches · 574,664 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-17). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
top_1k68.6%
top_10k72.3%
top_100k69.3%
top_1m68.1%
absent (404)
data table
top_1k12.4%
top_10k10.7%
top_100k12.6%
top_1m11.9%
disallow-all
data table
top_1k19.0%
top_10k8.4%
top_100k5.9%
top_1m4.3%
allow-all
data table
top_1k13.5%
top_10k16.7%
top_100k22.0%
top_1m26.0%
wildcard * UA
data table
top_1k94.5%
top_10k95.2%
top_100k94.5%
top_1m93.8%
declares a sitemap
data table
top_1k54.5%
top_10k58.5%
top_100k58.6%
top_1m59.7%
llms.txt adoption
data table
top_1k15.5%
top_10k11.7%
top_100k11.1%
top_1m10.9%

Full data table

Rank tier sample status 200status 404disallow allallow allwildcard uasitemapllms adoption
top_1k 839 68.6% 12.4% 19.0% 13.5% 94.5% 54.5% 15.5%
top_10k 7,252 72.3% 10.7% 8.4% 16.7% 95.2% 58.5% 11.7%
top_100k 63,980 69.3% 12.6% 5.9% 22.0% 94.5% 58.6% 11.1%
top_1m 502,593 68.1% 11.9% 4.3% 26.0% 93.8% 59.7% 10.9%

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.