robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-25). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
174.9%
272.2%
371.2%
472.7%
572.7%
672.0%
771.8%
871.7%
970.9%
1069.5%
absent (404)
data table
17.0%
28.5%
38.9%
47.7%
57.7%
67.8%
78.0%
810.1%
910.4%
108.2%
disallow-all
data table
12.5%
22.1%
32.1%
42.0%
52.0%
63.3%
74.6%
82.7%
92.4%
103.0%
allow-all
data table
120.4%
223.3%
325.7%
424.7%
525.3%
626.3%
726.8%
828.3%
930.4%
1028.9%
wildcard * UA
data table
195.4%
294.7%
394.2%
494.7%
594.6%
694.4%
794.3%
892.7%
993.2%
1094.6%
declares a sitemap
data table
165.9%
263.7%
362.5%
463.9%
563.9%
664.1%
764.5%
861.8%
962.5%
1063.0%
llms.txt adoption
data table
111.3%
212.9%
311.5%
49.9%
59.9%

Full data table

Rank decile sample status 200status 404disallow allallow allwildcard uasitemapllms adoption
1 58,555 74.9% 7.0% 2.5% 20.4% 95.4% 65.9% 11.3%
2 55,689 72.2% 8.5% 2.1% 23.3% 94.7% 63.7% 12.9%
3 50,345 71.2% 8.9% 2.1% 25.7% 94.2% 62.5% 11.5%
4 49,770 72.7% 7.7% 2.0% 24.7% 94.7% 63.9% 9.9%
5 51,381 72.7% 7.7% 2.0% 25.3% 94.6% 63.9% 9.9%
6 51,007 72.0% 7.8% 3.3% 26.3% 94.4% 64.1% not probed
7 50,785 71.8% 8.0% 4.6% 26.8% 94.3% 64.5% not probed
8 44,360 71.7% 10.1% 2.7% 28.3% 92.7% 61.8% not probed
9 36,234 70.9% 10.4% 2.4% 30.4% 93.2% 62.5% not probed
10 42,899 69.5% 8.2% 3.0% 28.9% 94.6% 63.0% not probed

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.