Data as of
2026-07-25 ·
robots.txt via Common Crawl
CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
·
crawls:
2026-07-25 ·
2026-07-17
687,794 robots.txt fetches ·
491,025 parsed ·
54,000 llms.txt probes
Segments
Headline metrics broken down by segment. Toggle the segmentation axis and
the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a
restriction on a top site counts far more than one deep in the tail.
One crawl ingested (2026-07-25). Cross-crawl trend lines appear
automatically when the next monthly crawl lands.
robots.txt returns 200
data table
com 82.5% org 70.4% de 80.8% ru 81.1% net 69.9% co.uk 74.5% com.br 72.2% fr 80.4% nl 82.7% it 81.7% pl 79.4% io 63.8% jp 76.1% com.au 77.2% cz 84.5% in 73.8% es 81.0% ca 76.9% ro 85.0% eu 74.7% co 60.7% co.jp 78.3% ai 87.5% edu 84.5% se 77.2% info 68.8% gr 80.2% ch 77.2% be 89.4% at 79.1%
absent (404)
data table
com 5.5% org 6.4% de 3.5% ru 2.7% net 7.2% co.uk 2.0% com.br 5.4% fr 1.6% nl 3.5% it 2.7% pl 3.9% io 8.3% jp 9.4% com.au 2.5% cz 3.5% in 6.2% es 2.4% ca 2.9% ro 3.5% eu 5.3% co 4.8% co.jp 9.9% ai 5.0% edu 5.0% se 4.6% info 4.8% gr 3.3% ch 7.3% be 1.1% at 3.5%
disallow-all
data table
com 16.2% org 1.1% de 3.3% ru 26.3% net 26.6% co.uk 6.0% com.br 0.6% fr 4.1% nl 1.0% it 2.4% pl 0.8% io 1.9% jp 3.2% com.au 6.4% cz 2.2% in 0.9% es 1.2% ca 1.2% ro 0.7% eu 0.9% co 1.6% co.jp 0.1% ai 2.0% edu 3.1% se 1.0% info 4.5% gr 1.9% ch 1.4% be 0.4% at 0.7%
allow-all
data table
com 17.7% org 20.1% de 10.3% ru 2.4% net 17.0% co.uk 10.2% com.br 17.3% fr 8.1% nl 14.6% it 11.6% pl 18.1% io 33.8% jp 14.9% com.au 12.9% cz 17.5% in 16.3% es 6.9% ca 14.8% ro 14.7% eu 14.2% co 34.3% co.jp 23.9% ai 21.9% edu 16.5% se 14.9% info 32.9% gr 18.9% ch 23.6% be 4.1% at 18.0%
wildcard * UA
data table
com 97.4% org 96.1% de 97.3% ru 98.9% net 97.2% co.uk 98.0% com.br 96.6% fr 92.3% nl 96.9% it 97.7% pl 97.9% io 94.0% jp 85.3% com.au 97.8% cz 97.1% in 96.4% es 98.4% ca 97.3% ro 97.4% eu 98.1% co 92.5% co.jp 93.3% ai 96.5% edu 98.9% se 97.1% info 92.5% gr 97.0% ch 97.4% be 97.5% at 93.3%
declares a sitemap
data table
com 76.3% org 55.9% de 64.2% ru 65.2% net 46.2% co.uk 72.4% com.br 70.1% fr 71.3% nl 75.6% it 62.1% pl 51.6% io 80.2% jp 54.7% com.au 68.9% cz 78.5% in 67.9% es 47.9% ca 53.0% ro 79.3% eu 38.5% co 71.4% co.jp 49.1% ai 85.3% edu 47.4% se 78.2% info 47.1% gr 71.7% ch 68.0% be 22.0% at 72.3%
llms.txt adoption
data table
com 17.3% org 10.8% de 4.7% ru 6.4% net 11.6% co.uk 3.2% com.br 11.1% fr 3.0% nl 9.8% it 5.8% pl 5.5% io 30.5% jp 3.2% com.au 9.1% cz 4.0% in 8.4% es 9.4% ca 3.9% ro 21.0% eu 3.7% co 10.5% co.jp 1.2% ai 32.7% edu 2.5% se 8.6% info 4.1% gr 17.6% ch 5.1% be 2.0% at 4.9%
Full data table
TLD
sample
status 200 status 404 disallow all allow all wildcard ua sitemap llms adoption
com
219,949
82.5%
5.5%
16.2%
17.7%
97.4%
76.3%
17.3%
org
25,762
70.4%
6.4%
1.1%
20.1%
96.1%
55.9%
10.8%
de
19,108
80.8%
3.5%
3.3%
10.3%
97.3%
64.2%
4.7%
ru
18,280
81.1%
2.7%
26.3%
2.4%
98.9%
65.2%
6.4%
net
17,167
69.9%
7.2%
26.6%
17.0%
97.2%
46.2%
11.6%
co.uk
7,948
74.5%
2.0%
6.0%
10.2%
98.0%
72.4%
3.2%
com.br
7,943
72.2%
5.4%
0.6%
17.3%
96.6%
70.1%
11.1%
fr
6,764
80.4%
1.6%
4.1%
8.1%
92.3%
71.3%
3.0%
nl
6,427
82.7%
3.5%
1.0%
14.6%
96.9%
75.6%
9.8%
it
6,186
81.7%
2.7%
2.4%
11.6%
97.7%
62.1%
5.8%
pl
5,500
79.4%
3.9%
0.8%
18.1%
97.9%
51.6%
5.5%
io
4,762
63.8%
8.3%
1.9%
33.8%
94.0%
80.2%
30.5%
jp
4,750
76.1%
9.4%
3.2%
14.9%
85.3%
54.7%
3.2%
com.au
3,713
77.2%
2.5%
6.4%
12.9%
97.8%
68.9%
9.1%
cz
3,512
84.5%
3.5%
2.2%
17.5%
97.1%
78.5%
4.0%
in
3,463
73.8%
6.2%
0.9%
16.3%
96.4%
67.9%
8.4%
es
3,346
81.0%
2.4%
1.2%
6.9%
98.4%
47.9%
9.4%
ca
3,297
76.9%
2.9%
1.2%
14.8%
97.3%
53.0%
3.9%
ro
2,728
85.0%
3.5%
0.7%
14.7%
97.4%
79.3%
21.0%
eu
2,703
74.7%
5.3%
0.9%
14.2%
98.1%
38.5%
3.7%
co
2,645
60.7%
4.8%
1.6%
34.3%
92.5%
71.4%
10.5%
co.jp
2,584
78.3%
9.9%
0.1%
23.9%
93.3%
49.1%
1.2%
ai
2,528
87.5%
5.0%
2.0%
21.9%
96.5%
85.3%
32.7%
edu
2,503
84.5%
5.0%
3.1%
16.5%
98.9%
47.4%
2.5%
se
2,363
77.2%
4.6%
1.0%
14.9%
97.1%
78.2%
8.6%
info
2,204
68.8%
4.8%
4.5%
32.9%
92.5%
47.1%
4.1%
gr
2,118
80.2%
3.3%
1.9%
18.9%
97.0%
71.7%
17.6%
ch
2,099
77.2%
7.3%
1.4%
23.6%
97.4%
68.0%
5.1%
be
1,986
89.4%
1.1%
0.4%
4.1%
97.5%
22.0%
2.0%
at
1,907
79.1%
3.5%
0.7%
18.0%
93.3%
72.3%
4.9%
Showing the 30 best-covered TLDs by sample size.
llms.txt not probed cells: this crawl's
cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.