Data as of
2026-07-25 ·
robots.txt via Common Crawl
CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
·
crawls:
2026-07-25 ·
2026-07-17
687,794 robots.txt fetches ·
491,025 parsed ·
54,000 llms.txt probes
Segments
Headline metrics broken down by segment. Toggle the segmentation axis and
the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a
restriction on a top site counts far more than one deep in the tail.
One crawl ingested (2026-07-25). Cross-crawl trend lines appear
automatically when the next monthly crawl lands.
robots.txt returns 200
data table
com 72.4% org 74.0% de 78.9% ru 77.6% net 68.8% co.uk 62.3% com.br 74.6% fr 74.7% nl 76.8% it 73.9% pl 78.3% io 74.9% jp 62.5% com.au 68.3% cz 80.4% in 66.2% es 73.3% ca 66.0% ro 80.7% eu 72.1% co 73.0% co.jp 58.8% ai 85.2% edu 81.0% se 77.0% info 75.2% gr 78.2% ch 74.9% be 75.4% at 80.7%
absent (404)
data table
com 7.3% org 8.0% de 6.6% ru 5.7% net 13.4% co.uk 4.0% com.br 9.6% fr 4.1% nl 4.6% it 6.0% pl 6.7% io 9.2% jp 17.9% com.au 3.9% cz 5.6% in 10.7% es 5.1% ca 4.8% ro 5.5% eu 8.2% co 7.3% co.jp 20.0% ai 5.2% edu 5.5% se 5.2% info 9.0% gr 4.7% ch 6.4% be 5.3% at 5.7%
disallow-all
data table
com 2.1% org 2.0% de 13.2% ru 3.2% net 3.0% co.uk 2.4% com.br 0.9% fr 1.1% nl 1.5% it 2.2% pl 1.0% io 1.7% jp 2.1% com.au 2.3% cz 2.2% in 1.9% es 1.8% ca 1.5% ro 1.3% eu 2.1% co 2.2% co.jp 0.6% ai 0.9% edu 1.2% se 2.2% info 3.0% gr 1.8% ch 1.9% be 1.1% at 1.5%
allow-all
data table
com 26.5% org 31.0% de 17.2% ru 8.9% net 30.0% co.uk 21.9% com.br 27.9% fr 18.5% nl 24.5% it 24.7% pl 21.0% io 33.9% jp 21.3% com.au 20.8% cz 23.6% in 28.9% es 17.4% ca 22.8% ro 21.6% eu 27.6% co 30.0% co.jp 18.8% ai 33.3% edu 20.9% se 20.6% info 26.0% gr 23.4% ch 24.4% be 17.6% at 25.8%
wildcard * UA
data table
com 93.9% org 93.8% de 96.4% ru 96.6% net 91.4% co.uk 95.8% com.br 95.8% fr 96.3% nl 95.6% it 95.6% pl 96.5% io 91.5% jp 88.6% com.au 96.6% cz 96.5% in 95.6% es 97.3% ca 96.2% ro 96.0% eu 95.2% co 92.1% co.jp 92.2% ai 93.2% edu 97.3% se 96.1% info 90.6% gr 95.0% ch 95.8% be 94.9% at 96.7%
declares a sitemap
data table
com 64.8% org 52.6% de 69.9% ru 73.2% net 51.6% co.uk 69.9% com.br 74.3% fr 64.7% nl 69.6% it 64.1% pl 62.9% io 71.1% jp 59.2% com.au 74.5% cz 73.9% in 73.7% es 68.4% ca 64.5% ro 70.1% eu 59.6% co 70.1% co.jp 60.6% ai 85.5% edu 57.9% se 74.1% info 52.3% gr 62.5% ch 60.4% be 62.5% at 68.5%
llms.txt adoption
data table
com 13.8% org 6.6% de 6.5% ru 6.1% net 7.4% co.uk 10.1% com.br 13.6% fr 6.4% nl 6.1% it 5.4% pl 10.0% io 20.9% jp 6.1% com.au 18.1% cz 6.6% in 14.2% es 8.8% ca 7.0% ro 15.1% eu 12.2% co 17.8% co.jp 6.2% ai 33.7% edu 6.3% se 6.9% info 9.5% gr 13.8% ch 8.5% be 6.4% at 7.3%
Full data table
TLD
sample
status 200 status 404 disallow all allow all wildcard ua sitemap llms adoption
com
219,949
72.4%
7.3%
2.1%
26.5%
93.9%
64.8%
13.8%
org
25,762
74.0%
8.0%
2.0%
31.0%
93.8%
52.6%
6.6%
de
19,108
78.9%
6.6%
13.2%
17.2%
96.4%
69.9%
6.5%
ru
18,280
77.6%
5.7%
3.2%
8.9%
96.6%
73.2%
6.1%
net
17,167
68.8%
13.4%
3.0%
30.0%
91.4%
51.6%
7.4%
co.uk
7,948
62.3%
4.0%
2.4%
21.9%
95.8%
69.9%
10.1%
com.br
7,943
74.6%
9.6%
0.9%
27.9%
95.8%
74.3%
13.6%
fr
6,764
74.7%
4.1%
1.1%
18.5%
96.3%
64.7%
6.4%
nl
6,427
76.8%
4.6%
1.5%
24.5%
95.6%
69.6%
6.1%
it
6,186
73.9%
6.0%
2.2%
24.7%
95.6%
64.1%
5.4%
pl
5,500
78.3%
6.7%
1.0%
21.0%
96.5%
62.9%
10.0%
io
4,762
74.9%
9.2%
1.7%
33.9%
91.5%
71.1%
20.9%
jp
4,750
62.5%
17.9%
2.1%
21.3%
88.6%
59.2%
6.1%
com.au
3,713
68.3%
3.9%
2.3%
20.8%
96.6%
74.5%
18.1%
cz
3,512
80.4%
5.6%
2.2%
23.6%
96.5%
73.9%
6.6%
in
3,463
66.2%
10.7%
1.9%
28.9%
95.6%
73.7%
14.2%
es
3,346
73.3%
5.1%
1.8%
17.4%
97.3%
68.4%
8.8%
ca
3,297
66.0%
4.8%
1.5%
22.8%
96.2%
64.5%
7.0%
ro
2,728
80.7%
5.5%
1.3%
21.6%
96.0%
70.1%
15.1%
eu
2,703
72.1%
8.2%
2.1%
27.6%
95.2%
59.6%
12.2%
co
2,645
73.0%
7.3%
2.2%
30.0%
92.1%
70.1%
17.8%
co.jp
2,584
58.8%
20.0%
0.6%
18.8%
92.2%
60.6%
6.2%
ai
2,528
85.2%
5.2%
0.9%
33.3%
93.2%
85.5%
33.7%
edu
2,503
81.0%
5.5%
1.2%
20.9%
97.3%
57.9%
6.3%
se
2,363
77.0%
5.2%
2.2%
20.6%
96.1%
74.1%
6.9%
info
2,204
75.2%
9.0%
3.0%
26.0%
90.6%
52.3%
9.5%
gr
2,118
78.2%
4.7%
1.8%
23.4%
95.0%
62.5%
13.8%
ch
2,099
74.9%
6.4%
1.9%
24.4%
95.8%
60.4%
8.5%
be
1,986
75.4%
5.3%
1.1%
17.6%
94.9%
62.5%
6.4%
at
1,907
80.7%
5.7%
1.5%
25.8%
96.7%
68.5%
7.3%
Showing the 30 best-covered TLDs by sample size.
llms.txt not probed cells: this crawl's
cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.