Data as of
2026-07-17 ·
robots.txt via Common Crawl
CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
·
crawls:
2026-07-25 ·
2026-07-17
746,000 robots.txt fetches ·
574,664 parsed ·
54,000 llms.txt probes
Segments
Headline metrics broken down by segment. Toggle the segmentation axis and
the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a
restriction on a top site counts far more than one deep in the tail.
One crawl ingested (2026-07-17). Cross-crawl trend lines appear
automatically when the next monthly crawl lands.
robots.txt returns 200
data table
com 69.4% org 69.3% de 79.5% net 54.0% ru 80.1% co.uk 72.6% com.br 69.8% fr 75.6% nl 78.2% it 78.5% pl 74.8% jp 74.2% io 53.9% com.au 74.2% in 69.5% ca 74.6% cz 79.4% es 79.4% co.jp 73.5% eu 70.9% co 54.1% edu 86.9% ro 80.3% ai 78.5% se 71.5% ch 78.0% info 65.4% be 89.2% gr 69.3% at 75.6%
absent (404)
data table
com 18.7% org 14.6% de 6.6% net 12.8% ru 6.0% co.uk 3.6% com.br 10.8% fr 6.6% nl 8.4% it 6.7% pl 9.0% jp 12.3% io 24.6% com.au 4.7% in 9.1% ca 5.6% cz 8.2% es 6.2% co.jp 11.3% eu 16.3% co 15.5% edu 5.1% ro 7.1% ai 8.6% se 9.1% ch 6.4% info 9.0% be 2.5% gr 8.5% at 8.7%
disallow-all
data table
com 20.0% org 5.3% de 6.8% net 38.3% ru 10.1% co.uk 8.3% com.br 5.8% fr 7.6% nl 3.2% it 4.5% pl 4.1% jp 5.1% io 5.3% com.au 13.4% in 2.5% ca 5.2% cz 4.6% es 2.7% co.jp 1.1% eu 13.5% co 14.7% edu 11.4% ro 3.7% ai 5.3% se 3.8% ch 4.9% info 5.5% be 1.8% gr 2.9% at 3.2%
allow-all
data table
com 20.5% org 20.3% de 15.2% net 15.4% ru 6.6% co.uk 10.4% com.br 18.0% fr 11.3% nl 16.4% it 13.6% pl 15.4% jp 18.3% io 23.8% com.au 14.2% in 18.3% ca 14.4% cz 18.7% es 9.3% co.jp 17.4% eu 16.0% co 27.1% edu 13.1% ro 17.9% ai 21.5% se 19.5% ch 21.0% info 31.6% be 6.1% gr 17.9% at 20.4%
wildcard * UA
data table
com 96.9% org 96.1% de 96.9% net 93.5% ru 98.7% co.uk 95.3% com.br 97.4% fr 90.4% nl 96.4% it 96.1% pl 98.1% jp 86.2% io 82.2% com.au 97.4% in 96.0% ca 95.8% cz 96.9% es 98.2% co.jp 94.6% eu 97.6% co 93.0% edu 97.6% ro 97.2% ai 96.2% se 96.4% ch 95.2% info 92.0% be 97.9% gr 97.9% at 90.4%
declares a sitemap
data table
com 63.6% org 48.2% de 59.7% net 32.5% ru 72.9% co.uk 71.2% com.br 65.0% fr 64.2% nl 71.2% it 57.3% pl 50.3% jp 55.2% io 66.5% com.au 68.1% in 64.9% ca 50.7% cz 73.7% es 48.6% co.jp 59.2% eu 62.6% co 57.1% edu 52.6% ro 72.9% ai 79.3% se 63.9% ch 59.3% info 42.8% be 20.0% gr 70.5% at 59.9%
llms.txt adoption
data table
com 17.3% org 10.8% de 4.7% net 11.6% ru 6.4% co.uk 3.2% com.br 11.1% fr 3.0% nl 9.8% it 5.8% pl 5.5% jp 3.2% io 30.5% com.au 9.1% in 8.4% ca 3.9% cz 4.0% es 9.4% co.jp 1.2% eu 3.7% co 10.5% edu 2.5% ro 21.0% ai 32.7% se 8.6% ch 5.1% info 4.1% be 2.0% gr 17.6% at 4.9%
Full data table
TLD
sample
status 200 status 404 disallow all allow all wildcard ua sitemap llms adoption
com
255,407
69.4%
18.7%
20.0%
20.5%
96.9%
63.6%
17.3%
org
29,696
69.3%
14.6%
5.3%
20.3%
96.1%
48.2%
10.8%
de
21,439
79.5%
6.6%
6.8%
15.2%
96.9%
59.7%
4.7%
net
21,139
54.0%
12.8%
38.3%
15.4%
93.5%
32.5%
11.6%
ru
20,542
80.1%
6.0%
10.1%
6.6%
98.7%
72.9%
6.4%
co.uk
9,197
72.6%
3.6%
8.3%
10.4%
95.3%
71.2%
3.2%
com.br
9,067
69.8%
10.8%
5.8%
18.0%
97.4%
65.0%
11.1%
fr
7,883
75.6%
6.6%
7.6%
11.3%
90.4%
64.2%
3.0%
nl
7,177
78.2%
8.4%
3.2%
16.4%
96.4%
71.2%
9.8%
it
6,954
78.5%
6.7%
4.5%
13.6%
96.1%
57.3%
5.8%
pl
6,207
74.8%
9.0%
4.1%
15.4%
98.1%
50.3%
5.5%
jp
6,019
74.2%
12.3%
5.1%
18.3%
86.2%
55.2%
3.2%
io
5,990
53.9%
24.6%
5.3%
23.8%
82.2%
66.5%
30.5%
com.au
4,537
74.2%
4.7%
13.4%
14.2%
97.4%
68.1%
9.1%
in
4,006
69.5%
9.1%
2.5%
18.3%
96.0%
64.9%
8.4%
ca
3,966
74.6%
5.6%
5.2%
14.4%
95.8%
50.7%
3.9%
cz
3,925
79.4%
8.2%
4.6%
18.7%
96.9%
73.7%
4.0%
es
3,866
79.4%
6.2%
2.7%
9.3%
98.2%
48.6%
9.4%
co.jp
3,335
73.5%
11.3%
1.1%
17.4%
94.6%
59.2%
1.2%
eu
3,256
70.9%
16.3%
13.5%
16.0%
97.6%
62.6%
3.7%
co
3,214
54.1%
15.5%
14.7%
27.1%
93.0%
57.1%
10.5%
edu
3,142
86.9%
5.1%
11.4%
13.1%
97.6%
52.6%
2.5%
ro
2,949
80.3%
7.1%
3.7%
17.9%
97.2%
72.9%
21.0%
ai
2,793
78.5%
8.6%
5.3%
21.5%
96.2%
79.3%
32.7%
se
2,677
71.5%
9.1%
3.8%
19.5%
96.4%
63.9%
8.6%
ch
2,508
78.0%
6.4%
4.9%
21.0%
95.2%
59.3%
5.1%
info
2,495
65.4%
9.0%
5.5%
31.6%
92.0%
42.8%
4.1%
be
2,305
89.2%
2.5%
1.8%
6.1%
97.9%
20.0%
2.0%
gr
2,284
69.3%
8.5%
2.9%
17.9%
97.9%
70.5%
17.6%
at
2,116
75.6%
8.7%
3.2%
20.4%
90.4%
59.9%
4.9%
Showing the 30 best-covered TLDs by sample size.
llms.txt not probed cells: this crawl's
cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.