Data as of
2026-07-17 ·
robots.txt via Common Crawl
CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
·
crawls:
2026-07-25 ·
2026-07-17
746,000 robots.txt fetches ·
574,664 parsed ·
54,000 llms.txt probes
Segments
Headline metrics broken down by segment. Toggle the segmentation axis and
the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a
restriction on a top site counts far more than one deep in the tail.
One crawl ingested (2026-07-17). Cross-crawl trend lines appear
automatically when the next monthly crawl lands.
robots.txt returns 200
data table
com 68.9% org 70.9% de 74.6% net 64.0% ru 74.1% co.uk 60.2% com.br 70.3% fr 71.8% nl 72.7% it 70.1% pl 74.9% jp 61.1% io 66.0% com.au 66.5% in 63.3% ca 63.1% cz 76.3% es 71.0% co.jp 58.8% eu 67.9% co 68.1% edu 73.2% ro 78.1% ai 78.2% se 72.6% ch 71.6% info 72.3% be 71.8% gr 74.9% at 75.7%
absent (404)
data table
com 10.5% org 11.3% de 10.1% net 18.1% ru 9.1% co.uk 6.0% com.br 13.5% fr 7.9% nl 8.3% it 9.8% pl 9.9% jp 19.8% io 16.4% com.au 6.4% in 13.4% ca 7.8% cz 9.6% es 8.9% co.jp 20.8% eu 12.6% co 11.0% edu 14.1% ro 7.8% ai 10.2% se 9.3% ch 9.7% info 11.3% be 9.4% gr 8.0% at 10.2%
disallow-all
data table
com 4.1% org 3.8% de 14.0% net 6.6% ru 5.0% co.uk 3.9% com.br 2.4% fr 3.2% nl 3.9% it 3.6% pl 2.1% jp 3.6% io 5.0% com.au 5.0% in 2.9% ca 4.2% cz 3.8% es 3.2% co.jp 2.1% eu 5.1% co 4.3% edu 4.8% ro 2.3% ai 3.3% se 4.1% ch 4.3% info 4.4% be 3.2% gr 2.2% at 3.8%
allow-all
data table
com 25.8% org 29.8% de 17.8% net 28.9% ru 10.6% co.uk 20.7% com.br 29.2% fr 19.5% nl 24.6% it 25.2% pl 21.3% jp 21.8% io 32.1% com.au 18.8% in 28.8% ca 21.7% cz 23.8% es 18.4% co.jp 21.0% eu 26.1% co 28.9% edu 21.3% ro 22.7% ai 32.9% se 21.4% ch 24.6% info 25.8% be 18.4% gr 23.3% at 25.0%
wildcard * UA
data table
com 93.6% org 93.3% de 95.3% net 91.0% ru 96.2% co.uk 95.7% com.br 95.4% fr 95.7% nl 95.3% it 95.3% pl 96.0% jp 88.9% io 90.8% com.au 96.4% in 95.3% ca 95.7% cz 96.3% es 96.9% co.jp 93.0% eu 94.7% co 91.7% edu 96.9% ro 95.8% ai 93.5% se 96.4% ch 94.8% info 89.9% be 94.6% gr 94.6% at 95.3%
declares a sitemap
data table
com 61.0% org 49.4% de 65.2% net 46.9% ru 68.8% co.uk 67.8% com.br 68.7% fr 60.1% nl 65.8% it 60.9% pl 59.4% jp 57.7% io 59.6% com.au 71.1% in 70.7% ca 60.5% cz 69.8% es 64.0% co.jp 60.0% eu 54.3% co 63.3% edu 52.2% ro 66.5% ai 77.3% se 66.1% ch 55.1% info 49.9% be 59.3% gr 60.7% at 61.6%
llms.txt adoption
data table
com 13.8% org 6.6% de 6.5% net 7.4% ru 6.1% co.uk 10.1% com.br 13.6% fr 6.4% nl 6.1% it 5.4% pl 10.0% jp 6.1% io 20.9% com.au 18.1% in 14.2% ca 7.0% cz 6.6% es 8.8% co.jp 6.2% eu 12.2% co 17.8% edu 6.3% ro 15.1% ai 33.7% se 6.9% ch 8.5% info 9.5% be 6.4% gr 13.8% at 7.3%
Full data table
TLD
sample
status 200 status 404 disallow all allow all wildcard ua sitemap llms adoption
com
255,407
68.9%
10.5%
4.1%
25.8%
93.6%
61.0%
13.8%
org
29,696
70.9%
11.3%
3.8%
29.8%
93.3%
49.4%
6.6%
de
21,439
74.6%
10.1%
14.0%
17.8%
95.3%
65.2%
6.5%
net
21,139
64.0%
18.1%
6.6%
28.9%
91.0%
46.9%
7.4%
ru
20,542
74.1%
9.1%
5.0%
10.6%
96.2%
68.8%
6.1%
co.uk
9,197
60.2%
6.0%
3.9%
20.7%
95.7%
67.8%
10.1%
com.br
9,067
70.3%
13.5%
2.4%
29.2%
95.4%
68.7%
13.6%
fr
7,883
71.8%
7.9%
3.2%
19.5%
95.7%
60.1%
6.4%
nl
7,177
72.7%
8.3%
3.9%
24.6%
95.3%
65.8%
6.1%
it
6,954
70.1%
9.8%
3.6%
25.2%
95.3%
60.9%
5.4%
pl
6,207
74.9%
9.9%
2.1%
21.3%
96.0%
59.4%
10.0%
jp
6,019
61.1%
19.8%
3.6%
21.8%
88.9%
57.7%
6.1%
io
5,990
66.0%
16.4%
5.0%
32.1%
90.8%
59.6%
20.9%
com.au
4,537
66.5%
6.4%
5.0%
18.8%
96.4%
71.1%
18.1%
in
4,006
63.3%
13.4%
2.9%
28.8%
95.3%
70.7%
14.2%
ca
3,966
63.1%
7.8%
4.2%
21.7%
95.7%
60.5%
7.0%
cz
3,925
76.3%
9.6%
3.8%
23.8%
96.3%
69.8%
6.6%
es
3,866
71.0%
8.9%
3.2%
18.4%
96.9%
64.0%
8.8%
co.jp
3,335
58.8%
20.8%
2.1%
21.0%
93.0%
60.0%
6.2%
eu
3,256
67.9%
12.6%
5.1%
26.1%
94.7%
54.3%
12.2%
co
3,214
68.1%
11.0%
4.3%
28.9%
91.7%
63.3%
17.8%
edu
3,142
73.2%
14.1%
4.8%
21.3%
96.9%
52.2%
6.3%
ro
2,949
78.1%
7.8%
2.3%
22.7%
95.8%
66.5%
15.1%
ai
2,793
78.2%
10.2%
3.3%
32.9%
93.5%
77.3%
33.7%
se
2,677
72.6%
9.3%
4.1%
21.4%
96.4%
66.1%
6.9%
ch
2,508
71.6%
9.7%
4.3%
24.6%
94.8%
55.1%
8.5%
info
2,495
72.3%
11.3%
4.4%
25.8%
89.9%
49.9%
9.5%
be
2,305
71.8%
9.4%
3.2%
18.4%
94.6%
59.3%
6.4%
gr
2,284
74.9%
8.0%
2.2%
23.3%
94.6%
60.7%
13.8%
at
2,116
75.7%
10.2%
3.8%
25.0%
95.3%
61.6%
7.3%
Showing the 30 best-covered TLDs by sample size.
llms.txt not probed cells: this crawl's
cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.