Data as of 2026-08-02 · robots.txt via Common Crawl CC-MAIN-2026-30 + our polite crawl · llms.txt via our polite crawl · Tranco list L5684 (2026-07-14) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,000,000 robots.txt fetches · 608,834 parsed · 55,770 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-08-02). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

com	67.3%
org	63.2%
net	22.1%
ru	66.6%
de	70.0%
co.uk	59.9%
nl	74.3%
com.br	64.9%
fr	71.3%
it	67.6%
io	46.9%
pl	72.4%
in	65.7%
ca	76.2%
jp	63.5%
com.au	75.1%
co	48.8%
es	76.2%
cz	77.7%
us	56.4%
xyz	28.7%
eu	56.9%
info	47.2%
ro	76.5%
ai	81.7%
se	74.4%
co.jp	79.1%
cc	44.7%
edu	87.7%
online	41.6%

absent (404)

data table

com	6.3%
org	7.0%
net	3.1%
ru	5.4%
de	5.4%
co.uk	2.2%
nl	3.7%
com.br	7.3%
fr	2.2%
it	3.1%
io	7.3%
pl	4.9%
in	9.7%
ca	3.7%
jp	10.0%
com.au	3.8%
co	7.1%
es	3.7%
cz	4.8%
us	6.4%
xyz	8.8%
eu	8.2%
info	5.9%
ro	5.4%
ai	4.6%
se	3.2%
co.jp	9.2%
cc	9.0%
edu	2.9%
online	7.0%

disallow-all

data table

com	16.8%
org	1.6%
net	24.3%
ru	25.0%
de	4.6%
co.uk	11.3%
nl	29.0%
com.br	1.4%
fr	4.7%
it	3.2%
io	4.5%
pl	1.1%
in	2.0%
ca	6.6%
jp	3.7%
com.au	9.3%
co	2.8%
es	1.5%
cz	1.9%
us	14.5%
xyz	13.8%
eu	1.7%
info	7.4%
ro	1.0%
ai	1.7%
se	4.2%
co.jp	0.1%
cc	8.4%
edu	2.5%
online	8.5%

allow-all

data table

com	14.8%
org	21.9%
net	16.1%
ru	2.8%
de	11.4%
co.uk	12.1%
nl	12.7%
com.br	18.3%
fr	9.0%
it	12.3%
io	32.1%
pl	20.2%
in	19.9%
ca	14.2%
jp	15.4%
com.au	13.3%
co	34.8%
es	9.2%
cz	18.8%
us	5.9%
xyz	43.8%
eu	13.8%
info	32.7%
ro	15.5%
ai	23.6%
se	19.5%
co.jp	19.7%
cc	39.3%
edu	17.0%
online	37.4%

wildcard * UA

data table

com	96.8%
org	95.4%
net	91.2%
ru	92.9%
de	96.4%
co.uk	97.4%
nl	95.8%
com.br	94.1%
fr	93.6%
it	97.0%
io	88.4%
pl	96.0%
in	93.0%
ca	96.8%
jp	84.3%
com.au	97.7%
co	89.4%
es	98.0%
cz	95.8%
us	96.5%
xyz	76.1%
eu	95.8%
info	86.6%
ro	96.2%
ai	91.4%
se	96.6%
co.jp	94.8%
cc	86.5%
edu	98.1%
online	82.6%

declares a sitemap

data table

com	75.9%
org	53.5%
net	47.2%
ru	59.3%
de	65.4%
co.uk	76.0%
nl	78.9%
com.br	69.2%
fr	71.4%
it	62.1%
io	72.9%
pl	52.7%
in	67.7%
ca	58.8%
jp	55.6%
com.au	73.1%
co	63.7%
es	53.8%
cz	76.3%
us	87.5%
xyz	39.0%
eu	38.1%
info	40.3%
ro	77.7%
ai	78.7%
se	74.1%
co.jp	53.3%
cc	37.2%
edu	46.7%
online	41.0%

Full data table

TLD	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
com	275,832	67.3%	6.3%	16.8%	14.8%	96.8%	75.9%
org	25,745	63.2%	7.0%	1.6%	21.9%	95.4%	53.5%
net	22,137	22.1%	3.1%	24.3%	16.1%	91.2%	47.2%
ru	22,003	66.6%	5.4%	25.0%	2.8%	92.9%	59.3%
de	19,231	70.0%	5.4%	4.6%	11.4%	96.4%	65.4%
co.uk	12,150	59.9%	2.2%	11.3%	12.1%	97.4%	76.0%
nl	11,210	74.3%	3.7%	29.0%	12.7%	95.8%	78.9%
com.br	8,889	64.9%	7.3%	1.4%	18.3%	94.1%	69.2%
fr	6,776	71.3%	2.2%	4.7%	9.0%	93.6%	71.4%
it	6,458	67.6%	3.1%	3.2%	12.3%	97.0%	62.1%
io	5,657	46.9%	7.3%	4.5%	32.1%	88.4%	72.9%
pl	5,378	72.4%	4.9%	1.1%	20.2%	96.0%	52.7%
in	4,862	65.7%	9.7%	2.0%	19.9%	93.0%	67.7%
ca	4,839	76.2%	3.7%	6.6%	14.2%	96.8%	58.8%
jp	4,750	63.5%	10.0%	3.7%	15.4%	84.3%	55.6%
com.au	4,152	75.1%	3.8%	9.3%	13.3%	97.7%	73.1%
co	3,631	48.8%	7.1%	2.8%	34.8%	89.4%	63.7%
es	3,554	76.2%	3.7%	1.5%	9.2%	98.0%	53.8%
cz	3,484	77.7%	4.8%	1.9%	18.8%	95.8%	76.3%
us	3,381	56.4%	6.4%	14.5%	5.9%	96.5%	87.5%
xyz	3,291	28.7%	8.8%	13.8%	43.8%	76.1%	39.0%
eu	3,091	56.9%	8.2%	1.7%	13.8%	95.8%	38.1%
info	2,931	47.2%	5.9%	7.4%	32.7%	86.6%	40.3%
ro	2,710	76.5%	5.4%	1.0%	15.5%	96.2%	77.7%
ai	2,586	81.7%	4.6%	1.7%	23.6%	91.4%	78.7%
se	2,566	74.4%	3.2%	4.2%	19.5%	96.6%	74.1%
co.jp	2,544	79.1%	9.2%	0.1%	19.7%	94.8%	53.3%
cc	2,404	44.7%	9.0%	8.4%	39.3%	86.5%	37.2%
edu	2,376	87.7%	2.9%	2.5%	17.0%	98.1%	46.7%
online	2,324	41.6%	7.0%	8.5%	37.4%	82.6%	41.0%

Showing the 30 best-covered TLDs by sample size.

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.