Data as of 2026-08-02 · robots.txt via Common Crawl CC-MAIN-2026-30 + our polite crawl · llms.txt via our polite crawl · Tranco list L5684 (2026-07-14) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,000,000 robots.txt fetches · 608,834 parsed · 55,770 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-08-02). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

com	64.5%
org	67.7%
net	46.0%
ru	61.3%
de	69.0%
co.uk	55.7%
nl	79.9%
com.br	64.4%
fr	63.1%
it	67.1%
io	55.9%
pl	72.4%
in	61.6%
ca	69.6%
jp	53.6%
com.au	71.7%
co	65.7%
es	68.4%
cz	74.3%
us	65.2%
xyz	40.4%
eu	58.0%
info	55.1%
ro	73.8%
ai	77.6%
se	73.8%
co.jp	58.2%
cc	48.7%
edu	79.2%
online	47.8%

absent (404)

data table

com	8.1%
org	8.5%
net	12.1%
ru	6.6%
de	7.2%
co.uk	3.4%
nl	4.3%
com.br	12.8%
fr	4.1%
it	6.9%
io	9.3%
pl	8.0%
in	14.4%
ca	5.1%
jp	16.8%
com.au	5.6%
co	8.2%
es	6.1%
cz	6.6%
us	3.8%
xyz	5.4%
eu	9.1%
info	9.1%
ro	8.3%
ai	6.6%
se	5.1%
co.jp	18.4%
cc	6.8%
edu	5.3%
online	8.5%

disallow-all

data table

com	8.1%
org	2.8%
net	6.4%
ru	4.5%
de	15.6%
co.uk	34.4%
nl	45.5%
com.br	2.3%
fr	1.8%
it	3.8%
io	3.4%
pl	1.6%
in	3.4%
ca	27.5%
jp	4.0%
com.au	2.0%
co	4.2%
es	3.1%
cz	3.7%
us	71.3%
xyz	22.5%
eu	3.8%
info	7.5%
ro	1.8%
ai	1.8%
se	11.9%
co.jp	0.7%
cc	8.7%
edu	1.1%
online	9.3%

allow-all

data table

com	25.1%
org	31.2%
net	29.5%
ru	9.8%
de	16.5%
co.uk	13.5%
nl	13.6%
com.br	27.7%
fr	18.1%
it	23.3%
io	33.3%
pl	21.5%
in	27.2%
ca	15.8%
jp	20.9%
com.au	18.7%
co	30.9%
es	18.1%
cz	22.7%
us	9.1%
xyz	39.7%
eu	26.1%
info	27.1%
ro	20.4%
ai	33.4%
se	19.1%
co.jp	18.8%
cc	42.7%
edu	20.9%
online	41.0%

wildcard * UA

data table

com	88.7%
org	91.8%
net	86.0%
ru	90.8%
de	94.7%
co.uk	96.2%
nl	96.8%
com.br	91.4%
fr	94.6%
it	94.2%
io	84.9%
pl	94.3%
in	91.3%
ca	95.9%
jp	88.8%
com.au	96.0%
co	86.4%
es	95.8%
cz	94.7%
us	96.5%
xyz	76.9%
eu	91.0%
info	84.0%
ro	94.3%
ai	89.8%
se	95.0%
co.jp	92.8%
cc	79.7%
edu	97.6%
online	81.0%

declares a sitemap

data table

com	61.1%
org	51.5%
net	47.0%
ru	67.2%
de	69.5%
co.uk	80.4%
nl	83.1%
com.br	69.4%
fr	63.9%
it	65.3%
io	62.5%
pl	60.7%
in	68.7%
ca	74.4%
jp	59.0%
com.au	76.7%
co	60.0%
es	68.3%
cz	73.1%
us	85.7%
xyz	44.8%
eu	55.9%
info	45.0%
ro	69.0%
ai	79.0%
se	74.8%
co.jp	61.8%
cc	29.4%
edu	57.9%
online	39.7%

Full data table

TLD	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
com	275,832	64.5%	8.1%	8.1%	25.1%	88.7%	61.1%
org	25,745	67.7%	8.5%	2.8%	31.2%	91.8%	51.5%
net	22,137	46.0%	12.1%	6.4%	29.5%	86.0%	47.0%
ru	22,003	61.3%	6.6%	4.5%	9.8%	90.8%	67.2%
de	19,231	69.0%	7.2%	15.6%	16.5%	94.7%	69.5%
co.uk	12,150	55.7%	3.4%	34.4%	13.5%	96.2%	80.4%
nl	11,210	79.9%	4.3%	45.5%	13.6%	96.8%	83.1%
com.br	8,889	64.4%	12.8%	2.3%	27.7%	91.4%	69.4%
fr	6,776	63.1%	4.1%	1.8%	18.1%	94.6%	63.9%
it	6,458	67.1%	6.9%	3.8%	23.3%	94.2%	65.3%
io	5,657	55.9%	9.3%	3.4%	33.3%	84.9%	62.5%
pl	5,378	72.4%	8.0%	1.6%	21.5%	94.3%	60.7%
in	4,862	61.6%	14.4%	3.4%	27.2%	91.3%	68.7%
ca	4,839	69.6%	5.1%	27.5%	15.8%	95.9%	74.4%
jp	4,750	53.6%	16.8%	4.0%	20.9%	88.8%	59.0%
com.au	4,152	71.7%	5.6%	2.0%	18.7%	96.0%	76.7%
co	3,631	65.7%	8.2%	4.2%	30.9%	86.4%	60.0%
es	3,554	68.4%	6.1%	3.1%	18.1%	95.8%	68.3%
cz	3,484	74.3%	6.6%	3.7%	22.7%	94.7%	73.1%
us	3,381	65.2%	3.8%	71.3%	9.1%	96.5%	85.7%
xyz	3,291	40.4%	5.4%	22.5%	39.7%	76.9%	44.8%
eu	3,091	58.0%	9.1%	3.8%	26.1%	91.0%	55.9%
info	2,931	55.1%	9.1%	7.5%	27.1%	84.0%	45.0%
ro	2,710	73.8%	8.3%	1.8%	20.4%	94.3%	69.0%
ai	2,586	77.6%	6.6%	1.8%	33.4%	89.8%	79.0%
se	2,566	73.8%	5.1%	11.9%	19.1%	95.0%	74.8%
co.jp	2,544	58.2%	18.4%	0.7%	18.8%	92.8%	61.8%
cc	2,404	48.7%	6.8%	8.7%	42.7%	79.7%	29.4%
edu	2,376	79.2%	5.3%	1.1%	20.9%	97.6%	57.9%
online	2,324	47.8%	8.5%	9.3%	41.0%	81.0%	39.7%

Showing the 30 best-covered TLDs by sample size.

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.