Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-09-07). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

com	63.3%
org	67.8%
net	47.2%
ru	60.4%
de	68.4%
nl	83.1%
co.uk	60.6%
it	76.9%
com.br	64.6%
uk	91.1%
fr	63.7%
pl	72.9%
io	56.9%
in	59.2%
ca	71.8%
jp	53.2%
com.au	69.4%
xyz	43.2%
co	64.3%
cz	74.8%
es	67.4%
eu	58.1%
us	66.9%
info	53.4%
cc	48.2%
ro	74.0%
online	47.3%
ai	78.8%
se	74.6%
site	52.6%

absent (404)

data table

com	8.1%
org	8.8%
net	11.9%
ru	6.7%
de	7.8%
nl	3.6%
co.uk	3.7%
it	5.0%
com.br	12.7%
uk	1.1%
fr	4.2%
pl	7.8%
io	8.8%
in	14.6%
ca	5.4%
jp	17.4%
com.au	5.0%
xyz	6.0%
co	7.7%
cz	6.8%
es	6.2%
eu	9.0%
us	4.0%
info	9.0%
cc	6.2%
ro	8.2%
online	8.9%
ai	6.3%
se	5.6%
site	6.3%

disallow-all

data table

com	7.5%
org	2.8%
net	6.2%
ru	4.1%
de	14.1%
nl	57.1%
co.uk	34.4%
it	40.1%
com.br	2.4%
uk	94.3%
fr	1.9%
pl	1.6%
io	3.4%
in	3.1%
ca	28.6%
jp	3.9%
com.au	1.9%
xyz	18.3%
co	4.1%
cz	3.8%
es	3.2%
eu	3.5%
us	68.8%
info	6.8%
cc	7.2%
ro	1.5%
online	7.7%
ai	1.8%
se	11.5%
site	5.2%

allow-all

data table

com	26.4%
org	32.0%
net	30.0%
ru	10.6%
de	17.6%
nl	10.9%
co.uk	14.1%
it	14.9%
com.br	28.1%
uk	1.9%
fr	19.7%
pl	22.6%
io	34.4%
in	28.8%
ca	16.0%
jp	21.5%
com.au	19.0%
xyz	44.5%
co	32.4%
cz	22.6%
es	19.4%
eu	26.4%
us	10.3%
info	29.0%
cc	41.6%
ro	20.6%
online	46.3%
ai	33.4%
se	19.6%
site	55.9%

wildcard * UA

data table

com	88.1%
org	91.4%
net	84.6%
ru	90.9%
de	95.1%
nl	97.4%
co.uk	96.3%
it	96.3%
com.br	91.7%
uk	99.1%
fr	94.5%
pl	94.6%
io	85.5%
in	91.1%
ca	96.1%
jp	88.5%
com.au	96.3%
xyz	78.7%
co	87.0%
cz	94.8%
es	95.7%
eu	91.6%
us	96.1%
info	86.6%
cc	82.3%
ro	94.5%
online	85.7%
ai	89.9%
se	94.5%
site	82.5%

declares a sitemap

data table

com	59.8%
org	51.0%
net	46.0%
ru	67.8%
de	68.2%
nl	86.3%
co.uk	79.6%
it	78.3%
com.br	69.1%
uk	97.0%
fr	62.6%
pl	60.9%
io	62.6%
in	67.7%
ca	74.6%
jp	58.8%
com.au	76.3%
xyz	49.2%
co	59.3%
cz	72.7%
es	67.5%
eu	55.6%
us	84.0%
info	46.8%
cc	34.5%
ro	69.0%
online	43.4%
ai	78.9%
se	73.5%
site	44.2%

Full data table

TLD	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
com	325,550	63.3%	8.1%	7.5%	26.4%	88.1%	59.8%
org	29,741	67.8%	8.8%	2.8%	32.0%	91.4%	51.0%
net	26,249	47.2%	11.9%	6.2%	30.0%	84.6%	46.0%
ru	25,538	60.4%	6.7%	4.1%	10.6%	90.9%	67.8%
de	21,780	68.4%	7.8%	14.1%	17.6%	95.1%	68.2%
nl	16,877	83.1%	3.6%	57.1%	10.9%	97.4%	86.3%
co.uk	15,289	60.6%	3.7%	34.4%	14.1%	96.3%	79.6%
it	12,404	76.9%	5.0%	40.1%	14.9%	96.3%	78.3%
com.br	10,185	64.6%	12.7%	2.4%	28.1%	91.7%	69.1%
uk	10,182	91.1%	1.1%	94.3%	1.9%	99.1%	97.0%
fr	8,118	63.7%	4.2%	1.9%	19.7%	94.5%	62.6%
pl	6,464	72.9%	7.8%	1.6%	22.6%	94.6%	60.9%
io	6,386	56.9%	8.8%	3.4%	34.4%	85.5%	62.6%
in	5,857	59.2%	14.6%	3.1%	28.8%	91.1%	67.7%
ca	5,743	71.8%	5.4%	28.6%	16.0%	96.1%	74.6%
jp	5,275	53.2%	17.4%	3.9%	21.5%	88.5%	58.8%
com.au	4,919	69.4%	5.0%	1.9%	19.0%	96.3%	76.3%
xyz	4,503	43.2%	6.0%	18.3%	44.5%	78.7%	49.2%
co	4,207	64.3%	7.7%	4.1%	32.4%	87.0%	59.3%
cz	4,138	74.8%	6.8%	3.8%	22.6%	94.8%	72.7%
es	4,085	67.4%	6.2%	3.2%	19.4%	95.7%	67.5%
eu	3,826	58.1%	9.0%	3.5%	26.4%	91.6%	55.6%
us	3,787	66.9%	4.0%	68.8%	10.3%	96.1%	84.0%
info	3,566	53.4%	9.0%	6.8%	29.0%	86.6%	46.8%
cc	3,445	48.2%	6.2%	7.2%	41.6%	82.3%	34.5%
ro	3,357	74.0%	8.2%	1.5%	20.6%	94.5%	69.0%
online	3,266	47.3%	8.9%	7.7%	46.3%	85.7%	43.4%
ai	3,093	78.8%	6.3%	1.8%	33.4%	89.9%	78.9%
se	2,964	74.6%	5.6%	11.5%	19.6%	94.5%	73.5%
site	2,962	52.6%	6.3%	5.2%	55.9%	82.5%	44.2%

Showing the 30 best-covered TLDs by sample size.

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.