Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-09-07). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

com	67.6%
org	63.8%
net	26.8%
ru	67.0%
de	73.5%
nl	79.4%
co.uk	60.6%
it	70.4%
com.br	65.4%
uk	83.9%
fr	70.5%
pl	74.5%
io	48.9%
in	63.9%
ca	76.6%
jp	61.9%
com.au	73.5%
xyz	35.5%
co	58.5%
cz	77.2%
es	74.5%
eu	54.8%
us	55.7%
info	48.4%
cc	45.7%
ro	77.4%
online	40.2%
ai	80.8%
se	75.4%
site	44.8%

absent (404)

data table

com	6.1%
org	8.4%
net	4.3%
ru	5.4%
de	5.6%
nl	3.0%
co.uk	2.4%
it	3.0%
com.br	7.0%
uk	1.3%
fr	2.3%
pl	5.1%
io	6.9%
in	10.4%
ca	3.9%
jp	9.7%
com.au	3.9%
xyz	8.4%
co	4.3%
cz	4.8%
es	3.5%
eu	9.1%
us	6.5%
info	6.4%
cc	8.1%
ro	5.6%
online	7.9%
ai	4.5%
se	4.7%
site	10.9%

disallow-all

data table

com	17.8%
org	1.6%
net	27.7%
ru	25.1%
de	5.0%
nl	32.2%
co.uk	12.3%
it	13.8%
com.br	1.3%
uk	87.7%
fr	2.8%
pl	1.2%
io	4.3%
in	2.0%
ca	7.5%
jp	4.1%
com.au	9.5%
xyz	12.5%
co	17.5%
cz	2.0%
es	1.5%
eu	1.6%
us	15.1%
info	6.7%
cc	6.6%
ro	0.9%
online	7.9%
ai	1.5%
se	4.3%
site	5.0%

allow-all

data table

com	17.3%
org	23.0%
net	15.0%
ru	3.1%
de	15.9%
nl	25.4%
co.uk	13.0%
it	11.3%
com.br	18.8%
uk	4.7%
fr	9.9%
pl	19.8%
io	31.3%
in	19.4%
ca	14.4%
jp	14.6%
com.au	13.5%
xyz	46.2%
co	29.9%
cz	19.4%
es	9.8%
eu	15.0%
us	6.6%
info	32.8%
cc	45.7%
ro	16.5%
online	41.1%
ai	24.3%
se	15.5%
site	43.5%

wildcard * UA

data table

com	96.8%
org	95.2%
net	89.2%
ru	96.5%
de	96.7%
nl	97.0%
co.uk	97.0%
it	97.3%
com.br	93.5%
uk	98.0%
fr	93.1%
pl	96.1%
io	87.6%
in	90.7%
ca	96.8%
jp	86.8%
com.au	97.8%
xyz	77.4%
co	91.0%
cz	95.8%
es	97.9%
eu	95.6%
us	96.5%
info	89.6%
cc	89.0%
ro	96.3%
online	85.3%
ai	93.3%
se	96.7%
site	78.5%

declares a sitemap

data table

com	75.4%
org	53.5%
net	44.3%
ru	62.1%
de	62.0%
nl	68.8%
co.uk	75.4%
it	68.3%
com.br	67.5%
uk	94.2%
fr	70.6%
pl	52.5%
io	72.2%
in	65.7%
ca	59.5%
jp	56.7%
com.au	71.8%
xyz	40.7%
co	53.0%
cz	76.1%
es	54.9%
eu	36.9%
us	86.9%
info	43.1%
cc	43.1%
ro	76.0%
online	42.4%
ai	79.9%
se	73.8%
site	48.0%

Full data table

TLD	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
com	325,550	67.6%	6.1%	17.8%	17.3%	96.8%	75.4%
org	29,741	63.8%	8.4%	1.6%	23.0%	95.2%	53.5%
net	26,249	26.8%	4.3%	27.7%	15.0%	89.2%	44.3%
ru	25,538	67.0%	5.4%	25.1%	3.1%	96.5%	62.1%
de	21,780	73.5%	5.6%	5.0%	15.9%	96.7%	62.0%
nl	16,877	79.4%	3.0%	32.2%	25.4%	97.0%	68.8%
co.uk	15,289	60.6%	2.4%	12.3%	13.0%	97.0%	75.4%
it	12,404	70.4%	3.0%	13.8%	11.3%	97.3%	68.3%
com.br	10,185	65.4%	7.0%	1.3%	18.8%	93.5%	67.5%
uk	10,182	83.9%	1.3%	87.7%	4.7%	98.0%	94.2%
fr	8,118	70.5%	2.3%	2.8%	9.9%	93.1%	70.6%
pl	6,464	74.5%	5.1%	1.2%	19.8%	96.1%	52.5%
io	6,386	48.9%	6.9%	4.3%	31.3%	87.6%	72.2%
in	5,857	63.9%	10.4%	2.0%	19.4%	90.7%	65.7%
ca	5,743	76.6%	3.9%	7.5%	14.4%	96.8%	59.5%
jp	5,275	61.9%	9.7%	4.1%	14.6%	86.8%	56.7%
com.au	4,919	73.5%	3.9%	9.5%	13.5%	97.8%	71.8%
xyz	4,503	35.5%	8.4%	12.5%	46.2%	77.4%	40.7%
co	4,207	58.5%	4.3%	17.5%	29.9%	91.0%	53.0%
cz	4,138	77.2%	4.8%	2.0%	19.4%	95.8%	76.1%
es	4,085	74.5%	3.5%	1.5%	9.8%	97.9%	54.9%
eu	3,826	54.8%	9.1%	1.6%	15.0%	95.6%	36.9%
us	3,787	55.7%	6.5%	15.1%	6.6%	96.5%	86.9%
info	3,566	48.4%	6.4%	6.7%	32.8%	89.6%	43.1%
cc	3,445	45.7%	8.1%	6.6%	45.7%	89.0%	43.1%
ro	3,357	77.4%	5.6%	0.9%	16.5%	96.3%	76.0%
online	3,266	40.2%	7.9%	7.9%	41.1%	85.3%	42.4%
ai	3,093	80.8%	4.5%	1.5%	24.3%	93.3%	79.9%
se	2,964	75.4%	4.7%	4.3%	15.5%	96.7%	73.8%
site	2,962	44.8%	10.9%	5.0%	43.5%	78.5%	48.0%

Showing the 30 best-covered TLDs by sample size.

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.