Data as of 2026-08-02 · robots.txt via Common Crawl CC-MAIN-2026-30 + our polite crawl · llms.txt via our polite crawl · Tranco list L5684 (2026-07-14) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,000,000 robots.txt fetches · 608,834 parsed · 55,770 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-08-02). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	59.3%
2	59.6%
3	59.2%
4	61.0%
5	63.3%
6	61.2%
7	61.4%
8	65.5%
9	64.8%
10	59.4%

absent (404)

data table

1	5.7%
2	8.5%
3	9.2%
4	7.6%
5	7.9%
6	8.2%
7	8.7%
8	10.0%
9	9.8%
10	8.3%

disallow-all

data table

1	15.5%
2	3.9%
3	4.1%
4	12.9%
5	12.1%
6	8.4%
7	9.0%
8	7.8%
9	6.1%
10	17.0%

allow-all

data table

1	14.2%
2	23.1%
3	26.2%
4	21.2%
5	22.2%
6	24.3%
7	24.9%
8	27.1%
9	33.4%
10	24.7%

wildcard * UA

data table

1	96.4%
2	91.3%
3	89.2%
4	91.5%
5	92.0%
6	91.2%
7	90.8%
8	84.6%
9	82.9%
10	91.7%

declares a sitemap

data table

1	71.1%
2	60.3%
3	57.6%
4	65.1%
5	64.6%
6	62.6%
7	62.5%
8	54.4%
9	50.8%
10	64.6%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
1	59,308	59.3%	5.7%	15.5%	14.2%	96.4%	71.1%
2	59,247	59.6%	8.5%	3.9%	23.1%	91.3%	60.3%
3	58,423	59.2%	9.2%	4.1%	26.2%	89.2%	57.6%
4	60,436	61.0%	7.6%	12.9%	21.2%	91.5%	65.1%
5	62,606	63.3%	7.9%	12.1%	22.2%	92.0%	64.6%
6	60,487	61.2%	8.2%	8.4%	24.3%	91.2%	62.6%
7	60,681	61.4%	8.7%	9.0%	24.9%	90.8%	62.5%
8	64,810	65.5%	10.0%	7.8%	27.1%	84.6%	54.4%
9	64,094	64.8%	9.8%	6.1%	33.4%	82.9%	50.8%
10	58,742	59.4%	8.3%	17.0%	24.7%	91.7%	64.6%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.