Data as of 2026-08-02 · robots.txt via Common Crawl CC-MAIN-2026-30 + our polite crawl · llms.txt via our polite crawl · Tranco list L5684 (2026-07-14) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,000,000 robots.txt fetches · 608,834 parsed · 55,770 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-08-02). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	59.9%
2	59.9%
3	59.1%
4	61.1%
5	63.3%
6	61.2%
7	61.4%
8	65.6%
9	64.8%
10	59.4%

absent (404)

data table

1	7.4%
2	8.6%
3	9.1%
4	7.6%
5	7.9%
6	8.2%
7	8.7%
8	10.1%
9	9.7%
10	8.3%

disallow-all

data table

1	3.9%
2	3.9%
3	4.1%
4	13.0%
5	12.0%
6	8.4%
7	9.0%
8	7.7%
9	6.2%
10	17.0%

allow-all

data table

1	21.5%
2	23.0%
3	25.9%
4	21.2%
5	22.2%
6	24.3%
7	25.0%
8	27.2%
9	33.6%
10	24.7%

wildcard * UA

data table

1	92.0%
2	91.3%
3	89.5%
4	91.5%
5	91.9%
6	91.2%
7	90.8%
8	84.5%
9	82.8%
10	91.7%

declares a sitemap

data table

1	61.5%
2	60.4%
3	58.0%
4	65.1%
5	64.5%
6	62.6%
7	62.5%
8	54.3%
9	50.8%
10	64.6%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
1	59,308	59.9%	7.4%	3.9%	21.5%	92.0%	61.5%
2	59,247	59.9%	8.6%	3.9%	23.0%	91.3%	60.4%
3	58,423	59.1%	9.1%	4.1%	25.9%	89.5%	58.0%
4	60,436	61.1%	7.6%	13.0%	21.2%	91.5%	65.1%
5	62,606	63.3%	7.9%	12.0%	22.2%	91.9%	64.5%
6	60,487	61.2%	8.2%	8.4%	24.3%	91.2%	62.6%
7	60,681	61.4%	8.7%	9.0%	25.0%	90.8%	62.5%
8	64,810	65.6%	10.1%	7.7%	27.2%	84.5%	54.3%
9	64,094	64.8%	9.7%	6.2%	33.6%	82.8%	50.8%
10	58,742	59.4%	8.3%	17.0%	24.7%	91.7%	64.6%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.