Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-09-07). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	62.1%
2	59.2%
3	59.5%
4	62.0%
5	62.3%
6	61.7%
7	62.1%
8	63.9%
9	63.3%
10	61.4%

absent (404)

data table

1	5.8%
2	8.7%
3	9.0%
4	8.2%
5	7.6%
6	7.8%
7	7.9%
8	9.4%
9	9.3%
10	8.2%

disallow-all

data table

1	16.8%
2	3.7%
3	3.9%
4	9.7%
5	12.9%
6	15.2%
7	16.0%
8	10.8%
9	7.5%
10	14.9%

allow-all

data table

1	16.1%
2	25.5%
3	27.1%
4	22.8%
5	23.0%
6	22.9%
7	23.1%
8	26.6%
9	32.1%
10	28.4%

wildcard * UA

data table

1	96.4%
2	91.0%
3	89.2%
4	92.2%
5	92.6%
6	91.7%
7	92.0%
8	85.5%
9	84.1%
10	89.7%

declares a sitemap

data table

1	70.9%
2	58.5%
3	57.1%
4	63.4%
5	65.1%
6	64.7%
7	65.2%
8	56.1%
9	53.7%
10	62.1%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
1	60,113	62.1%	5.8%	16.8%	16.1%	96.4%	70.9%
2	60,989	59.2%	8.7%	3.7%	25.5%	91.0%	58.5%
3	63,440	59.5%	9.0%	3.9%	27.1%	89.2%	57.1%
4	67,362	62.0%	8.2%	9.7%	22.8%	92.2%	63.4%
5	70,534	62.3%	7.6%	12.9%	23.0%	92.6%	65.1%
6	73,017	61.7%	7.8%	15.2%	22.9%	91.7%	64.7%
7	77,927	62.1%	7.9%	16.0%	23.1%	92.0%	65.2%
8	78,233	63.9%	9.4%	10.8%	26.6%	85.5%	56.1%
9	93,935	63.3%	9.3%	7.5%	32.1%	84.1%	53.7%
10	94,563	61.4%	8.2%	14.9%	28.4%	89.7%	62.1%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.