Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-25). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	80.2%
2	72.2%
3	71.1%
4	72.7%
5	72.7%
6	72.0%
7	71.8%
8	71.7%
9	71.0%
10	69.5%

absent (404)

data table

1	5.7%
2	8.4%
3	9.0%
4	7.7%
5	7.7%
6	7.8%
7	8.0%
8	10.0%
9	10.5%
10	8.2%

disallow-all

data table

1	15.2%
2	2.1%
3	2.1%
4	2.0%
5	2.0%
6	3.3%
7	4.6%
8	2.7%
9	2.4%
10	3.0%

allow-all

data table

1	16.4%
2	23.2%
3	25.8%
4	24.8%
5	25.3%
6	26.3%
7	26.7%
8	28.3%
9	30.3%
10	28.9%

wildcard * UA

data table

1	97.4%
2	94.7%
3	94.2%
4	94.7%
5	94.6%
6	94.4%
7	94.3%
8	92.7%
9	93.2%
10	94.6%

declares a sitemap

data table

1	72.4%
2	63.7%
3	62.4%
4	63.9%
5	63.9%
6	64.1%
7	64.5%
8	61.8%
9	62.6%
10	63.0%

llms.txt adoption

data table

1	14.6%
2	12.8%
3	11.6%
4	9.9%
5	10.0%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap	llms adoption
1	58,555	80.2%	5.7%	15.2%	16.4%	97.4%	72.4%	14.6%
2	55,689	72.2%	8.4%	2.1%	23.2%	94.7%	63.7%	12.8%
3	50,345	71.1%	9.0%	2.1%	25.8%	94.2%	62.4%	11.6%
4	49,770	72.7%	7.7%	2.0%	24.8%	94.7%	63.9%	9.9%
5	51,381	72.7%	7.7%	2.0%	25.3%	94.6%	63.9%	10.0%
6	51,007	72.0%	7.8%	3.3%	26.3%	94.4%	64.1%	not probed
7	50,785	71.8%	8.0%	4.6%	26.7%	94.3%	64.5%	not probed
8	44,360	71.7%	10.0%	2.7%	28.3%	92.7%	61.8%	not probed
9	36,234	71.0%	10.5%	2.4%	30.3%	93.2%	62.6%	not probed
10	42,899	69.5%	8.2%	3.0%	28.9%	94.6%	63.0%	not probed

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.