Data as of 2026-07-17 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 746,000 robots.txt fetches · 574,664 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-17). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	68.4%
2	67.1%
3	66.6%
4	68.7%
5	68.9%
6	68.6%
7	68.8%
8	69.2%
9	68.5%
10	66.9%

absent (404)

data table

1	17.2%
2	13.4%
3	13.3%
4	11.6%
5	11.2%
6	10.9%
7	10.7%
8	12.4%
9	12.8%
10	10.8%

disallow-all

data table

1	19.4%
2	5.1%
3	4.4%
4	4.0%
5	3.7%
6	4.5%
7	5.6%
8	3.6%
9	3.1%
10	4.3%

allow-all

data table

1	18.9%
2	23.3%
3	25.6%
4	24.5%
5	25.0%
6	25.8%
7	26.2%
8	28.0%
9	30.3%
10	27.9%

wildcard * UA

data table

1	96.4%
2	94.0%
3	93.8%
4	94.2%
5	94.4%
6	94.1%
7	93.9%
8	92.7%
9	93.0%
10	94.1%

declares a sitemap

data table

1	60.3%
2	57.7%
3	57.6%
4	59.7%
5	60.0%
6	60.7%
7	61.4%
8	59.7%
9	60.5%
10	59.9%

llms.txt adoption

data table

1	14.6%
2	12.8%
3	11.6%
4	9.9%
5	10.0%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap	llms adoption
1	72,071	68.4%	17.2%	19.4%	18.9%	96.4%	60.3%	14.6%
2	67,661	67.1%	13.4%	5.1%	23.3%	94.0%	57.7%	12.8%
3	60,090	66.6%	13.3%	4.4%	25.6%	93.8%	57.6%	11.6%
4	59,091	68.7%	11.6%	4.0%	24.5%	94.2%	59.7%	9.9%
5	60,083	68.9%	11.2%	3.7%	25.0%	94.4%	60.0%	10.0%
6	58,851	68.6%	10.9%	4.5%	25.8%	94.1%	60.7%	not probed
7	57,950	68.8%	10.7%	5.6%	26.2%	93.9%	61.4%	not probed
8	49,784	69.2%	12.4%	3.6%	28.0%	92.7%	59.7%	not probed
9	40,093	68.5%	12.8%	3.1%	30.3%	93.0%	60.5%	not probed
10	48,990	66.9%	10.8%	4.3%	27.9%	94.1%	59.9%	not probed

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.