Data as of 2026-07-17 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 746,000 robots.txt fetches · 574,664 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-17). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

1	69.6%
2	67.2%
3	66.6%
4	68.7%
5	68.9%
6	68.6%
7	68.8%
8	69.2%
9	68.5%
10	66.9%

absent (404)

data table

1	12.4%
2	13.3%
3	13.2%
4	11.6%
5	11.2%
6	10.9%
7	10.7%
8	12.5%
9	12.7%
10	10.8%

disallow-all

data table

1	6.3%
2	5.0%
3	4.4%
4	4.0%
5	3.7%
6	4.6%
7	5.5%
8	3.6%
9	3.1%
10	4.3%

allow-all

data table

1	21.4%
2	23.4%
3	25.4%
4	24.5%
5	25.0%
6	25.8%
7	26.2%
8	28.1%
9	30.4%
10	27.9%

wildcard * UA

data table

1	94.6%
2	94.0%
3	93.8%
4	94.2%
5	94.3%
6	94.1%
7	93.8%
8	92.7%
9	93.0%
10	94.1%

declares a sitemap

data table

1	58.5%
2	57.8%
3	57.7%
4	59.7%
5	60.1%
6	60.8%
7	61.4%
8	59.7%
9	60.5%
10	59.9%

llms.txt adoption

data table

1	11.3%
2	12.9%
3	11.5%
4	9.9%
5	9.9%

Full data table

Rank decile	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap	llms adoption
1	72,071	69.6%	12.4%	6.3%	21.4%	94.6%	58.5%	11.3%
2	67,661	67.2%	13.3%	5.0%	23.4%	94.0%	57.8%	12.9%
3	60,090	66.6%	13.2%	4.4%	25.4%	93.8%	57.7%	11.5%
4	59,091	68.7%	11.6%	4.0%	24.5%	94.2%	59.7%	9.9%
5	60,083	68.9%	11.2%	3.7%	25.0%	94.3%	60.1%	9.9%
6	58,851	68.6%	10.9%	4.6%	25.8%	94.1%	60.8%	not probed
7	57,950	68.8%	10.7%	5.5%	26.2%	93.8%	61.4%	not probed
8	49,784	69.2%	12.5%	3.6%	28.1%	92.7%	59.7%	not probed
9	40,093	68.5%	12.7%	3.1%	30.4%	93.0%	60.5%	not probed
10	48,990	66.9%	10.8%	4.3%	27.9%	94.1%	59.9%	not probed

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.