Data as of 2026-07-17 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 746,000 robots.txt fetches · 574,664 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-17). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

all

68.3%

absent (404)

data table

all

16.6%

disallow-all

data table

all

17.7%

allow-all

data table

all

19.6%

wildcard * UA

data table

all

96.1%

declares a sitemap

data table

all

60.1%

llms.txt adoption

data table

all

14.6%

Full data table

Segment	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap	llms adoption
all	574,664	68.3%	16.6%	17.7%	19.6%	96.1%	60.1%	14.6%

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.