Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-09-07). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall

Weighting: Raw Rank-weighted

robots.txt returns 200

data table

all

61.7%

absent (404)

data table

all

8.4%

disallow-all

data table

all

10.3%

allow-all

data table

all

25.7%

wildcard * UA

data table

all

89.8%

declares a sitemap

data table

all

60.7%

Full data table

Segment	sample	status 200	status 404	disallow all	allow all	wildcard ua	sitemap
all	740,113	61.7%	8.4%	10.3%	25.7%	89.8%	60.7%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.