robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 · crawls: 2026-07-25 · 2026-07-17 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Segments

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-25). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
top_1k73.2%
top_10k77.3%
top_100k74.7%
top_1m71.7%
absent (404)
data table
top_1k5.0%
top_10k4.9%
top_100k7.3%
top_1m8.5%
disallow-all
data table
top_1k8.8%
top_10k3.0%
top_100k2.4%
top_1m2.7%
allow-all
data table
top_1k10.5%
top_10k15.0%
top_100k21.1%
top_1m26.4%
wildcard * UA
data table
top_1k97.1%
top_10k96.3%
top_100k95.2%
top_1m94.2%
declares a sitemap
data table
top_1k69.2%
top_10k67.7%
top_100k65.7%
top_1m63.4%
llms.txt adoption
data table
top_1k15.5%
top_10k11.7%
top_100k11.1%
top_1m10.9%

Full data table

Rank tier sample status 200status 404disallow allallow allwildcard uasitemapllms adoption
top_1k 582 73.2% 5.0% 8.8% 10.5% 97.1% 69.2% 15.5%
top_10k 5,816 77.3% 4.9% 3.0% 15.0% 96.3% 67.7% 11.7%
top_100k 52,157 74.7% 7.3% 2.4% 21.1% 95.2% 65.7% 11.1%
top_1m 432,470 71.7% 8.5% 2.7% 26.4% 94.2% 63.4% 10.9%

llms.txt not probed cells: this crawl's cohort covers the top 500k (rank deciles 1–5); the rest join in a later crawl.