robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-09-07). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
Top 1,00053.3%
Top 10,00060.7%
Top 100,00059.7%
Top 1,000,00061.8%
absent (404)
data table
Top 1,0004.6%
Top 10,0005.4%
Top 100,0007.9%
Top 1,000,0008.5%
disallow-all
data table
Top 1,0008.9%
Top 10,0003.7%
Top 100,0004.0%
Top 1,000,00010.8%
allow-all
data table
Top 1,00011.3%
Top 10,00016.4%
Top 100,00022.6%
Top 1,000,00026.0%
wildcard * UA
data table
Top 1,00095.1%
Top 10,00094.1%
Top 100,00092.1%
Top 1,000,00089.6%
declares a sitemap
data table
Top 1,00066.0%
Top 10,00065.6%
Top 100,00061.7%
Top 1,000,00060.6%

Full data table

Rank tier sample status 200status 404disallow allallow allwildcard uasitemap
Top 1,000 529 53.3% 4.6% 8.9% 11.3% 95.1% 66.0%
Top 10,000 5,437 60.7% 5.4% 3.7% 16.4% 94.1% 65.6%
Top 100,000 54,147 59.7% 7.9% 4.0% 22.6% 92.1% 61.7%
Top 1,000,000 680,000 61.8% 8.5% 10.8% 26.0% 89.6% 60.6%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.