robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco list ID not recorded for this snapshot · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-07-25). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
Top 1,00073.2%
Top 10,00077.3%
Top 100,00074.7%
Top 1,000,00071.7%
absent (404)
data table
Top 1,0005.0%
Top 10,0004.9%
Top 100,0007.3%
Top 1,000,0008.5%
disallow-all
data table
Top 1,0008.8%
Top 10,0003.0%
Top 100,0002.4%
Top 1,000,0002.7%
allow-all
data table
Top 1,00010.5%
Top 10,00015.0%
Top 100,00021.1%
Top 1,000,00026.4%
wildcard * UA
data table
Top 1,00097.1%
Top 10,00096.3%
Top 100,00095.2%
Top 1,000,00094.2%
declares a sitemap
data table
Top 1,00069.2%
Top 10,00067.7%
Top 100,00065.7%
Top 1,000,00063.4%

Full data table

Rank tier sample status 200status 404disallow allallow allwildcard uasitemap
Top 1,000 582 73.2% 5.0% 8.8% 10.5% 97.1% 69.2%
Top 10,000 5,816 77.3% 4.9% 3.0% 15.0% 96.3% 67.7%
Top 100,000 52,157 74.7% 7.3% 2.4% 21.1% 95.2% 65.7%
Top 1,000,000 432,470 71.7% 8.5% 2.7% 26.4% 94.2% 63.4%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.