robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-08-02 · robots.txt via Common Crawl CC-MAIN-2026-30 + our polite crawl · llms.txt via our polite crawl · Tranco list L5684 (2026-07-14) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,000,000 robots.txt fetches · 608,834 parsed · 55,770 llms.txt probes

Does robots.txt practice change with site popularity?

Headline metrics broken down by segment. Toggle the segmentation axis and the weighting. Rank-weighting (FR-23b) weights each domain by 1/Tranco-rank, so a restriction on a top site counts far more than one deep in the tail.

One crawl ingested (2026-08-02). Cross-crawl trend lines appear automatically when the next monthly crawl lands.

Segment: Rank tier Rank decile TLD Overall
Weighting: Raw Rank-weighted
robots.txt returns 200
data table
Top 1,00059.2%
Top 10,00060.2%
Top 100,00058.6%
Top 1,000,00060.9%
absent (404)
data table
Top 1,0005.4%
Top 10,0005.3%
Top 100,0007.2%
Top 1,000,0008.6%
disallow-all
data table
Top 1,00022.7%
Top 10,0003.9%
Top 100,0003.8%
Top 1,000,0007.6%
allow-all
data table
Top 1,00012.0%
Top 10,00015.2%
Top 100,00020.7%
Top 1,000,00024.4%
wildcard * UA
data table
Top 1,00098.2%
Top 10,00094.3%
Top 100,00092.4%
Top 1,000,00090.1%
declares a sitemap
data table
Top 1,00076.0%
Top 10,00063.9%
Top 100,00062.4%
Top 1,000,00060.5%

Full data table

Rank tier sample status 200status 404disallow allallow allwildcard uasitemap
Top 1,000 519 59.2% 5.4% 22.7% 12.0% 98.2% 76.0%
Top 10,000 5,384 60.2% 5.3% 3.9% 15.2% 94.3% 63.9%
Top 100,000 53,405 58.6% 7.2% 3.8% 20.7% 92.4% 62.4%
Top 1,000,000 549,526 60.9% 8.6% 7.6% 24.4% 90.1% 60.5%

Representative llms.txt prevalence uses a different, balanced ten-band cohort and is published separately on the llms.txt page. The generic observed-pool rate remains internal because its top-50k census would overweight the most popular band.