Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
Compare sites
Put two measured sites beside each other. Botfy shows the evidence by crawler purpose and public file—without turning policy into a score or declaring a winner.
Start with a familiar pair
anthropic.com
and openai.com
Compare how two AI labs separate training, search, and live fetch.
Compare evidence →
cnn.com
and nytimes.com
See how two major publishers express crawler policy.
Compare evidence →
github.com
and stackoverflow.com
Contrast two widely used developer knowledge ecosystems.
Compare evidence →
amazon.com
and walmart.com
Compare crawler treatment at two large commerce sites.
Compare evidence →
shopify.com
and wordpress.com
Inspect two platforms that shape policies across many sites.
Compare evidence →
reddit.com
and wikipedia.org
Contrast two recognizable, community-produced information sources.
Compare evidence →