What does a site tell bots?
Look up any of the 1,000,000 sites on this month's Tranco top-million list: which AI crawlers its robots.txt names and blocks, whether it points to a sitemap, and whether it publishes an llms.txt.
Results describe the file each site served when it was captured for Common Crawl CC-MAIN-2026-39 (or fetched by our own crawl when Common Crawl missed it). llms.txt is checked for a random sample of the top 500,000 plus every site in the top 50,000.