robots.txt & llms.txt across the Tranco top 1M

Domain lookup

Try: openai.com · nytimes.com · github.com · shopify.com

Enter a domain to see which AI crawlers it blocks, whether its robots.txt is healthy, and how its crawl posture has changed.