Every AI bot
Who sites keep out, and who they let in
Each row is an AI crawler named in at least 5,000 robots.txt files on this month's Tranco top 1M. Click a bot for who runs it, what it's for and which sites block it.
| Bot | Runs it | Purpose | Files naming it | Of those, block it | Change, same sites |
|---|---|---|---|---|---|
| CCBot | Common Crawl | Collects pages to train AI | 90,568 | 60.7% | −0.6 pts |
| Bytespider | ByteDance | Collects pages to train AI | 84,014 | 62.0% | −0.5 pts |
| GPTBot | OpenAI | Collects pages to train AI | 78,270 | 71.9% | −0.6 pts |
| ClaudeBot | Anthropic | Collects pages to train AI | 70,491 | 71.5% | −0.6 pts |
| Google-Extended opt-out name | Collects pages to train AI | 62,050 | 72.3% | −0.6 pts | |
| Amazonbot | Amazon | Fetches a page for a chatbot user | 60,353 | 81.6% | −0.6 pts |
| Meta-ExternalAgent | Meta | Collects pages to train AI | 54,207 | 76.5% | −0.6 pts |
| Applebot-Extended opt-out name | Apple | Collects pages to train AI | 52,680 | 80.3% | −0.6 pts |
| petalbot | Huawei | Builds an AI search index | 46,881 | 40.4% | +0.1 pts |
| Applebot | Apple | Builds an AI search index | 37,767 | 11.3% | +0.0 pts |
| ChatGPT-User | OpenAI | Fetches a page for a chatbot user | 34,242 | 48.3% | +0.0 pts |
| PerplexityBot | Perplexity | Builds an AI search index | 33,061 | 41.7% | +0.0 pts |
| anthropic-ai | Anthropic | Purpose not documented | 28,018 | 62.2% | +0.1 pts |
| OAI-SearchBot | OpenAI | Builds an AI search index | 23,100 | 25.7% | +0.0 pts |
| cohere-ai | Cohere | Collects pages to train AI | 21,203 | 66.9% | +0.1 pts |
| claude-web | Anthropic | Fetches a page for a chatbot user | 21,076 | 65.7% | +0.1 pts |
| omgili | Webz.io | Collects pages to train AI | 17,288 | 79.6% | +0.1 pts |
| youbot | You.com | Builds an AI search index | 17,268 | 64.4% | +0.1 pts |
| omgilibot | Webz.io | Collects pages to train AI | 16,565 | 77.7% | +0.1 pts |
| facebookbot | Meta | Collects pages to train AI | 16,505 | 60.2% | +0.0 pts |
| diffbot | Diffbot | Collects pages to train AI | 14,575 | 74.6% | +0.1 pts |
| Claude-SearchBot | Anthropic | Builds an AI search index | 10,987 | 36.2% | +0.0 pts |
| ai2bot | Allen Institute | Collects pages to train AI | 10,135 | 61.2% | +0.0 pts |
| Perplexity-User | Perplexity | Fetches a page for a chatbot user | 10,053 | 32.3% | +0.0 pts |
| Claude-User | Anthropic | Fetches a page for a chatbot user | 9,951 | 32.9% | +0.0 pts |
| timpibot | Timpi | Collects pages to train AI | 9,731 | 74.1% | +0.1 pts |
| scrapy | Scrapy | Purpose not documented | 9,396 | 74.5% | +0.0 pts |
| googleother | Purpose not documented | 8,410 | 38.3% | +0.0 pts | |
| duckassistbot | DuckDuckGo | Fetches a page for a chatbot user | 8,117 | 49.3% | +0.0 pts |
| Meta-ExternalFetcher | Meta | Fetches a page for a chatbot user | 7,983 | 51.6% | +0.0 pts |
| img2dataset | img2dataset | Collects pages to train AI | 7,369 | 57.4% | +0.0 pts |
| ai2bot-dolma | Allen Institute | Collects pages to train AI | 7,022 | 55.0% | +0.0 pts |
| webzio-extended | Webz.io | Collects pages to train AI | 6,507 | 68.4% | +0.0 pts |
| cohere-training-data-crawler | Cohere | Collects pages to train AI | 5,298 | 71.3% | +0.0 pts |
"Block it" means the rules that apply to that bot shut it out of the whole site. "Change, same sites" is the shift in the share of the same 394,690 sites, read by Common Crawl in both CC-MAIN-2026-34 and CC-MAIN-2026-39, that block the bot by name. Bots in Cloudflare's ready-made file fell this month because Cloudflare stopped adding it on 16 September 2026 (what happened). Robots.txt files we could read, one per site on this month's Tranco top 1M.