Every AI bot · Anthropic

anthropic-ai

A name associated with Anthropic whose purpose hasn't been clearly documented.

In robots.txt: User-agent: anthropic-ai

Named by4.7%of readable robots.txt files on the top 1M (28,018 of 600,979)

Of those, block it62.2%shut it out of the whole site (17,419 of 28,018)

Same sites, since CC-MAIN-2026-34+0.1 ptsshare of the same 394,690 sites that block it by name: 2.1% → 2.2%

Via Cloudflare's file2.2%of its blocks come from Cloudflare's ready-made robots.txt (391 of 17,419)

62.2%

22.0%

15.8%

Block the whole site: 17,419Block some paths: 6,177Name it but allow it: 4,422

Every robots.txt file on the top 1M that names anthropic-ai, by what the rules that apply to it say.

Anthropic's other bots

Sites often treat one company's bots differently, depending on what each is for.

Bot	Purpose	Files naming it	Of those, block it
ClaudeBot	Collects pages to train AI	70,491	71.5%
claude-web	Fetches a page for a chatbot user	21,076	65.7%
Claude-SearchBot	Builds an AI search index	10,987	36.2%
Claude-User	Fetches a page for a chatbot user	9,951	32.9%

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

Block some paths

Name it and allow it

Lists skip sites we don't showcase (adult content).

What this means for you: its purpose isn't documented, so what blocking it changes is unclear. Decide each bot on what it's for, not who runs it.