Do sites block AI companies or particular crawlers?

Particular crawlers. Measured across parsed robots.txt files in the Tranco top 1,000,000 on 2026-07-25, the split follows declared purpose rather than operator: a company's model-training crawler is blocked far more often than the same company's search or live-fetch crawler.

Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco list ID not recorded for this snapshot · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Which AI crawlers do websites block?

Q: Which AI crawlers do websites block?

This page ranks every tracked AI crawler by how often robots.txt files in the Tranco top 1,000,000 name it, with measured block rates where resolved verdicts exist, as of 2026-07-25.

Share of parsed robots.txt files across the top 1M that name each AI user-agent in a directive — a targeting rate (the bot is addressed by a rule), not an allow-vs-deny split (the data records that a bot is addressed, not the decision). Bars are tinted by vendor.

Weighting: Raw Rank-weighted

Sort: By vendor By rate

directive tokens (Google-Extended, Applebot-Extended) are robots.txt opt-out directives, NOT crawlers that appear in request logs. We model declared opt-out separately from actual-traffic presence (PRD FR-17b).

Related: which sites let AI crawlers in while blocking Google — the inverse configuration, and rarer than it looks.

Do sites that publish llms.txt also block AI crawlers?

The flagship cross-tab: does a site's llms.txt (a welcome mat for LLMs) agree with its robots.txt AI-crawler rules? The mixed cell — publishes an llms.txt and blocks AI crawlers — is a genuine contradiction.

data table

quadrant	domains
welcoming (llms.txt + allows AI)	631
mixed (llms.txt + blocks AI)	560
blocking (blocks AI, no llms.txt)	56276
indifferent (neither)	14862

Domains with a parsed robots.txt this crawl, crossed with genuine llms.txt presence (a 200 whose body is a real document). The mixed cell is the headline contradiction.

Which AI crawlers are named most often?

OpenAI 33.2% Anthropic 31.6% Meta 21.2% Common Crawl 18.3% Google 18.0% Apple 17.7%

ai2bot · 82.5% blocked 1.9%

ai2bot-dolma · 88.2% blocked 1.4%

semanticscholarbot · 84.6% blocked 0.1%

amazonbot · 89.1% blocked 16.3%

claudebot · 79.0% blocked 18.5%

anthropic-ai · 63.8% blocked 5.4%

claude-web · 64.1% blocked 4.1%

claude-searchbot · 42.2% blocked 1.9%

claude-user · 36.9% blocked 1.7%

applebot-extended directive · 88.9% blocked 14.1%

applebot · 31.8% blocked 3.6%

bytespider · 89.7% blocked 16.6%

tiktokspider · 82.5% blocked 0.8%

cohere-ai · 69.5% blocked 3.5%

cohere-training-data-crawler · 86.3% blocked 1.0%

ccbot · 85.8% blocked 18.3%

deepseekbot · 74.5% blocked 0.5%

diffbot · 82.1% blocked 3.0%

duckassistbot · 55.9% blocked 1.3%

google-extended directive · 80.2% blocked 16.6%

googleother · 47.0% blocked 1.4%

petalbot · 80.6% blocked 6.1%

img2dataset · 89.2% blocked 1.5%

kangaroo bot · 88.1% blocked 1.0%

meta-externalagent · 87.1% blocked 15.8%

facebookbot · 68.2% blocked 3.7%

meta-externalfetcher · 65.0% blocked 1.7%

mistralai-user · 66.1% blocked 0.9%

gptbot · 78.3% blocked 21.1%

chatgpt-user · 47.8% blocked 7.2%

oai-searchbot · 32.4% blocked 4.9%

panscient.com · 95.5% blocked 0.4%

perplexitybot · 39.3% blocked 6.7%

perplexity-user · 34.1% blocked 1.8%

scrapy · 88.2% blocked 2.3%

timpibot · 88.6% blocked 2.1%

imgproxy · 98.0% blocked 0.1%

omgili · 91.0% blocked 2.7%

omgilibot · 89.1% blocked 2.7%

webzio-extended · 91.6% blocked 1.5%

youbot · 67.0% blocked 2.9%

Prior-art anchor (HTTP Almanac 2025, broad-web home-page census): GPTBot named on ~4.5% of files, up from ~2.9% the prior year — a near-doubling. Our top-1M numbers skew higher because popular sites have more (and more complex) robots.txt.