The state of robots.txt & llms.txt
Headline crawler-policy metrics across the Tranco top 1,000,000, sourced primarily from Common Crawl. All numbers are measured; prior-art baselines (Almanac 2025) are shown for context.
Effective crawl posture
Google's interpretation of robots.txt over status history (§3.3).
data table
| posture | share |
|---|---|
| open | 92.7% |
| blocked | 7.2% |
| cached | 0.1% |
| unknown | 0.0% |
open vs blocked vs cached vs unknown.
Published → conformant → read
We measure adoption & conformance — never consumption. The "~3% ever read" bar is Ahrefs prior art (gray), not ours.
data table
Of published files, 88.7% conform to the llmstxt.org spec.
AI-crawler targeting leaderboard
Share of parsed robots.txt files that name each AI user-agent in a directive — a targeting rate (the bot is addressed), not an allow/deny split. directive marks opt-out tokens (e.g. Google-Extended) that are not real request-log crawlers.
Sample this crawl: 687,794 robots.txt fetches, 491,025 parsed, 54,000 llms.txt probes.