The state of robots.txt & llms.txt
We measured 746,000 robots.txt files across the Tranco top 1,000,000 — and probed 50,000 sites for llms.txt. Here is what the web has actually decided about crawlers, and what it means if you run a site.
1 — The baseline
Most of the web has not decided anything
— of parsed robots.txt files are effectively wide open, and 91.7% of the observed corpus is crawlable as far as Google is concerned. Blocking is the exception, not the rule. The default posture on the web is no posture.
data table
| posture | share |
|---|---|
| open | 91.7% |
| blocked | 8.3% |
| cached | 0.0% |
| unknown | 0.0% |
Google's effective crawl posture across the observed corpus.
What this means for you: if you have never edited your robots.txt, you are in the majority — and you are fully open to every crawler in this report, including the AI ones. That may be exactly what you want. It is worth it being a decision rather than a default.
How this was measured
Posture is Google's documented interpretation of a robots.txt over its recent status history (§3.3), not our own rule. It runs over the full observed corpus — every ranked domain we saw, including hosts that never answered — so it is not directly comparable to the CrUX-panel rates elsewhere on this site. Populations and denominators →
2 — Where the decisions are
Sites are not blocking companies — they are blocking purposes
Apple's AI opt-out token is blocked by around nine in ten of the files that name it. Apple's search crawler is blocked by fewer than one in ten. Same company, opposite answer. The same split shows up inside OpenAI, where the model-training crawler is blocked far more often than the fetcher that retrieves a page to answer a live user question.
Read the purpose column, not the operator column. The web is not taking a position on who these companies are. It is taking a position on what the request is for.
data table
| bot | operator | declared purpose | named by | blocked |
|---|---|---|---|---|
| applebot-extended | Apple | opt-out directive | 33,749 | 90.2% |
| bytespider | ByteDance | model training | 39,825 | 90.1% |
| amazonbot | Amazon | live assistant fetch | 38,547 | 90.0% |
| meta-externalagent | Meta | model training | 36,545 | 87.9% |
| ccbot | Common Crawl | model training | 43,337 | 86.7% |
| google-extended | opt-out directive | 39,351 | 82.0% | |
| claudebot | Anthropic | model training | 43,722 | 80.7% |
| petalbot | Huawei | AI search index | 13,645 | 80.4% |
| gptbot | OpenAI | model training | 49,557 | 79.9% |
| anthropic-ai | Anthropic | undocumented | 11,896 | 65.3% |
| chatgpt-user | OpenAI | live assistant fetch | 15,756 | 49.3% |
| perplexitybot | Perplexity | AI search index | 14,630 | 40.6% |
The most-named AI crawlers, ordered by the share of naming files that block them. Bots named by fewer than 100 files are excluded.
What this means for you: “should I block AI?” is the wrong shape of question, and blocking a company wholesale is usually not what you want. Training on your archive, fetching one page to answer a live question, and indexing you for an AI search result are three different trades, and most operators run a separate token for each. Decide per purpose, then check that your file actually says that. See which ones your site addresses →
How this was measured
The denominator is files that name the bot, not all files. A rate over
every file would conflate “chose not to block” with “never considered it”, which would make
every bot look near-universally allowed. Verdicts come from RFC 9309 §2.2.1 group matching:
the most specific matching user-agent group wins, falling back to *.
Two selection rules, stated because they change what you see. Bots named by fewer than 100 files are excluded — a block rate over a handful of files is noise, and ranked purely by rate it would lead the chart. The bots shown are then the most-named ones, ordered by block rate: choosing them by rate instead would produce an extreme-looking chart whatever the data said. The unabridged list, including the long tail, is on the crawler page.
3 — The clearest pattern in the data
The bigger the site, the more deliberate the file
Wide-open files get steadily more common as you go down the rank list, while the largest sites are far more likely to have written something specific. Popularity does not just correlate with traffic — it correlates with having made a choice.
data table
| rank tier | allow-all | disallow-all |
|---|---|---|
| Top 1,000 | 13.5% | 19.0% |
| Top 10,000 | 16.7% | 8.4% |
| Top 100,000 | 22.0% | 5.9% |
| Top 1,000,000 | 26.0% | 4.3% |
Share of parsed files by rank tier. Both series are percentages of parsed robots.txt files.
What this means for you: this is the beat to benchmark against. Find your rank tier, and you know what your peer group does — not what “the web” does, which is dominated by a long tail of sites that never configured anything. If you are a top-10k site running a default file, you are the outlier in your tier.
How this was measured
Rates are shares of successfully parsed robots.txt files in each tier, so absent and unfetchable files are excluded rather than counted as “not blocking”. Tiers are cumulative Tranco bands, so “Top 1,000,000” includes everything above it. Segment breakdowns →
4 — The emerging convention
llms.txt is published, but barely read
Adoption is real: 10.9% of live top-500k sites publish an llms.txt, and 89.6% of those files conform to the spec — publishers are doing it properly. But the best available prior art puts the share ever actually read by an AI system at roughly 3%.
data table
Published and conformant are ours. “Ever read” is Ahrefs prior art, not our measurement.
What this means for you: if you are deciding where to spend effort, this is not yet where the return is. Publishing an llms.txt is cheap and harmless, and it may matter later — but it is not currently a lever on how AI systems see your site. Your robots.txt is, today, the file that actually gets obeyed.
How this was measured
Adoption is live-count-weighted across ten equal 50k rank bands, conditioned on a strict-200 live homepage — not a full-1M crawl. We measure adoption and conformance, never consumption: the “ever read” figure is Ahrefs prior art over 137k sites and is labelled as theirs everywhere it appears, because we have no request logs and will not model what we cannot observe. Full llms.txt analysis →
5 — The contradiction
484 sites roll out a welcome mat and bar the door
These domains publish an llms.txt — a file whose entire purpose is to help LLMs read the site — while their robots.txt blocks AI crawlers. Both files are deliberate. Together they cancel out.
data table
| llms.txt | AI bots | domains |
|---|---|---|
| publishes | allowed | 478 |
| publishes | blocked | 484 |
| none | allowed | 13,224 |
| none | blocked | 52,988 |
Domains with a parsed robots.txt this crawl, crossed with genuine llms.txt presence.
What this means for you: these two files are usually owned by different people — marketing ships the llms.txt, infrastructure owns the robots.txt — and nobody reads them together. It is a five-minute audit and one of the few genuinely unambiguous mistakes in this whole dataset. Check your own site →
How this was measured
Restricted to domains with a parsed robots.txt this crawl and a genuine llms.txt — a 200 whose body is a real markdown document, not an HTML page or an error string served at that path. “Blocks AI bots” means at least one named AI crawler resolves to a blocking group under RFC 9309 matching. The full cross-tab →
6 — Now check your own
What does your site actually say?
Every figure above is an aggregate. The only one that affects you is your own file — which AI crawlers it names, which it blocks, and whether it agrees with itself.
Or try: openai.com · nytimes.com · github.com · shopify.com
Headline rates
The CrUX-listed panel of the Tranco top 1M — the same population the HTTP Almanac measures, so the baselines compare like with like.
The full observed corpus
Every ranked domain we observed, including those Common Crawl did not
capture and we fetched ourselves — among them a large population that never answered at
all. Dead and parked hosts serve a blanket Disallow: / far more often than
live ones, so these rates run lower on serving and higher on blocking. True of the corpus;
not a statement about the live web. The per-source breakdown is on
Methodology.
| metric | observed corpus | CrUX panel (above) |
|---|---|---|
| robots.txt returns 200 | 68.3% | — |
| absent (404) | 12.0% | — |
| disallow-all | 4.5% | — |
| allow-all | 25.5% | — |
| declares a sitemap | 59.5% | — |
AI-crawler targeting leaderboard
Share of parsed robots.txt files that name each AI user-agent in a directive — a targeting rate (the bot is addressed), not an allow/deny split. directive marks opt-out tokens (e.g. Google-Extended) that are not real request-log crawlers.
Robots.txt observations this crawl: 746,000 robots.txt fetches, 574,664 parsed. Balanced llms.txt panel: 50,000 selected across ten equal 50k rank bands, 28,783 live sites.