How the top million websites talk to bots
Every month we read the robots.txt and llms.txt files of the web's most popular sites. Here's what they say, what changed, and what it means for yours.
This month
The numbers this month
What changed since August
- Cloudflare's ready-made AI block disappeared on 16 September: of the 24,759 sites that had it last month, 96% still showed it when Common Crawl saved them before that day and 2% after.
- 486 sites in our llms.txt sample added one since last month; 85 took theirs down.
- Blocking of GPTBot fell 0.6 points on the 394,690 sites we read both months.
- Cloudflare's ready-made AI-blocking file dropped 0.8 points on those sites.
Behind this month's AI numbers
Cloudflare's ready-made AI block disappeared on 16 September 2026
A setting in Cloudflare's dashboard used to add a block for AI training bots to a site's robots.txt. Then it stopped: 6.49% of the files Common Crawl saved on 15 September 2026 carried the block, against 0.07% of those saved on 16 September 2026.
Saved before 16 September 202696.1%
Saved on or after2.2%
This month's crawl spans the change, so its GPTBot numbers mix files from before and after it. Next month's will be the first measured entirely after it. We read what robots.txt files say. Cloudflare can still turn bots away at its network edge, so fewer files blocking GPTBot does not mean more sites welcome it.
Changes compare the same sites in both months, measured the same way, so sites joining or leaving the top million can't fake a trend.
New here? Start with a guess
Three questions most people get wrong
1 of 3
What share of llms.txt files are the same Shopify template?
20.7%: 2,204 of 10,638 files share Shopify's store template, matching it section for section. See the files →
2 of 3
Who is more likely to publish an llms.txt?
About the same. 13.1% of the biggest 50,000 live sites vs 14.2% of those ranked 450,000–500,000. Every band lands between 11.9% and 14.9%. See why →
3 of 3
Sites that mention Apple's crawlers mostly block which one?
The AI-training opt-out, by far. 80.3% of sites that name Applebot-Extended block it; only 11.3% block Applebot. Same company, opposite answers →
Chapter 1 · The basics
Two small text files, two different jobs
robots.txt is a sign on the door. It tells crawlers which parts of a site they may not visit. It has been around since 1994, became an internet standard in 2022, and search engines and the big AI companies say they follow it.
llms.txt is a menu for AI tools. Proposed in 2024, it's a short Markdown file that points chatbots and agents to a site's most useful pages. Nobody is required to read it, and there's little evidence that AI tools do yet.
example.com/robots.txtsimplified
User-agent: * Disallow: /cart Sitemap: https://example.com/sitemap.xml # AI training bots: stay out User-agent: GPTBot Disallow: /
Rules, read by crawlers before they visit. The
Sitemap line points them to a full list of pages.example.com/llms.txtsimplified
# Example Ceramics > Handmade mugs and bowls, > shipped worldwide. ## Products - [Mugs](https://example.com/mugs) - [Bowls](https://example.com/bowls) ## Optional - [Blog](https://example.com/blog)
A guide, not a rule: a title, a one-line summary, then sections of links. Its sibling
llms-full.txt pastes in the full page text.Chapter 2 · Adoption
One in seven top sites has an llms.txt, and size doesn't matter
Across the top 500,000 sites, 13.7% of those that are up and running publish a real llms.txt (4,211 of 30,783 live sites in our sample). The biggest sites are no more likely to have one than sites ranked 400,000th.
Show the numbers
| Rank band | Live sites | With llms.txt | Share |
|---|---|---|---|
| 1–50,000 | 2,871 | 375 | 13.1% |
| 50,001–100,000 | 3,067 | 366 | 11.9% |
| 100,001–150,000 | 3,004 | 357 | 11.9% |
| 150,001–200,000 | 3,038 | 429 | 14.1% |
| 200,001–250,000 | 3,058 | 401 | 13.1% |
| 250,001–300,000 | 3,046 | 435 | 14.3% |
| 300,001–350,000 | 3,124 | 465 | 14.9% |
| 350,001–400,000 | 3,167 | 455 | 14.4% |
| 400,001–450,000 | 3,208 | 472 | 14.7% |
| 450,001–500,000 | 3,200 | 456 | 14.2% |
Other studies land far apart because they count different things. The HTTP Archive found llms.txt on 2.1% of home pages across the whole web in 2025. Ahrefs found 28% in its own panel of 137,000 sites. The top 500,000 sits in between.
What this means for you: if you're weighing an llms.txt, you aren't behind the big players. They haven't moved faster than anyone else.
Chapter 3 · Inside the files
One in five llms.txt files is the same Shopify template
Each dot below is one real llms.txt file, sized by its length and grouped by layout. forms one big cluster. like Yoast and Rank Math add another. Most of the rest have .
Hover or tap a dot to see the site. Click to keep it selected.
88lines in the median file
21links
5sections
98.4%start with a title
60.4%have a one-line summary
96.9%follow the llmstxt.org format
Good, typical, broken and extreme
cloudflare.com/llms.txt
Follows the format: a title, a one-line summary and organised sections, in a size a model can read at once.
10 B1 KB100 KB4 MB
github.com/llms.txt
Follows the format: a title, a one-line summary and organised sections, in a size a model can read at once.
10 B1 KB100 KB4 MB
intuit.com/llms.txt
Closest to the median of 88 lines among files that follow the format.
10 B1 KB100 KB4 MB
salesforce.com/llms.txt
Missing the title or the sections the format asks for.
no title, no sections
10 B1 KB100 KB4 MB
playstation.com/llms.txt
Missing the title or the sections the format asks for.
10 B1 KB100 KB4 MB
littlejohninc.com/llms.txt
The largest file that still follows the format.
10 B1 KB100 KB4 MB
atalanta.it/llms.txt
The smallest file that still follows the format.
10 B1 KB100 KB4 MB
Each example is picked by the rule shown on its card, not by taste, and screened so nothing unsuitable is showcased. Files can change after our crawl; links go to the live version.
Chapter 4 · AI crawlers
Sites block purposes, not companies
The same company runs several bots: one collects pages to train AI models, one builds an AI search index, one fetches a page when a user asks a chatbot a question. Sites treat them very differently.
Collects pages to train AIBuilds an AI search indexFetches a page for a chatbot user
Show the numbers
| Bot | Company | Purpose | Files naming it | Of those, block it |
|---|---|---|---|---|
| Applebot-Extended | Apple | Collects pages to train AI | 52,680 | 80.3% |
| Applebot | Apple | Builds an AI search index | 37,767 | 11.3% |
| Meta-ExternalAgent | Meta | Collects pages to train AI | 54,207 | 76.5% |
| Meta-ExternalFetcher | Meta | Fetches a page for a chatbot user | 7,983 | 51.6% |
| ClaudeBot | Anthropic | Collects pages to train AI | 70,491 | 71.5% |
| Claude-SearchBot | Anthropic | Builds an AI search index | 10,987 | 36.2% |
| Claude-User | Anthropic | Fetches a page for a chatbot user | 9,951 | 32.9% |
| GPTBot | OpenAI | Collects pages to train AI | 78,270 | 71.9% |
| ChatGPT-User | OpenAI | Fetches a page for a chatbot user | 34,242 | 48.3% |
| OAI-SearchBot | OpenAI | Builds an AI search index | 23,100 | 25.7% |
| PerplexityBot | Perplexity | Builds an AI search index | 33,061 | 41.7% |
| Perplexity-User | Perplexity | Fetches a page for a chatbot user | 10,053 | 32.3% |
And much of that blocking is one setting
Of the 56,270 sites that block GPTBot, 26,935 (47.9%) use the same file: one that Cloudflare added to a site's robots.txt when a setting in its dashboard is turned on. It blocks several AI bots at once. Cloudflare stopped adding it on 16 September 2026, so this count only includes files saved before then (what happened).
47.9%
52.1%
Cloudflare's ready-made file: 26,935Written by the site: 29,335
What this means for you: blocking AI is rarely one decision. Training, search and live answers are different trades, so decide each one. And if your site sits behind Cloudflare, check its AI settings and your live robots.txt: the ready-made block that used to appear there is gone from most sites since 16 September 2026, whatever Cloudflare still does at its network edge.
Chapter 5 · Mixed signals
707 sites roll out a welcome mat and bar the door
These sites publish an llms.txt, a file whose whole point is to help AI tools read the site, while their robots.txt blocks AI crawlers. The two files are usually owned by different teams, and nobody reads them side by side.
1,046
707
Publish llms.txt and let named AI crawlers in: 1,046Publish llms.txt but block at least one: 707
What this means for you: this is a five-minute check. Open both files and make sure they say the same thing.
Chapter 6 · Hall of fame
The strangest files we found
Come back in 10⁶² seconds
Crawl-delay: 1e62Asks crawlers to wait more than 10⁴⁴ times the age of the universe between visits. Google ignores Crawl-delay entirely.
Open the live file →27,653 doors marked "keep out"
Disallow: … × 27,653A 1.7 MB file of individual Disallow lines. Google stops reading robots.txt after 500 KB.
Open the live file →40 AI bots, one by one
User-agent: GPTBot
Disallow: /
… 39 moreTied for the most AI crawlers blocked by name in the top million.
Open the live file (jwz.org) →Everybody out
User-agent: *
Disallow: /The highest-ranked sites in Chrome's real-visit data that tell every crawler to stay away.
Open the live file (gstatic.com) →The 2.0 MB menu
6,443 lines · 6,432 linksThe largest file that still follows the format.
The 38 bytes menu
3 lines · 0 linksThe smallest file that still follows the format.
Chapter 7 · Your turn
What does a site actually say?
Explore on your own
Dig into the data
Check any site
Every site on this month's top-million list: its rules, which AI crawlers it blocks, and whether it has an llms.txt.
Download the numbers
This month's data pack: every figure on this page with the population behind it.
Every AI bot
A page per bot: who runs it, what it's for, how often it's blocked.
Example gallery
Real llms.txt and robots.txt files, filterable by good, typical, broken and extreme.
Platforms
What Shopify, WordPress, Wix and other platforms put in their sites' robots.txt, against every site.
Compare sites
Up to five sites side by side: which AI crawlers each one blocks, and who has an llms.txt.