robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains · crawls: 2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

vagaro.com

This site disallows everything and blocks 22 of 34 named AI crawlers.

Tranco rank 15,746 Observed snapshot 2026-09-07 Google posture blocked Crawls observed 2

How this site compares

Each row is one measured trait, not a composite score. Rank peers are Ranks 10,001–100,000; the second comparison is the full observed Tranco panel. Populations differ by metric and are printed with every rate.

TraitThis site Ranks 10,001–100,000Observed Tranco panel
robots.txt served Yes 59.7% of 91,507 61.7% of 1,212,578
robots.txt returned 404 No 7.9% of 91,507 8.4% of 1,212,578
Blanket disallow Yes 4.0% of 54,147 10.3% of 740,113
Effectively open file No 22.6% of 54,147 25.7% of 740,113
Wildcard group Yes 92.1% of 54,147 89.8% of 740,113
Sitemap declared Yes 61.7% of 54,147 60.7% of 740,113
Over the RFC parse limit No 0.2% of 54,147 0.3% of 740,113
How these comparisons are measured

Status rates use canonical robots.txt observations. File traits use only parsed canonical files. A missing metric stays unavailable rather than becoming 0%. Rank bands are exclusive, so Ranks 10,001–100,000 does not include more popular bands.

Treatment by purpose

Declared purpose comes from the versioned agent registry. These are separate policy dimensions, not inputs to a score. An unnamed purpose is not automatically allowed; it inherits the wildcard rule described below.

Model training Blocked
AI2Bot · AI2Bot-Dolma · ByteSpider · CCBot · ClaudeBot · cohere-ai · cohere-training-data-crawler · Diffbot · FacebookBot · GPTBot · Kangaroo Bot · Meta-ExternalAgent · Omgili · Omgilibot · Timpibot · Webzio-Extended
AI search Mixed
Applebot · Claude-SearchBot · OAI-SearchBot · PerplexityBot · PetalBot · YouBot
Live assistant fetch Mixed
Amazonbot · ChatGPT-User · Claude-User · Claude-Web · DuckAssistBot · Meta-ExternalFetcher · MistralAI-User · Perplexity-User
Opt-out directives Blocked
Applebot-Extended · Google-Extended
Ambiguous / undocumented Blocked
anthropic-ai · Scrapy

Named agents

Every tracked agent this robots.txt names, grouped by operator, with the rule that applies under RFC 9309 group matching. A peer targeting rate means sites that name the token; it is not a block rate.

Anthropic 3 of 5 blocked
  • blocked Claude-Web live assistant fetch 4.2% of rank peers name it 63.9% block among 21,751 observed namers
  • blocked ClaudeBot model training 17.1% of rank peers name it 83.4% block among 117,092 observed namers
  • blocked anthropic-ai undocumented 5.6% of rank peers name it 62.5% block among 27,783 observed namers
  • partly blocked Claude-SearchBot AI search index 2.6% of rank peers name it 40.3% block among 10,183 observed namers
  • partly blocked Claude-User live assistant fetch 2.4% of rank peers name it 34.4% block among 8,980 observed namers
Webz.io 3 of 3 blocked
  • blocked Omgili model training 3.5% of rank peers name it 81.4% block among 16,162 observed namers
  • blocked Omgilibot model training 3.3% of rank peers name it 80.4% block among 16,263 observed namers
  • blocked Webzio-Extended model training 1.8% of rank peers name it 65.6% block among 6,666 observed namers
Allen Institute 2 of 2 blocked
  • blocked AI2Bot model training 2.0% of rank peers name it 63.9% block among 9,650 observed namers
  • blocked AI2Bot-Dolma model training 1.4% of rank peers name it 59.8% block among 6,542 observed namers
Cohere 2 of 2 blocked
  • blocked cohere-ai model training 4.1% of rank peers name it 67.2% block among 20,240 observed namers
  • blocked cohere-training-data-crawler model training 1.5% of rank peers name it 82.9% block among 4,332 observed namers
Meta 2 of 3 blocked
  • blocked FacebookBot model training 3.7% of rank peers name it 61.7% block among 16,667 observed namers
  • blocked Meta-ExternalAgent model training 14.7% of rank peers name it 87.6% block among 100,566 observed namers
  • partly blocked Meta-ExternalFetcher live assistant fetch 2.1% of rank peers name it 49.6% block among 7,913 observed namers
Apple 1 of 2 blocked
  • blocked Applebot-Extended opt-out directive directive 13.4% of rank peers name it 90.3% block among 98,492 observed namers
  • partly blocked Applebot AI search index 2.9% of rank peers name it 7.3% block among 63,771 observed namers
ByteDance 1 of 1 blocked
  • blocked ByteSpider model training 15.0% of rank peers name it 63.9% block among 154,750 observed namers
Common Crawl 1 of 1 blocked
  • blocked CCBot model training 16.2% of rank peers name it 63.3% block among 162,331 observed namers
Diffbot 1 of 1 blocked
  • blocked Diffbot model training 3.5% of rank peers name it 72.2% block among 14,397 observed namers
Google 1 of 1 blocked
  • blocked Google-Extended opt-out directive directive 15.6% of rank peers name it 84.9% block among 108,283 observed namers
Huawei 1 of 1 blocked
  • blocked PetalBot AI search index 4.6% of rank peers name it 26.9% block among 72,787 observed namers
Kangaroo LLM 1 of 1 blocked
  • blocked Kangaroo Bot model training 1.1% of rank peers name it 55.2% block among 4,930 observed namers
OpenAI 1 of 3 blocked
  • blocked GPTBot model training 18.9% of rank peers name it 82.8% block among 125,713 observed namers
  • partly blocked ChatGPT-User live assistant fetch 7.2% of rank peers name it 49.5% block among 35,854 observed namers
  • partly blocked OAI-SearchBot AI search index 5.2% of rank peers name it 26.9% block among 23,333 observed namers
Scrapy 1 of 1 blocked
  • blocked Scrapy undocumented 2.2% of rank peers name it 73.3% block among 10,043 observed namers
Timpi 1 of 1 blocked
  • blocked Timpibot model training 2.3% of rank peers name it 72.5% block among 9,957 observed namers
Amazon 0 of 1 blocked
  • partly blocked Amazonbot live assistant fetch 14.5% of rank peers name it 90.1% block among 107,461 observed namers
DuckDuckGo 0 of 1 blocked
  • partly blocked DuckAssistBot live assistant fetch 1.8% of rank peers name it 55.6% block among 7,406 observed namers
Mistral 0 of 1 blocked
  • partly blocked MistralAI-User live assistant fetch 1.3% of rank peers name it 62.3% block among 4,248 observed namers
Perplexity 0 of 2 blocked
  • partly blocked Perplexity-User live assistant fetch 2.5% of rank peers name it 33.1% block among 9,189 observed namers
  • partly blocked PerplexityBot AI search index 7.2% of rank peers name it 42.2% block among 33,828 observed namers
You.com 0 of 1 blocked
  • partly blocked YouBot AI search index 3.2% of rank peers name it 65.9% block among 16,910 observed namers
7 other tracked AI crawlers are not named individually. They are blocked — the wildcard (*) group disallows everything, so they are blocked without being named.

File evidence

/robots.txt Served and parsed Observed in Botfy's published crawl.
/llms.txt Not found when probed A related variant does not establish base-file adoption.

robots.txt details

Fetch HTTP 200 the file was retrieved and parsed
Size 5,408 bytes within the 500 KiB limit crawlers must honour
Wildcard group present a User-agent: * group governs any crawler without its own rules
Sitemap 1 declared declaring a sitemap helps crawlers discover pages

llms.txt observations

CrawlVariantStatusConformant SectionsLinksGenerator
CC-MAIN-2026-34 — 404 — — — —
CC-MAIN-2026-30 — 404 — — — —

Crawl posture history

Google's interpretation of this robots.txt over status history (§3.3).

blocked CC-MAIN-2026-34 2xx: robots.txt disallows all crawling (Disallow: /)
blocked CC-MAIN-2026-30 2xx: robots.txt disallows all crawling (Disallow: /)

Compare vagaro.com

Put this site's measured policy beside another site. Botfy keeps missing evidence visible and does not assign a score or winner.

Suggested: theice.com #15,745 free-daily-spins.com #15,747 openreview.net #15,744 mckesson.com #15,748

Nearby in Tranco

Close in the current popularity ranking. This does not imply a similar crawler policy.

theice.com #15,745 1 rank place away Compare profile → free-daily-spins.com #15,747 1 rank place away Compare profile → openreview.net #15,744 2 rank places away Compare profile → mckesson.com #15,748 2 rank places away Compare profile →

Live files on vagaro.com

The file as it exists on the host right now — may differ from what we measured. We link out rather than mirror (we store derived metrics, not raw bodies).

/robots.txt ↗ /llms.txt ↗ /llms-full.txt ↗ /ai.txt ↗

Raw observations

2 crawls, every source

A domain usually has more than one observation per crawl because we see it in the Common Crawl shard stream and fetch it ourselves. These are not duplicates — they are independent observations of the same file, and one is marked canonical. The rest of this page reads the canonical row.

CC-MAIN-2026-34

SourceRoleStatusSizeGroups AllowDisallowSitemaps
cc_shard
from the Common Crawl bulk shard stream		 canonical 200 5.3 KiB 5 0 48 1

CC-MAIN-2026-30

SourceRoleStatusSizeGroups AllowDisallowSitemaps
cc_shard
from the Common Crawl bulk shard stream		 canonical 200 2.3 KiB 4 2 47 0