indeed.com
This site serves a robots.txt and blocks 5 of 31 named AI crawlers.
How this site compares
Each row is one measured trait, not a composite score. Rank peers are Ranks 1–1,000; the second comparison is the full observed Tranco panel. Populations differ by metric and are printed with every rate.
| Trait | This site | Ranks 1–1,000 | Observed Tranco panel |
|---|---|---|---|
| robots.txt served | Yes | 53.3% of 1,000 | 61.7% of 1,212,578 |
| robots.txt returned 404 | No | 4.6% of 1,000 | 8.4% of 1,212,578 |
| Blanket disallow | No | 8.9% of 529 | 10.3% of 740,113 |
| Effectively open file | No | 11.3% of 529 | 25.7% of 740,113 |
| Wildcard group | Yes | 95.1% of 529 | 89.8% of 740,113 |
| Sitemap declared | No | 66.0% of 529 | 60.7% of 740,113 |
| Over the RFC parse limit | No | 0.8% of 529 | 0.3% of 740,113 |
How these comparisons are measured
Status rates use canonical robots.txt observations. File traits use only parsed canonical files. A missing metric stays unavailable rather than becoming 0%. Rank bands are exclusive, so Ranks 1–1,000 does not include more popular bands.
Treatment by purpose
Declared purpose comes from the versioned agent registry. These are separate policy dimensions, not inputs to a score. An unnamed purpose is not automatically allowed; it inherits the wildcard rule described below.
Named agents
Every tracked agent this robots.txt names, grouped by operator, with the rule that applies under RFC 9309 group matching. A peer targeting rate means sites that name the token; it is not a block rate.
- blocked Webzio-Extended model training 7.8% of rank peers name it 65.6% block among 6,666 observed namers
- blocked omgili model training 12.1% of rank peers name it 81.4% block among 16,162 observed namers
- blocked omgilibot model training 9.1% of rank peers name it 80.4% block among 16,263 observed namers
- blocked GoogleOther undocumented 6.6% of rank peers name it 34.6% block among 7,220 observed namers
- partly blocked Google-Extended opt-out directive directive 22.1% of rank peers name it 84.9% block among 108,283 observed namers
- blocked Scrapy undocumented 9.5% of rank peers name it 73.3% block among 10,043 observed namers
- partly blocked AI2Bot model training 7.6% of rank peers name it 63.9% block among 9,650 observed namers
- partly blocked AmazonBot live assistant fetch 13.0% of rank peers name it 90.1% block among 107,461 observed namers
- partly blocked Claude-SearchBot AI search index 12.3% of rank peers name it 40.3% block among 10,183 observed namers
- partly blocked Claude-User live assistant fetch 13.0% of rank peers name it 34.4% block among 8,980 observed namers
- partly blocked ClaudeBot model training 23.3% of rank peers name it 83.4% block among 117,092 observed namers
- partly blocked anthropic-ai undocumented 12.7% of rank peers name it 62.5% block among 27,783 observed namers
- partly blocked Applebot-Extended opt-out directive directive 15.7% of rank peers name it 90.3% block among 98,492 observed namers
- partly blocked Bytespider model training 19.7% of rank peers name it 63.9% block among 154,750 observed namers
- partly blocked cohere-training-data-crawler model training 7.2% of rank peers name it 82.9% block among 4,332 observed namers
- partly blocked CCBot model training 21.7% of rank peers name it 63.3% block among 162,331 observed namers
- partly blocked DeepSeekBot model training 5.7% of rank peers name it 79.7% block among 2,304 observed namers
- partly blocked Diffbot model training 12.5% of rank peers name it 72.2% block among 14,397 observed namers
- partly blocked DuckAssistBot live assistant fetch 9.6% of rank peers name it 55.6% block among 7,406 observed namers
- partly blocked PetalBot AI search index 11.9% of rank peers name it 26.9% block among 72,787 observed namers
- partly blocked FacebookBot model training 10.0% of rank peers name it 61.7% block among 16,667 observed namers
- partly blocked Meta-ExternalAgent model training 17.8% of rank peers name it 87.6% block among 100,566 observed namers
- partly blocked Meta-ExternalFetcher live assistant fetch 8.7% of rank peers name it 49.6% block among 7,913 observed namers
- partly blocked ChatGPT-User live assistant fetch 17.8% of rank peers name it 49.5% block among 35,854 observed namers
- partly blocked GPTBot model training 25.0% of rank peers name it 82.8% block among 125,713 observed namers
- partly blocked OAI-SearchBot AI search index 16.3% of rank peers name it 26.9% block among 23,333 observed namers
- partly blocked Perplexity-User live assistant fetch 11.2% of rank peers name it 33.1% block among 9,189 observed namers
- partly blocked PerplexityBot AI search index 21.4% of rank peers name it 42.2% block among 33,828 observed namers
- partly blocked Timpibot model training 8.9% of rank peers name it 72.5% block among 9,957 observed namers
- partly blocked YouBot AI search index 10.8% of rank peers name it 65.9% block among 16,910 observed namers
- partly blocked img2dataset model training 6.0% of rank peers name it 62.1% block among 6,942 observed namers
File evidence
robots.txt details
Crawl posture history
Google's interpretation of this robots.txt over status history (§3.3).
Compare indeed.com
Put this site's measured policy beside another site. Botfy keeps missing evidence visible and does not assign a score or winner.
Live files on indeed.com
The file as it exists on the host right now — may differ from what we measured. We link out rather than mirror (we store derived metrics, not raw bodies).
/robots.txt ↗ /llms.txt ↗ /llms-full.txt ↗ /ai.txt ↗
Raw observations
2 crawls, every source
A domain usually has more than one observation per crawl because we see it in the Common Crawl shard stream and fetch it ourselves. These are not duplicates — they are independent observations of the same file, and one is marked canonical. The rest of this page reads the canonical row.
CC-MAIN-2026-34
| Source | Role | Status | Size | Groups | Allow | Disallow | Sitemaps |
|---|---|---|---|---|---|---|---|
| cc_shard from the Common Crawl bulk shard stream |
canonical | 200 | 12.8 KiB | 9 | 37 | 468 | 0 |
CC-MAIN-2026-30
| Source | Role | Status | Size | Groups | Allow | Disallow | Sitemaps |
|---|---|---|---|---|---|---|---|
| cc_shard from the Common Crawl bulk shard stream |
canonical | 200 | 12.2 KiB | 9 | 7 | 468 | 0 |
Inferred tech hypothesis
Derived from robots.txt patterns, not observed directly — labelled as a hypothesis because it is inference, not measurement.