2osb.com
This site serves a robots.txt and blocks 8 of 8 named AI crawlers.
How this site compares
Each row is one measured trait, not a composite score. Rank peers are Ranks 10,001–100,000; the second comparison is the full observed Tranco panel. Populations differ by metric and are printed with every rate.
| Trait | This site | Ranks 10,001–100,000 | Observed Tranco panel |
|---|---|---|---|
| robots.txt served | Yes | 59.7% of 91,507 | 61.7% of 1,212,578 |
| robots.txt returned 404 | No | 7.9% of 91,507 | 8.4% of 1,212,578 |
| Blanket disallow | No | 4.0% of 54,147 | 10.3% of 740,113 |
| Effectively open file | Yes | 22.6% of 54,147 | 25.7% of 740,113 |
| Wildcard group | Yes | 92.1% of 54,147 | 89.8% of 740,113 |
| Sitemap declared | No | 61.7% of 54,147 | 60.7% of 740,113 |
| Over the RFC parse limit | No | 0.2% of 54,147 | 0.3% of 740,113 |
How these comparisons are measured
Status rates use canonical robots.txt observations. File traits use only parsed canonical files. A missing metric stays unavailable rather than becoming 0%. Rank bands are exclusive, so Ranks 10,001–100,000 does not include more popular bands.
Treatment by purpose
Declared purpose comes from the versioned agent registry. These are separate policy dimensions, not inputs to a score. An unnamed purpose is not automatically allowed; it inherits the wildcard rule described below.
Named agents
Every tracked agent this robots.txt names, grouped by operator, with the rule that applies under RFC 9309 group matching. A peer targeting rate means sites that name the token; it is not a block rate.
- blocked Amazonbot live assistant fetch 14.5% of rank peers name it 90.1% block among 107,461 observed namers
- blocked ClaudeBot model training 17.1% of rank peers name it 83.4% block among 117,092 observed namers
- blocked Applebot-Extended opt-out directive directive 13.4% of rank peers name it 90.3% block among 98,492 observed namers
- blocked Bytespider model training 15.0% of rank peers name it 63.9% block among 154,750 observed namers
- blocked CCBot model training 16.2% of rank peers name it 63.3% block among 162,331 observed namers
- blocked Google-Extended opt-out directive directive 15.6% of rank peers name it 84.9% block among 108,283 observed namers
- blocked meta-externalagent model training 14.7% of rank peers name it 87.6% block among 100,566 observed namers
- blocked GPTBot model training 18.9% of rank peers name it 82.8% block among 125,713 observed namers
File evidence
robots.txt details
llms.txt observations
| Crawl | Variant | Status | Conformant | Sections | Links | Generator |
|---|---|---|---|---|---|---|
| CC-MAIN-2026-34 | — | 404 | — | — | — | — |
| CC-MAIN-2026-30 | — | 404 | — | — | — | — |
Crawl posture history
Google's interpretation of this robots.txt over status history (§3.3).
Compare 2osb.com
Put this site's measured policy beside another site. Botfy keeps missing evidence visible and does not assign a score or winner.
Live files on 2osb.com
The file as it exists on the host right now — may differ from what we measured. We link out rather than mirror (we store derived metrics, not raw bodies).
/robots.txt ↗ /llms.txt ↗ /llms-full.txt ↗ /ai.txt ↗
Raw observations
2 crawls, every source
A domain usually has more than one observation per crawl because we see it in the Common Crawl shard stream and fetch it ourselves. These are not duplicates — they are independent observations of the same file, and one is marked canonical. The rest of this page reads the canonical row.
CC-MAIN-2026-34
| Source | Role | Status | Size | Groups | Allow | Disallow | Sitemaps |
|---|---|---|---|---|---|---|---|
| self_crawl fetched directly by our own polite crawler |
canonical | 200 | 1.8 KiB | 10 | 1 | 9 | 0 |
CC-MAIN-2026-30
| Source | Role | Status | Size | Groups | Allow | Disallow | Sitemaps |
|---|---|---|---|---|---|---|---|
| self_crawl fetched directly by our own polite crawler |
canonical | 200 | 1.8 KiB | 10 | 1 | 9 | 0 |