Methodology & Open Data
Bot For You is an independent, neutral, data-first tracker. Numbers are measured, not modelled. This page documents how, pins the sources for reproducibility, and links the open-data exports.
Data sources
- Ranked panel: Tranco top-1,000,000 — a research-grade, manipulation-hardened list combining CrUX, Cloudflare Radar, Umbrella, Majestic, Farsight.
- robots.txt (primary): Common Crawl robotstxt WARC-shard streaming for crawl CC-MAIN-2026-25 — bulk file GETs, filtered to our Tranco set. Zero origin load. Gaps CC didn't capture (and the per-bot AI verdict backfill) are filled by our own polite self-crawl.
- llms.txt (entirely self-crawled): collected by our own polite crawl engine — not Common Crawl, which doesn't capture /llms.txt. Probed across a rank-stratified cohort of the Tranco top-500k (deciles 1–5), so llms.txt figures are of-live-sample, not the full 1M.
- Polite crawl engine: honest User-Agent + contact URL, ≤1 in-flight per host, global concurrency cap, backoff on 429/5xx, ≤5 redirects. Shared by the robots.txt gap/verdict fallback and the llms.txt cohort probe.
Popularity ranking lenses
Popularity is not one measurable quantity. Bot For You keeps Tranco as its ranked statistical panel, then offers bounded Top-100 views through two complementary source lenses. There is no synthetic Bot For You rank in this release, and vendor ranks are not treated as independent votes: Cloudflare and Majestic already contribute to Tranco.
- Tranco: the default research-consensus order. Its current configuration combines CrUX, Farsight DNSDB, Majestic, Cloudflare Radar, and Cisco Umbrella over a 30-day window. Source and methodology.
- Cloudflare Radar: an exact ordered Top 100 reflecting popularity from Cloudflare's network vantage. Cloudflare exposes deeper Top-1M data as rank buckets, not exact ranks; Bot For You does not invent an order within those buckets. Source.
- Majestic Million: an exact ranking based on referring-subnet/link authority. The bounded view is attributed to Majestic; Bot For You neither exposes nor retains the full source list. Source and attribution.
- Similarweb: its public list is a shallow proprietary traffic benchmark. Bot For You links to it for context but does not ingest or republish it without an appropriate license.
Vendor hostnames are preserved exactly as ranked, then mapped to the registrable domain Bot For You measures. For example, docs.google.com is displayed as the source entry and associated with google.com. The ranking source, source snapshot, file-fetch source, and analytical crawl ID are separate provenance dimensions.
Corrections
Bot For You publishes measurement corrections rather than restating numbers silently.
- 2026-07-25 — robots.txt observations bound to the canonical file
-
Common Crawl's robotstxt dataset contains /robots.txt captures for arbitrary subdomains. Our ingest matched a capture to a measured domain by registrable domain alone, so an unrelated subdomain — at any path — could stand in for the domain itself. The clearest case: google.com was represented by sites.google.com/site/austinscrabble/robots.txt, an unrelated user-published Google Site, and therefore reported as HTTP 404.
An observation is now accepted only from a domain's own <domain>/robots.txt or www.<domain>/robots.txt. Representative selection was also made independent of the order shards are streamed in, and derived directive metrics can no longer outlive the response they were parsed from.
Scope: 177,821 of 662,174 Common Crawl robots observations (26.9%) were affected. Of those, 137,759 carried directive metrics belonging to a different site. Affected observations were removed, not overwritten; 121,709 were replaced with the domain's genuine canonical record, and domains with no canonical capture are now measured by our own polite crawl instead of carrying another site's result.
Effect on published figures (overall, unweighted): robots.txt HTTP 200 moved 68.3% → 71.9%, HTTP 404 12.0% → 8.4%, sitemap presence 59.5% → 63.5%, and effective posture open 91.7% → 93.0%. Disallow-all fell 4.5% → 2.8%: the mismatched subdomains were disproportionately blocking, which had overstated how much of the ranked web fully disallows crawling. llms.txt figures are unaffected — that dataset comes entirely from our own crawl.
Known limitation: recovered records restore status and provenance from the Common Crawl index but not file bodies, so a subset of corrected domains shows a status with file size and directive metrics blank — Common Crawl's index records the compressed size of its own archive record, which is not the file's true byte length, so we leave the field empty rather than publish a number that is not the size of the file. Both repopulate from real files at the next crawl. We would rather show a gap than a number we cannot attribute.
How we classify
- Standard vs uncommon directives are judged against RFC 9309 plus documented Google/Bing/Yandex extensions.
- Effective crawl posture encodes Google's documented status-history behavior (§3.3): 4xx (except 429) → crawl freely; persistent 5xx → cached for 30 days then crawl freely; 429 treated as a server error.
- llms.txt is reported as adoption + conformance, never consumption. Adoption is measured on the base /llms.txt only. The other family variants are reported as counts, not rates — see the known limitation below.
- Tech inference from path signatures is a hypothesis, confidence-scored and corroborated with response headers — labelled as inference everywhere it appears, never as fact.
Known limitation: llms.txt variant denominators
The llms.txt self-crawl probes the family tiered — base /llms.txt first, stopping at the first conformant hit — and stores one record per domain (the winning variant). That makes the base-variant adoption rate cheap and sound, but it means the other variants have no usable denominator:
| Variant | Probed when | Reportable as |
|---|---|---|
| /llms.txt | every live domain in the cohort | adoption rate |
| /llms-full.txt /llms-ctx.txt /llms-ctx-full.txt |
only when /llms.txt returned a non-conformant 200 | count only |
| /ai.txt | every live domain, but recorded only when no conformant llms.txt variant won | count only (a floor) |
Concretely: a site publishing a good /llms.txt and a /llms-full.txt contributes only to the base row — the supplement is never requested. So the supplement counts are a floor over an unknown subset, not a rate over the cohort. We report them as raw counts and decline to publish a percentage. Closing this needs a crawl that records every variant independently (one row per domain-variant, ~5 requests per domain instead of ~2); until that runs, the honest number is the count.
Prior-art baselines (for sanity-checking)
We validate our numbers against the HTTP Archive Web Almanac 2025 (SEO chapter). Note the methodology difference: the Almanac crawls home pages of the broad web, NOT the Tranco top-1M — so expect our popular-site numbers to skew higher-presence.
| Metric | Almanac 2025 |
|---|---|
| robots.txt returns 200 | 84.9% |
| robots.txt 404 (absent) | 13.0% |
| wildcard user-agent present | 77.0% |
| files exceeding 500 KB | 0.1% |
| llms.txt adoption | 2.13% |
| GPTBot named | 4.5% |
Open data
Each snapshot is exportable to CSV, Parquet, and JSON via the pipeline botfy export command (the same pre-aggregated daily_snapshots table this site reads). Exports pin the Tranco list_id and Common Crawl crawl_id for citability.
Reproducibility & separation
The pipeline (ingest → parse → infer → aggregate) and this website are decoupled halves sharing one database. The web app is strictly read-only and never crawls; a pipeline failure degrades freshness but never takes the site down. botfy.com's own robots.txt and llms.txt are exemplary and self-referential — we practice what we measure.