Methodology & Open Data
Bot For You is an independent, neutral, data-first tracker. Numbers are measured, not modelled. This page documents how, pins the sources for reproducibility, and links the open-data exports.
Data sources
- Ranked panel: — a research-grade, manipulation-hardened list combining CrUX, Cloudflare Radar, Umbrella, Majestic, Farsight.
- robots.txt (primary): Common Crawl robotstxt WARC-shard streaming for crawl None — bulk file GETs, filtered to our Tranco set. Zero origin load. Gaps CC didn't capture (and the per-bot AI verdict backfill) are filled by our own polite self-crawl. The target population is the complete Tranco top-1M ranked panel for each published robots.txt snapshot.
- llms.txt (entirely self-crawled): collected by our own polite crawl engine — not Common Crawl, which doesn't capture /llms.txt. Probed across a rank-stratified cohort of the Tranco top-500k: 5,000 randomly selected domains from each of ten equal 50k rank bands. The separate top-50k census is reported on its own and never pooled at ten times the weight. Adoption is conditioned on a strict-200 live homepage; it is not a full-1M llms.txt crawl.
- Polite crawl engine: honest User-Agent + contact URL, ≤1 in-flight per host, global concurrency cap, backoff on 429/5xx, ≤5 redirects. Shared by the robots.txt gap/verdict fallback and the llms.txt cohort probe.
Published populations & denominators
A percentage is only meaningful with the population that qualifies for its denominator. Bot For You keeps these metric families separate:
| Metric family | Population / denominator | What is excluded |
|---|---|---|
| robots.txt HTTP status | All measured robots.txt fetch outcomes in the Tranco-ranked snapshot. | Nothing is silently converted to 404: throttle, block, and transport failures remain fetch errors. |
| robots.txt directives & AI targeting | Successfully fetched and parsed canonical robots.txt files. | Absent, unavailable, and unparsed files; these rates are not percentages of every ranked domain. |
| llms.txt adoption | Strict-200 live sites in the balanced top-500k panel; the headline is live-count-weighted across ten equal 50k bands. | 403 opt-outs from the measured denominator; the separate top-50k census from the representative estimate. |
HTTP Almanac covers a broader web population. Its figures are valuable external context and sanity checks, but are not treated as if they shared Bot For You's Tranco-ranked or liveness-gated denominators.
Popularity ranking lenses
Popularity is not one measurable quantity. Bot For You keeps Tranco as its ranked statistical panel, adds a CrUX membership view, then offers bounded Top-100 views through two complementary vendor sources. There is no synthetic Bot For You rank in this release, and source lenses are not treated as independent votes: CrUX, Cloudflare, and Majestic already contribute to Tranco.
- Tranco: the default research-consensus order. Its current configuration combines CrUX, Farsight DNSDB, Majestic, Cloudflare Radar, and Cisco Umbrella over a 30-day window. Source and methodology.
- Chrome UX Report (CrUX): monthly membership based on sufficient real-user Chrome traffic. CrUX publishes coarse rank buckets rather than a unique ordinal rank, so Bot For You shows member domains in Tranco order and does not invent an order within a bucket. Presence also reflects Chrome's eligibility and privacy thresholds, not all web traffic. Source and methodology.
- Cloudflare Radar: an exact ordered Top 100 reflecting popularity from Cloudflare's network vantage. Cloudflare exposes deeper Top-1M data as rank buckets, not exact ranks; Bot For You does not invent an order within those buckets. Source.
- Majestic Million: an exact ranking based on referring-subnet/link authority. The bounded view is attributed to Majestic; Bot For You neither exposes nor retains the full source list. Source and attribution.
Vendor hostnames are preserved exactly as ranked, then mapped to the registrable domain Bot For You measures. For example, docs.google.com is displayed as the source entry and associated with google.com. The ranking source, source snapshot, file-fetch source, and analytical crawl ID are separate provenance dimensions.
Corrections
Bot For You publishes measurement corrections rather than restating numbers silently.
- 2026-07-25 — robots.txt observations bound to the canonical file
-
Common Crawl's robotstxt dataset contains /robots.txt captures for arbitrary subdomains. Our ingest matched a capture to a measured domain by registrable domain alone, so an unrelated subdomain — at any path — could stand in for the domain itself. The clearest case: google.com was represented by sites.google.com/site/austinscrabble/robots.txt, an unrelated user-published Google Site, and therefore reported as HTTP 404.
An observation is now accepted only from a domain's own <domain>/robots.txt or www.<domain>/robots.txt. Representative selection was also made independent of the order shards are streamed in, and derived directive metrics can no longer outlive the response they were parsed from.
Scope: 177,821 of 662,174 Common Crawl robots observations (26.9%) were affected. Of those, 137,759 carried directive metrics belonging to a different site. Affected observations were removed, not overwritten; 121,709 were replaced with the domain's genuine canonical record, and domains with no canonical capture are now measured by our own polite crawl instead of carrying another site's result.
Effect on published figures (overall, unweighted): robots.txt HTTP 200 moved 68.3% → 71.9%, HTTP 404 12.0% → 8.4%, sitemap presence 59.5% → 63.5%, and effective posture open 91.7% → 93.0%. Disallow-all fell 4.5% → 2.8%: the mismatched subdomains were disproportionately blocking, which had overstated how much of the ranked web fully disallows crawling. llms.txt figures are unaffected — that dataset comes entirely from our own crawl.
Known limitation: recovered records restore status and provenance from the Common Crawl index but not file bodies, so a subset of corrected domains shows a status with file size and directive metrics blank — Common Crawl's index records the compressed size of its own archive record, which is not the file's true byte length, so we leave the field empty rather than publish a number that is not the size of the file. Both repopulate from real files at the next crawl. We would rather show a gap than a number we cannot attribute.
How we classify
- Standard vs uncommon directives are judged against RFC 9309 plus documented Google/Bing/Yandex extensions.
- Effective crawl posture encodes Google's documented status-history behavior (§3.3): 4xx (except 429) → crawl freely; persistent 5xx → cached for 30 days then crawl freely; 429 treated as a server error.
- llms.txt is reported as adoption + conformance, never consumption. Adoption is measured on the base /llms.txt only, and only for a 200 whose body is actually a document — many origins answer a missing path with 200. The other family variants are reported as counts, not rates — see the known limitation below.
- Tech inference from path signatures is a hypothesis, confidence-scored and corroborated with response headers — labelled as inference everywhere it appears, never as fact.
How we decide a file is really there
This is the single most consequential integrity rule on the site, because almost every llms.txt number divides by it. An origin that has no /llms.txt very often answers 200 anyway, so counting 200s counts fiction.
We originally tested whether the body looked like an HTML page, on the assumption that a missing path returns the site's homepage. We measured that assumption against 400 files fetched live for this purpose, and it was wrong: the HTML test caught only 5 of 266 non-files. Origins decline in far more ways than one, and 65 of them contained no HTML at all.
| what the path actually returned | seen in the 400-file sample |
|---|---|
| an HTML page or single-page-app shell | 91 |
| a bare stub — 404, OK, a lone newline, bot detected | ~50 |
| the site's robots.txt, served at this path | 15 |
| a database or framework error string | ~8 |
| an ads.txt, or compressed data sent as text | 2 |
| a genuine file | 134 |
So we invert the question. Instead of asking is this one of the known ways to fake a file — an unbounded list — we ask is a document present at all. A body counts as a file when it has a markdown heading, a section, several links, or a substantial block of prose with URLs. Non-files are substanceless, and that is what makes them detectable. On the same 400 files this identifies 265 of 266 non-files while wrongly rejecting none of the 134 real ones.
why not just require a heading Because it would be wrong far more often than the rule it replaced. A heading-only test flags every non-file — and destroys 130 of the 134 real ones. Plenty of legitimate publishers ship a link index or a structured prose document with no markdown heading at all. We tested that rule and rejected it; a stricter filter is not automatically a more honest one.
Two consequences we state plainly. First, this is not a judgement about file quality — conformance is measured separately, and a file can be genuine but poorly formed. Second, the rule changed after the crawl it is described here alongside: because we deliberately store no third-party file bodies, it cannot be applied retroactively, and it takes effect from the next full crawl onward. Figures published before that point used the older, more permissive HTML test and therefore overstate adoption slightly. We would rather say so than quietly restate history.
Known limitation: llms.txt variant denominators
The llms.txt self-crawl probes the family tiered — base /llms.txt first, stopping at the first conformant hit — and stores one record per domain (the winning variant). The base-variant adoption rate is sound provided the query filters to that variant; the other tiered variants have no usable denominator:
Correction, 2026-07 — base adoption revised down from 11.12% to 10.47%.
Storing one row per domain means the row holds whichever variant won the probe. Our adoption query read that lane without filtering to /llms.txt, so it answered “served some member of the family”. On the current cohort that folded 351 domains into the headline: 328 /ai.txt wins (of which exactly one is conformant), 20 /llms-full.txt, 3 /llms-ctx-full.txt. All 351 are real observations — they are simply not /llms.txt publishers, and /ai.txt is a count-only floor signal by design (see its row below). The corrected figure is 5,656 of 54,000 = 10.47%; the 351 are now reported separately with their probe condition rather than dropped. This page previously called the base rate “cheap and sound” while the query behind it did not implement that guarantee — the caveat was right, the code did not match it.
| Variant | Probed when | Reportable as |
|---|---|---|
| /llms.txt | every live domain in the cohort | adoption rate |
| /llms-full.txt | every live domain in the cohort — independently probed since the 2026-07 crawl (FR-18d), regardless of the /llms.txt outcome | adoption rate (see the llms-full.txt section) |
| /llms-ctx.txt /llms-ctx-full.txt |
only when /llms.txt returned a non-conformant 200 | count only |
| /ai.txt | every live domain, but recorded only when no conformant llms.txt variant won | count only (a floor) |
Concretely, for the count-only rows: a site publishing a good /llms.txt and a /llms-ctx.txt contributes only to the base row — the supplement is never requested. So those supplement counts are a floor over an unknown subset, not a rate over the cohort. We report them as raw counts and decline to publish a percentage.
update, 2026-07 /llms-full.txt is no longer count-only. It is now probed on every live cohort domain in its own lane, so it shares the base variant's denominator and gets a real adoption rate — the first non-base variant to earn one. That also makes the publishes-both / publishes-only-one breakdown measurable, which is the substantive question (does the full-context dump get maintained alongside the index?) the tiered design could not answer. The remaining supplements stay count-only by choice: they are far rarer, and making each one independently measurable costs a request per domain per variant.
Prior-art baselines (for sanity-checking)
We validate our numbers against the HTTP Archive Web Almanac 2025 (SEO chapter). Note the methodology difference: the Almanac crawls home pages of the broad web, NOT the Tranco top-1M — so expect our popular-site numbers to skew higher-presence.
| Metric | Almanac 2025 |
|---|---|
| robots.txt returns 200 | 84.9% |
| robots.txt 404 (absent) | 13.0% |
| wildcard user-agent present | 77.0% |
| files exceeding 500 KB | 0.1% |
| llms.txt adoption | 2.13% |
| GPTBot named | 4.5% |
Open data
Each snapshot is exportable to CSV, Parquet, and JSON via the pipeline botfy export command (the same pre-aggregated daily_snapshots table this site reads). Exports pin the Tranco list_id and Common Crawl crawl_id for citability.
Reproducibility & separation
The pipeline (ingest → parse → infer → aggregate) and this website are decoupled halves sharing one database. The web app is strictly read-only and never crawls; a pipeline failure degrades freshness but never takes the site down. botfy.com's own robots.txt and llms.txt are exemplary and self-referential — we practice what we measure.