Methodology & Open Data
Bot For You is an independent, neutral, data-first tracker. Numbers are measured, not modelled. This page documents how, pins the sources for reproducibility, and links the open-data exports.
Data sources
- Ranked panel: — a research-grade, manipulation-hardened list combining CrUX, Cloudflare Radar, Umbrella, Majestic, Farsight.
- robots.txt (primary): Common Crawl robotstxt WARC-shard streaming for crawl CC-MAIN-2026-30 — bulk file GETs, filtered to our Tranco set. Zero origin load. Gaps CC didn't capture (and the per-bot AI verdict backfill) are filled by our own polite self-crawl. The target population is the complete Tranco top-1M ranked panel for each published robots.txt snapshot.
- llms.txt (entirely self-crawled): collected by our own polite crawl engine — not Common Crawl, which doesn't capture /llms.txt. Probed across a rank-stratified cohort of the Tranco top-500k: 5,000 randomly selected domains from each of ten equal 50k rank bands. The separate top-50k census is reported on its own and never pooled at ten times the weight. Adoption is conditioned on a strict-200 live homepage; it is not a full-1M llms.txt crawl.
- Polite crawl engine: honest User-Agent + contact URL, ≤1 in-flight per host, global concurrency cap, backoff on 429/5xx, ≤5 redirects. Shared by the robots.txt gap/verdict fallback and the llms.txt cohort probe.
Published populations & denominators
A percentage is only meaningful with the population that qualifies for its denominator. Bot For You keeps these metric families separate:
| Metric family | Population / denominator | What is excluded |
|---|---|---|
| robots.txt HTTP status | All measured robots.txt fetch outcomes in the Tranco-ranked snapshot. | Nothing is silently converted to 404: throttle, block, and transport failures remain fetch errors. |
| robots.txt directives & AI targeting | Successfully fetched and parsed canonical robots.txt files. | Absent, unavailable, and unparsed files; these rates are not percentages of every ranked domain. |
| llms.txt adoption | Strict-200 live sites in the balanced top-500k panel; the headline is live-count-weighted across ten equal 50k bands. | 403 opt-outs from the measured denominator; the separate top-50k census from the representative estimate. |
HTTP Almanac covers a broader web population. Its figures are valuable external context and sanity checks, but are not treated as if they shared Bot For You's Tranco-ranked or liveness-gated denominators.
Popularity ranking lenses
Popularity is not one measurable quantity. Bot For You keeps Tranco as its ranked statistical panel, adds a CrUX membership view, then offers bounded Top-100 views through two complementary vendor sources. There is no synthetic Bot For You rank in this release, and source lenses are not treated as independent votes: CrUX, Cloudflare, and Majestic already contribute to Tranco.
- Tranco: the default research-consensus order. Its current configuration combines CrUX, Farsight DNSDB, Majestic, Cloudflare Radar, and Cisco Umbrella over a 30-day window. Source and methodology.
- Chrome UX Report (CrUX): monthly membership based on sufficient real-user Chrome traffic. CrUX publishes coarse rank buckets rather than a unique ordinal rank, so Bot For You shows member domains in Tranco order and does not invent an order within a bucket. Presence also reflects Chrome's eligibility and privacy thresholds, not all web traffic. Source and methodology.
- Cloudflare Radar: an exact ordered Top 100 reflecting popularity from Cloudflare's network vantage. Cloudflare exposes deeper Top-1M data as rank buckets, not exact ranks; Bot For You does not invent an order within those buckets. Source.
- Majestic Million: an exact ranking based on referring-subnet/link authority. The bounded view is attributed to Majestic; Bot For You neither exposes nor retains the full source list. Source and attribution.
Vendor hostnames are preserved exactly as ranked, then mapped to the registrable domain Bot For You measures. For example, docs.google.com is displayed as the source entry and associated with google.com. The ranking source, source snapshot, file-fetch source, and analytical crawl ID are separate provenance dimensions.
Corrections
Bot For You publishes measurement corrections rather than restating numbers silently.
- 2026-07-25 — robots.txt observations bound to the canonical file
-
Common Crawl's robotstxt dataset contains /robots.txt captures for arbitrary subdomains. Our ingest matched a capture to a measured domain by registrable domain alone, so an unrelated subdomain — at any path — could stand in for the domain itself. The clearest case: google.com was represented by sites.google.com/site/austinscrabble/robots.txt, an unrelated user-published Google Site, and therefore reported as HTTP 404.
An observation is now accepted only from a domain's own <domain>/robots.txt or www.<domain>/robots.txt. Representative selection was also made independent of the order shards are streamed in, and derived directive metrics can no longer outlive the response they were parsed from.
Scope: 177,821 of 662,174 Common Crawl robots observations (26.9%) were affected. Of those, 137,759 carried directive metrics belonging to a different site. Affected observations were removed, not overwritten; 121,709 were replaced with the domain's genuine canonical record, and domains with no canonical capture are now measured by our own polite crawl instead of carrying another site's result.
Effect on published figures (overall, unweighted): robots.txt HTTP 200 moved 68.3% → 71.9%, HTTP 404 12.0% → 8.4%, sitemap presence 59.5% → 63.5%, and effective posture open 91.7% → 93.0%. Disallow-all fell 4.5% → 2.8%: the mismatched subdomains were disproportionately blocking, which had overstated how much of the ranked web fully disallows crawling. llms.txt figures are unaffected — that dataset comes entirely from our own crawl.
Known limitation: recovered records restore status and provenance from the Common Crawl index but not file bodies, so a subset of corrected domains shows a status with file size and directive metrics blank — Common Crawl's index records the compressed size of its own archive record, which is not the file's true byte length, so we leave the field empty rather than publish a number that is not the size of the file. Both repopulate from real files at the next crawl. We would rather show a gap than a number we cannot attribute.
How we classify
- Standard vs uncommon directives are judged against RFC 9309 plus documented Google/Bing/Yandex extensions.
- Effective crawl posture encodes Google's documented status-history behavior (§3.3): 4xx (except 429) → crawl freely; persistent 5xx → cached for 30 days then crawl freely; 429 treated as a server error.
- llms.txt is reported as adoption + conformance, never consumption. Adoption is measured on the base /llms.txt only, and only for a 200 whose body is actually a document — many origins answer a missing path with 200. The other family variants are reported as counts, not rates — see the known limitation below.
- Tech inference from path signatures is a hypothesis, confidence-scored and corroborated with response headers — labelled as inference everywhere it appears, never as fact.
How we decide a file is really there
This is the single most consequential integrity rule on the site, because almost every llms.txt number divides by it. An origin that has no /llms.txt very often answers 200 anyway, so counting 200s counts fiction.
We originally tested whether the body looked like an HTML page, on the assumption that a missing path returns the site's homepage. We measured that assumption against 400 files fetched live for this purpose, and it was wrong: the HTML test caught only 5 of 266 non-files. Origins decline in far more ways than one, and 65 of them contained no HTML at all.
| what the path actually returned | seen in the 400-file sample |
|---|---|
| an HTML page or single-page-app shell | 91 |
| a bare stub — 404, OK, a lone newline, bot detected | ~50 |
| the site's robots.txt, served at this path | 15 |
| a database or framework error string | ~8 |
| an ads.txt, or compressed data sent as text | 2 |
| a genuine file | 134 |
So we invert the question. Instead of asking is this one of the known ways to fake a file — an unbounded list — we ask is a document present at all. A body counts as a file when it has a markdown heading, a section, several links, or a substantial block of prose with URLs. Non-files are substanceless, and that is what makes them detectable. On the same 400 files this identifies 265 of 266 non-files while wrongly rejecting none of the 134 real ones.
why not just require a heading Because it would be wrong far more often than the rule it replaced. A heading-only test flags every non-file — and destroys 130 of the 134 real ones. Plenty of legitimate publishers ship a link index or a structured prose document with no markdown heading at all. We tested that rule and rejected it; a stricter filter is not automatically a more honest one.
Two consequences we state plainly. First, this is not a judgement about file quality — conformance is measured separately, and a file can be genuine but poorly formed. Second, the rule changed after the crawl it is described here alongside: because we deliberately store no third-party file bodies, it cannot be applied retroactively, and it takes effect from the next full crawl onward. Figures published before that point used the older, more permissive HTML test and therefore overstate adoption slightly. We would rather say so than quietly restate history.
Known limitation: llms.txt variant denominators
The llms.txt self-crawl probes the family tiered — base /llms.txt first, stopping at the first conformant hit — and stores one record per domain (the winning variant). The base-variant adoption rate is sound provided the query filters to that variant; the other tiered variants have no usable denominator:
Correction, 2026-07 — base adoption revised down from 11.12% to 10.47%.
Storing one row per domain means the row holds whichever variant won the probe. Our adoption query read that lane without filtering to /llms.txt, so it answered “served some member of the family”. On the current cohort that folded 351 domains into the headline: 328 /ai.txt wins (of which exactly one is conformant), 20 /llms-full.txt, 3 /llms-ctx-full.txt. All 351 are real observations — they are simply not /llms.txt publishers, and /ai.txt is a count-only floor signal by design (see its row below). The corrected figure is 5,656 of 54,000 = 10.47%; the 351 are now reported separately with their probe condition rather than dropped. This page previously called the base rate “cheap and sound” while the query behind it did not implement that guarantee — the caveat was right, the code did not match it.
| Variant | Probed when | Reportable as |
|---|---|---|
| /llms.txt | every live domain in the cohort | adoption rate |
| /llms-full.txt | every live domain in the cohort — independently probed since the 2026-07 crawl (FR-18d), regardless of the /llms.txt outcome | adoption rate (see the llms-full.txt section) |
| /llms-ctx.txt /llms-ctx-full.txt |
only when /llms.txt returned a non-conformant 200 | count only |
| /ai.txt | every live domain, but recorded only when no conformant llms.txt variant won | count only (a floor) |
Concretely, for the count-only rows: a site publishing a good /llms.txt and a /llms-ctx.txt contributes only to the base row — the supplement is never requested. So those supplement counts are a floor over an unknown subset, not a rate over the cohort. We report them as raw counts and decline to publish a percentage.
update, 2026-07 /llms-full.txt is no longer count-only. It is now probed on every live cohort domain in its own lane, so it shares the base variant's denominator and gets a real adoption rate — the first non-base variant to earn one. That also makes the publishes-both / publishes-only-one breakdown measurable, which is the substantive question (does the full-context dump get maintained alongside the index?) the tiered design could not answer. The remaining supplements stay count-only by choice: they are far rarer, and making each one independently measurable costs a request per domain per variant.
Prior-art baselines (for sanity-checking)
We validate our numbers against the HTTP Archive Web Almanac 2025 (SEO chapter). Note the methodology difference: the Almanac crawls home pages of the broad web, NOT the Tranco top-1M — so expect our popular-site numbers to skew higher-presence.
| Metric | Almanac 2025 |
|---|---|
| robots.txt returns 200 | 84.9% |
| robots.txt 404 (absent) | 13.0% |
| wildcard user-agent present | 77.0% |
| files exceeding 500 KB | 0.1% |
| llms.txt adoption | 2.13% |
| GPTBot named | 4.5% |
Open data
Each snapshot is exportable to CSV, Parquet, and JSON via the pipeline botfy export command (the same pre-aggregated daily_snapshots table this site reads). Exports pin the Tranco list_id and Common Crawl crawl_id for citability.
Comparable strata — what each number counts
A percentage is only as honest as its denominator. In July a published figure moved from 72.05% to 61.56% purely because the denominator changed — the web had not moved, our sample had. Every rate below therefore states the population it was measured on, and the rule defining that population is printed in full beneath it, qualifications included.
Two things a reader should not assume. The rate column covers the whole stratum in this crawl, while before, after and Δ are computed only on the intersection — the domains observed in both crawls. And Δ values are not comparable across rows: each is measured on its own stratum's intersection, which is a different set of domains.
| stratum | n | rate | intersection | coverage | before | after | Δ |
|---|---|---|---|---|---|---|---|
| head_non_crux | 3,112 | 23.01% | 1,080 | 34.7% | — | — | no delta |
| crux_slice | 438,517 | 73.58% | 327,299 | 74.6% | 74.43% | 78.74% | +4.31 |
| determinate_intersection | 497,335 | 88.63% | 346,546 | 69.7% | 89.65% | 89.82% | +0.17 |
| cc_intersection | 611,885 | 72.83% | 447,245 | 73.1% | 71.63% | 75.69% | +4.06 |
| panel_intersection | 1,000,000 | 61.56% | 615,892 | 61.6% | 69.85% | 73.56% | +3.71 |
| full_observed | 1,000,000 | 61.56% | — | — | — | — | no delta |
Membership rules
These rules are the denominators. They are quoted in full because the qualification that matters sits at the end of each one.
head_non_crux — n=3,112
FR-38: panel domains ranked in the Tranco top 10,000 that are ABSENT from the Chrome UX Report. Named for what it IS, not what it mostly contains: the classifier is ~53% precise, so calling it an 'infrastructure stratum' would mislead every reader who quotes the number. DOMINANT composition IS infrastructure -- Tranco ranks by DNS QUERY VOLUME (Umbrella + Farsight), so CDN edge zones and nameserver domains rank high with zero human visitors (akamai.net, cloudfront.net, googlevideo.com, gtld-servers.net). But MEASURED: 29.6% of members serve a real HTML page and ~15% are unambiguous human-facing websites (nyt.com, pin.it, service-public.fr, discordapp.com). KNOWN CONFOUND: CrUX needs opted-in Chrome users, so absence partly measures Chrome under-sampling -- 63 members are google.*/blogspot.* ccTLDs for small markets. Validated on apex A-record presence, a signal independent of robots.txt: 46.8% no apex A vs 4.8% for CrUX members. Bounded at rank 10,000 because that is where the statistical regime ENDS (1-5k vs 5-10k p=0.238, 5-10k vs 10-25k p=0.0005); beyond the head, absence from CrUX means unpopular, not infrastructure.
crux_slice — n=438,517
Panel domains present in the Chrome UX Report -- the population HTTP Almanac actually measures (PRD §1004), and therefore the only same-population bridge to its ~85%/~13% baseline. CAVEAT THAT MUST TRAVEL WITH ANY IN/OUT COMPARISON: CrUX contains only sites with real Chrome traffic, so it structurally excludes dead domains. The raw +21.41pt availability gap is ~15.6pt liveness (a tautology) and only ~5.8pt genuine publishing behaviour among equally-alive origins. Not a pure rank proxy either -- the gap persists per rank band and steepens to +53pt in the top 10k, where the excluded population is DNS infrastructure. A SLICE, never the headline denominator (FR-37.3).
determinate_intersection — n=497,335
Domains where we KNOW the robots.txt answer in BOTH crawls: either we parsed the file, or the server returned 404. A 404 is not silence -- it means definitively NO RULES, which is a measurement, and it is exactly the "allow-all" case. Only 404 is promoted: a 403 is an explicit refusal to say and a 5xx or timeout is a failure to observe, so both are excluded rather than guessed at. Wider than parse_intersection by 36,604 domains whose answer we actually hold. IMPUTATION THAT MUST TRAVEL WITH ANY FIGURE ON THIS STRATUM: a 404 has no file, so it is counted as allow-all TRUE, disallow-all FALSE, no sitemap, no wildcard user-agent. Those four follow from the single observation that the file is absent -- they are derived, not invented -- but they ARE imputed, and metrics read off a file's contents (sitemap, wildcard user-agent) therefore sit lower here than on parse_intersection because the denominator includes domains that have no file to read: 58.40% vs 65.17% for sitemap, 84.47% vs 94.23% for wildcard. The DELTAS agree between the two strata; the LEVELS do not.
cc_intersection — n=611,885
Domains Common Crawl captured in BOTH crawls being compared. Same source, same domains -- the only apples-to-apples robots.txt comparison. Excludes the FR-4b self-crawl stratum entirely.
panel_intersection — n=1,000,000
Domains present in the Tranco panel for BOTH crawls (FR-35 panel_membership), regardless of how each was measured. Isolates real-web change from list turnover; the turnover itself is published separately as drift.
full_observed — n=1,000,000
Every domain measured in this crawl, any source. Breadth only. Membership changes between crawls, so this stratum has NO stable denominator and cannot produce a trend.
Reproducibility & separation
The pipeline (ingest → parse → infer → aggregate) and this website are decoupled halves sharing one database. The web app is strictly read-only and never crawls; a pipeline failure degrades freshness but never takes the site down. botfy.com's own robots.txt and llms.txt are exemplary and self-referential — we practice what we measure.