robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-09-07 · robots.txt via Common Crawl CC-MAIN-2026-34 + our polite crawl · llms.txt via our polite crawl · Tranco list XN67N (2026-09-03T22:00:02.533886) · 1,000,000 panel domains 1,212,578 robots.txt fetches · 740,113 parsed · 56,498 llms.txt probes

Two public profiles · no score or winner

nouvelobs.com and outsideonline.com

nouvelobs.com #7,356 outsideonline.com #7,355

Measured takeaway

nouvelobs.com blocks live assistant fetchers; outsideonline.com has mixed rules for live assistant fetchers.

nouvelobs.com blocks ChatGPT-User; outsideonline.com partly blocks ChatGPT-User.

Measured in Botfy's 2026-09-07 publication.

At a glance

A difference appears only when both sites have comparable evidence. Unavailable or unresolved observations remain incomplete.

Evidencenouvelobs.com outsideonline.com
robots.txt Served and parsed Served and parsed
Google crawl posture Open Open
Unnamed crawler fallback Unrestricted no wildcard (*) group exists, so no rule applies to them at all Partly Restricted they inherit the wildcard (*) group's rules
Base /llms.txt Not found when probed Not found when probed

Treatment by purpose

Only resolved blocked, allowed, or mixed states count as a policy agreement or difference. Unnamed agents still inherit each site's wildcard rules.

Declared purposenouvelobs.com outsideonline.com
Model training Blocked Blocked
AI search Mixed Mixed
Live assistant fetch Blocked Mixed
Opt-out directives Blocked Blocked
Ambiguous / undocumented Blocked Mixed

Named agents

Measured differences appear first. “Not explicitly addressed” is not the same as allowed; the site's wildcard fallback governs that crawler.

Agentnouvelobs.com outsideonline.com
PerplexityBot Perplexity Blocked Partly blocked
ChatGPT-User OpenAI Blocked Partly blocked
GPTBot OpenAI Blocked Blocked
ClaudeBot Anthropic Blocked Blocked
CCBot Common Crawl Blocked Blocked
Bytespider ByteDance Blocked Blocked
FacebookBot Meta Blocked Blocked
cohere-ai Cohere Blocked Blocked
Diffbot Diffbot Blocked Blocked
omgili Webz.io Blocked Blocked
omgilibot Webz.io Blocked Blocked
Applebot Apple Partly blocked Partly blocked
21 additional tracked agents
Agentnouvelobs.com outsideonline.com
anthropic-ai BlockedBlocked
meta-externalagent Not explicitly addressedBlocked
cohere-training-data-crawler Not explicitly addressedBlocked
DeepseekBot Not explicitly addressedBlocked
Timpibot Not explicitly addressedBlocked
AI2Bot Not explicitly addressedBlocked
AI2Bot-Dolma Not explicitly addressedBlocked
webzio-extended Not explicitly addressedBlocked
OAI-SearchBot Not explicitly addressedPartly blocked
Claude-SearchBot Not explicitly addressedPartly blocked
YouBot BlockedNot explicitly addressed
Petalbot Not explicitly addressedBlocked
Claude-User Not explicitly addressedPartly blocked
Claude-Web BlockedNot explicitly addressed
Perplexity-User Not explicitly addressedPartly blocked
Amazonbot Not explicitly addressedBlocked
meta-externalfetcher Not explicitly addressedPartly blocked
MistralAI-User Not explicitly addressedPartly blocked
Google-Extended BlockedNot explicitly addressed
Applebot-Extended Not explicitly addressedBlocked
GoogleOther Not explicitly addressedPartly blocked

How each site compares with its own rank peers

The sites may belong to different rank bands. Each percentage keeps its own population and denominator; Botfy does not subtract rates from unlike cohorts.

Traitnouvelobs.com outsideonline.com
Wildcard group No 94.1% Ranks 1,001–10,000 · n=5,437 Yes 94.1% Ranks 1,001–10,000 · n=5,437
Effectively open file No 16.4% Ranks 1,001–10,000 · n=5,437 No 16.4% Ranks 1,001–10,000 · n=5,437
Sitemap declared Yes 65.6% Ranks 1,001–10,000 · n=5,437 Yes 65.6% Ranks 1,001–10,000 · n=5,437
robots.txt served Yes 60.7% Ranks 1,001–10,000 · n=9,037 Yes 60.7% Ranks 1,001–10,000 · n=9,037
robots.txt returned 404 No 5.4% Ranks 1,001–10,000 · n=9,037 No 5.4% Ranks 1,001–10,000 · n=9,037
Over the RFC parse limit No 0.2% Ranks 1,001–10,000 · n=5,437 No 0.2% Ranks 1,001–10,000 · n=5,437
Blanket disallow No 3.7% Ranks 1,001–10,000 · n=5,437 No 3.7% Ranks 1,001–10,000 · n=5,437
How this comparison is measured

Both sites use Botfy's latest published observations. A missing fetch, unresolved agent verdict, and measured absence are different states. Peer percentages use each site's exclusive Tranco rank band and print their own denominator. Populations, sources, and limits →

Keep comparing

Each suggestion replaces one side using an editorial collection or current rank proximity. Neither means the sites have similar policies.