Domain explorer
Look up a single domain's robots.txt & llms.txt metrics, effective AI policy, inferred tech stack (labelled hypothesis), and crawl history. Read-only; one indexed lookup per query.
netflix.com
Tranco rank
41
popularity rank
TLD
com
netflix.com
robots.txt records
2
across crawls
llms.txt records
0
across crawls
Live files on netflix.com
Open the file as it exists on the host right now — may differ from what we measured. We link out rather than mirror (we store derived metrics, not raw bodies).
Effective AI / crawl policy (Google posture, §3.3)
| Crawl | Posture | Consec. 5xx | Reason | As of |
|---|---|---|---|---|
| CC-MAIN-2026-25 | blocked | 0 | 2xx: robots.txt disallows all crawling (Disallow: /) | 2026-06-18T14:03:11+00:00 |
robots.txt
| Crawl | Status | Bytes | Groups | Allow | Disallow | Sitemaps | Wildcard * | Disallow-all | AI tokens | AI verdicts |
|---|---|---|---|---|---|---|---|---|---|---|
| CC-MAIN-2026-25 | 200 | 3,790 | 4 | 4 | 133 | 1 | yes | yes | Applebot, GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | applebot:partial, gptbot:partial, chatgpt-user:partial, oai-searchbot:partial, google-extended:partial, claudebot:partial, claude-user:partial, claude-searchbot:partial, perplexitybot:partial |
| CC-MAIN-2026-25 | 200 | 3,790 | 4 | 4 | 133 | 1 | yes | yes | Applebot, GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | — |
llms.txt
Collected by our own polite crawler, not Common Crawl — Common Crawl doesn't capture /llms.txt. See Methodology.
No llms.txt records for this domain.
Inferred tech stack hypothesis
No tech inference for this domain.