Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations through each source's available ranking or membership view.
Network popularity
Cloudflare Radar Top 100
An exact Top 100 ordered from Cloudflare's global network vantage.
| Cloudflare Radar rank | Source domain | robots.txt | llms.txt |
|---|---|---|---|
| 1 | google.com | Present | HTTP 404 |
| 2 | googleapis.com | HTTP 404 | Not probed |
| 3 | cloudflare.com | Present | Published · conformant |
| 4 | gstatic.com | Present | Not probed |
| 5 | apple.com | Present | HTTP 404 |
| 6 | microsoft.com | Present | HTTP 404 |
| 7 | facebook.com | Present | Published · non-conformant |
| 8 | amazonaws.com | HTTP 301 | HTTP 404 |
| 9 | googlevideo.com | Fetch failed · connect error | Not probed |
| 10 | fbcdn.net | Present | Published · non-conformant |
| 11 | whatsapp.net | HTTP 302 | HTTP 404 |
| 12 | amazon.com | Present | HTTP 404 |
| 13 | instagram.com | Present | Published · non-conformant |
| 14 | youtube.com | Present | HTTP 404 |
| 15 | bing.com | Present | HTTP 404 |
| 16 | doubleclick.net | Present | HTTP 404 |
| 17 | akadns.net | Fetch failed · dns error | Not probed |
| 18 | apple-dns.net | Fetch failed · dns error | Not probed |
| 19 | googleusercontent.com | HTTP 404 | Not probed |
| 20 | netflix.com | Present | Not probed |
| 21 | icloud.com | Present | HTTP 404 |
| 22 | chatgpt.com | Present | Not probed |
| 23 | cdninstagram.com | Fetch failed · dns error | Not probed |
| 24 | live.com | HTTP 301 | Published · non-conformant |
| 25 | ntp.org | Present | HTTP 404 |
| 26 | aaplimg.com | Fetch failed · dns error | Not probed |
| 27 | googlesyndication.com | HTTP 404 | HTTP 404 |
| 28 | cloudflare-dns.com | HTTP 301 | HTTP 404 |
| 29 | akamai.net | Fetch failed · dns error | Not probed |
| 30 | cloudfront.net | Fetch failed · dns error | Not probed |
| 31 | tiktokv.com | Present | Not probed |
| 32 | tiktokcdn.com | Fetch failed · dns error | Not probed |
| 33 | ytimg.com | Fetch failed · dns error | Not probed |
| 34 | akamaiedge.net | Fetch failed · dns error | Not probed |
| 35 | gvt2.com | Fetch failed · dns error | Not probed |
| 36 | ui.com | Present | HTTP 404 |
| 37 | yahoo.com | Present | Not probed |
| 38 | office.com | HTTP 302 | Published · non-conformant |
| 39 | spotify.com | Present | Published · non-conformant |
| 40 | fastly.net | HTTP 301 | Published · conformant |
| 41 | samsung.com | Present | Published · conformant |
| 42 | gvt1.com | Fetch failed · dns error | Not probed |
| 43 | sentry.io | Present | Published · conformant |
| 44 | prodregistryv2.org | HTTP 404 | Not probed |
| 45 | dns.google | HTTP 404 | HTTP 404 |
| 46 | roblox.com | Present | HTTP 404 |
| 47 | app-analytics-services.com | HTTP 404 | Not probed |
| 48 | baidu.com | Present | HTTP 404 |
| 49 | app-measurement.com | HTTP 404 | Not probed |
| 50 | msn.com | Present | HTTP 404 |
| 51 | google-analytics.com | Present | HTTP 404 |
| 52 | criteo.com | Present | Not probed |
| 53 | one.one | Present | Published · conformant |
| 54 | msftncsi.com | Fetch failed · dns error | Not probed |
| 55 | whatsapp.com | Present | HTTP 404 |
| 56 | edgcdn.net | Fetch failed · dns error | Not probed |
| 57 | 3gppnetwork.org | Fetch failed · dns error | Not probed |
| 58 | azure.com | HTTP 301 | Published · conformant |
| 59 | trafficmanager.net | Fetch failed · dns error | Not probed |
| 60 | steamserver.net | Fetch failed · dns error | Not probed |
| 61 | snapchat.com | Present | HTTP 404 |
| 62 | wikipedia.org | HTTP 301 | HTTP 404 |
| 63 | appsflyersdk.com | HTTP 403 | Not probed |
| 64 | windows.com | HTTP 301 | Published · non-conformant |
| 65 | linkedin.com | Present | Published · non-conformant |
| 66 | applovin.com | Present | Published · non-conformant |
| 67 | cloud.microsoft | HTTP 301 | Published · non-conformant |
| 68 | amazon-adsystem.com | HTTP 404 | Not probed |
| 69 | windows.net | Fetch failed · connect error | Not probed |
| 70 | googletagmanager.com | HTTP 404 | Not probed |
| 71 | microsoftonline.com | Fetch failed · dns error | Not probed |
| 72 | amazon.dev | Fetch failed · dns error | Not probed |
| 73 | adtrafficquality.google | HTTP 404 | HTTP 404 |
| 74 | ggpht.com | Fetch failed · dns error | Not probed |
| 75 | a2z.com | Fetch failed · dns error | Not probed |
| 76 | msftconnecttest.com | Fetch failed · dns error | Not probed |
| 77 | oxylabs.io | Present | Published · conformant |
| 78 | skype.com | HTTP 301 | Published · non-conformant |
| 79 | unity3d.com | HTTP 302 | Published · conformant |
| 80 | rubiconproject.com | Present | Not probed |
| 81 | openai.com | Present | Not probed |
| 82 | taboola.com | Present | Published · conformant |
| 83 | windowsupdate.com | Fetch failed · dns error | Not probed |
| 84 | xiaomi.com | HTTP 301 | Not probed |
| 85 | office.net | Fetch failed · dns error | Not probed |
| 86 | tiktok.com | Present | Published · non-conformant |
| 87 | digicert.com | Present | Published · conformant |
| 88 | playstation.net | Fetch failed · dns error | Not probed |
| 89 | vungle.com | Present | Not probed |
| 90 | adsafeprotected.com | HTTP 302 | Published · non-conformant |
| 91 | gmail.com | Present | HTTP 404 |
| 92 | cdn-apple.com | Fetch failed · dns error | Not probed |
| 93 | pubmatic.com | Present | Published · conformant |
| 94 | qq.com | Present | Published · non-conformant |
| 95 | adsrvr.org | HTTP 302 | HTTP 403 |
| 96 | fastly-edge.com | Fetch failed · dns error | Not probed |
| 97 | googleadservices.com | Present | Not probed |
| 98 | sharepoint.com | HTTP 301 | Not probed |
| 99 | lencr.org | HTTP 302 | Published · non-conformant |
| 100 | aws.dev | Fetch failed · dns error | Not probed |
These are source-grounded views, not claims of one universal popularity truth. CrUX is a membership slice ordered here by Tranco rank; vendor lists retain their native order. See ranking methodology, normalization, and attribution.