Data as of 2026-07-25 ·
robots.txt via Common Crawl CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
687,794 robots.txt fetches ·
491,025 parsed ·
54,000 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations in each source's native Top-100 order.
Network popularity
Cloudflare Radar Top 100
An exact Top 100 ordered from Cloudflare's global network vantage.
| Cloudflare Radar rank | Source domain | Tranco rank | robots.txt | llms.txt |
|---|---|---|---|---|
| 1 | google.com | 1 | Present | HTTP 404 |
| 2 | googleapis.com | 7 | HTTP 404 | Not probed |
| 3 | cloudflare.com | 3 | Present | Published · conformant |
| 4 | gstatic.com | 4 | Present | Not probed |
| 5 | apple.com | 10 | Present | HTTP 404 |
| 6 | microsoft.com | 6 | Present | HTTP 404 |
| 7 | facebook.com | 5 | Present | Published · non-conformant |
| 8 | amazonaws.com | 8 | Present | HTTP 404 |
| 9 | googlevideo.com | 20 | HTTP 404 | Not probed |
| 10 | fbcdn.net | 15 | Present | Published · non-conformant |
| 11 | whatsapp.net | 32 | HTTP 302 | HTTP 404 |
| 12 | amazon.com | 25 | Present | HTTP 404 |
| 13 | instagram.com | 11 | Present | Published · non-conformant |
| 14 | youtube.com | 9 | Present | HTTP 404 |
| 15 | bing.com | 31 | Present | HTTP 404 |
| 16 | doubleclick.net | 34 | Present | HTTP 404 |
| 17 | akadns.net | 26 | Not measured | Not probed |
| 18 | apple-dns.net | 35 | Not measured | Not probed |
| 19 | googleusercontent.com | 36 | HTTP 404 | Not probed |
| 20 | netflix.com | 41 | Present | Not probed |
| 21 | icloud.com | 58 | Present | HTTP 404 |
| 22 | chatgpt.com | 68 | Present | Not probed |
| 23 | cdninstagram.com | 79 | Fetch failed · connect error | Not probed |
| 24 | live.com | 22 | HTTP 301 | Published · non-conformant |
| 25 | ntp.org | 69 | Present | HTTP 404 |
| 26 | aaplimg.com | 40 | Not measured | Not probed |
| 27 | googlesyndication.com | 62 | HTTP 404 | HTTP 404 |
| 28 | cloudflare-dns.com | 89 | HTTP 301 | HTTP 404 |
| 29 | akamai.net | 14 | Not measured | Not probed |
| 30 | cloudfront.net | 50 | Fetch failed · connect error | Not probed |
| 31 | tiktokv.com | 86 | Present | Not probed |
| 32 | tiktokcdn.com | 74 | Not measured | Not probed |
| 33 | ytimg.com | 87 | Not measured | Not probed |
| 34 | akamaiedge.net | 24 | Not measured | Not probed |
| 35 | gvt2.com | 76 | Not measured | Not probed |
| 36 | ui.com | 101 | Present | HTTP 404 |
| 37 | yahoo.com | 59 | Present | Not probed |
| 38 | office.com | 21 | HTTP 302 | Published · non-conformant |
| 39 | spotify.com | 65 | Present | Published · non-conformant |
| 40 | fastly.net | 33 | Not measured | Published · conformant |
| 41 | samsung.com | 94 | Present | Published · conformant |
| 42 | gvt1.com | 71 | Not measured | Not probed |
| 43 | sentry.io | 100 | Present | Published · conformant |
| 44 | prodregistryv2.org | 130 | HTTP 302 | Not probed |
| 45 | dns.google | 131 | HTTP 404 | HTTP 404 |
| 46 | roblox.com | 54 | Present | HTTP 404 |
| 47 | app-analytics-services.com | 128 | Not measured | Not probed |
| 48 | baidu.com | 83 | Present | HTTP 404 |
| 49 | app-measurement.com | 118 | Not measured | Not probed |
| 50 | msn.com | 61 | Present | HTTP 404 |
| 51 | google-analytics.com | 104 | Present | HTTP 404 |
| 52 | criteo.com | 113 | HTTP 403 | Not probed |
| 53 | one.one | 156 | Not measured | Published · conformant |
| 54 | msftncsi.com | 135 | HTTP 404 | Not probed |
| 55 | whatsapp.com | 56 | Present | HTTP 404 |
| 56 | edgcdn.net | 209 | Not measured | Not probed |
| 57 | 3gppnetwork.org | 164 | Not measured | Not probed |
| 58 | azure.com | 27 | HTTP 301 | Published · conformant |
| 59 | trafficmanager.net | 38 | Not measured | Not probed |
| 60 | steamserver.net | 149 | Not measured | Not probed |
| 61 | snapchat.com | 119 | Present | HTTP 404 |
| 62 | wikipedia.org | 29 | HTTP 301 | HTTP 404 |
| 63 | appsflyersdk.com | 37 | Not measured | Not probed |
| 64 | windows.com | 85 | HTTP 301 | Published · non-conformant |
| 65 | linkedin.com | 18 | Present | HTTP 404 |
| 66 | applovin.com | 163 | HTTP 307 | Published · non-conformant |
| 67 | cloud.microsoft | 55 | HTTP 301 | Published · non-conformant |
| 68 | amazon-adsystem.com | 121 | Not measured | Not probed |
| 69 | windows.net | 67 | Not measured | Not probed |
| 70 | googletagmanager.com | 19 | Not measured | Not probed |
| 71 | microsoftonline.com | 39 | Not measured | Not probed |
| 72 | amazon.dev | 132 | Not measured | Not probed |
| 73 | adtrafficquality.google | 161 | HTTP 404 | HTTP 404 |
| 74 | ggpht.com | 169 | Not measured | Not probed |
| 75 | a2z.com | 109 | Not measured | Not probed |
| 76 | msftconnecttest.com | 165 | Not measured | Not probed |
| 77 | oxylabs.io | 240 | Present | Published · conformant |
| 78 | skype.com | 47 | HTTP 301 | Published · non-conformant |
| 79 | unity3d.com | 120 | HTTP 301 | HTTP 404 |
| 80 | rubiconproject.com | 195 | Present | Not probed |
| 81 | openai.com | 205 | Present | Not probed |
| 82 | taboola.com | 194 | Present | Published · conformant |
| 83 | windowsupdate.com | 57 | Not measured | Not probed |
| 84 | xiaomi.com | 148 | HTTP 307 | Not probed |
| 85 | office.net | 42 | Not measured | Not probed |
| 86 | tiktok.com | 60 | Present | Published · non-conformant |
| 87 | digicert.com | 44 | Present | Published · conformant |
| 88 | playstation.net | 218 | Not measured | Not probed |
| 89 | vungle.com | 192 | Present | Not probed |
| 90 | adsafeprotected.com | 220 | HTTP 302 | Published · non-conformant |
| 91 | gmail.com | 215 | Present | HTTP 404 |
| 92 | cdn-apple.com | 180 | Not measured | Not probed |
| 93 | pubmatic.com | 233 | Present | Not probed |
| 94 | qq.com | 84 | Present | Published · non-conformant |
| 95 | adsrvr.org | 237 | Not measured | HTTP 403 |
| 96 | fastly-edge.com | 232 | Not measured | Not probed |
| 97 | googleadservices.com | 122 | Present | Not probed |
| 98 | sharepoint.com | 45 | Not measured | Not probed |
| 99 | lencr.org | 279 | HTTP 302 | Published · non-conformant |
| 100 | aws.dev | 171 | Not measured | Not probed |
These are source-native views, not claims of one universal popularity truth. Bot For You stores only the bounded rows used here. See ranking methodology, normalization, and attribution.