Data as of 2026-07-25 ·
robots.txt via Common Crawl CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
687,794 robots.txt fetches ·
491,025 parsed ·
54,000 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations in each source's native Top-100 order.
Research consensus
Tranco Top 100
A 30-day, multi-provider ranking designed for stable web research.
| Tranco rank | Source domain | robots.txt | llms.txt |
|---|---|---|---|
| 1 | google.com | Present | HTTP 404 |
| 2 | gtld-servers.net | Not measured | Not probed |
| 3 | cloudflare.com | Present | Published · conformant |
| 4 | gstatic.com | Present | Not probed |
| 5 | facebook.com | Present | Published · non-conformant |
| 6 | microsoft.com | Present | HTTP 404 |
| 7 | googleapis.com | HTTP 404 | Not probed |
| 8 | amazonaws.com | Present | HTTP 404 |
| 9 | youtube.com | Present | HTTP 404 |
| 10 | apple.com | Present | HTTP 404 |
| 11 | instagram.com | Present | Published · non-conformant |
| 12 | ezviz7.com | Not measured | Not probed |
| 13 | mail.ru | Present | Published · non-conformant |
| 14 | akamai.net | Not measured | Not probed |
| 15 | fbcdn.net | Present | Published · non-conformant |
| 16 | twitter.com | Present | Published · non-conformant |
| 17 | dzen.ru | Present | HTTP 404 |
| 18 | linkedin.com | Present | HTTP 404 |
| 19 | googletagmanager.com | Not measured | Not probed |
| 20 | googlevideo.com | HTTP 404 | Not probed |
| 21 | office.com | HTTP 302 | Published · non-conformant |
| 22 | live.com | HTTP 301 | Published · non-conformant |
| 23 | hicloudcam.com | Not measured | Not probed |
| 24 | akamaiedge.net | Not measured | Not probed |
| 25 | amazon.com | Present | HTTP 404 |
| 26 | akadns.net | Not measured | Not probed |
| 27 | azure.com | HTTP 301 | Published · conformant |
| 28 | domaincontrol.com | Not measured | Not probed |
| 29 | wikipedia.org | HTTP 301 | HTTP 404 |
| 30 | github.com | Present | Published · conformant |
| 31 | bing.com | Present | HTTP 404 |
| 32 | whatsapp.net | HTTP 302 | HTTP 404 |
| 33 | fastly.net | Not measured | Published · conformant |
| 34 | doubleclick.net | Present | HTTP 404 |
| 35 | apple-dns.net | Not measured | Not probed |
| 36 | googleusercontent.com | HTTP 404 | Not probed |
| 37 | appsflyersdk.com | Not measured | Not probed |
| 38 | trafficmanager.net | Not measured | Not probed |
| 39 | microsoftonline.com | Not measured | Not probed |
| 40 | aaplimg.com | Not measured | Not probed |
| 41 | netflix.com | Present | Not probed |
| 42 | office.net | Not measured | Not probed |
| 43 | wordpress.org | Present | Published · conformant |
| 44 | digicert.com | Present | Published · conformant |
| 45 | sharepoint.com | Not measured | Not probed |
| 46 | youtu.be | Present | Published · non-conformant |
| 47 | skype.com | HTTP 301 | Published · non-conformant |
| 48 | ripn.net | Not measured | Not probed |
| 49 | gandi.net | Present | Not probed |
| 50 | cloudfront.net | Fetch failed · connect error | Not probed |
| 51 | pinterest.com | Present | Published · non-conformant |
| 52 | goo.gl | HTTP 301 | Not probed |
| 53 | x.com | Present | Published · non-conformant |
| 54 | roblox.com | Present | HTTP 404 |
| 55 | cloud.microsoft | HTTP 301 | Published · non-conformant |
| 56 | whatsapp.com | Present | HTTP 404 |
| 57 | windowsupdate.com | Not measured | Not probed |
| 58 | icloud.com | Present | HTTP 404 |
| 59 | yahoo.com | Present | Not probed |
| 60 | tiktok.com | Present | Published · non-conformant |
| 61 | msn.com | Present | HTTP 404 |
| 62 | googlesyndication.com | HTTP 404 | HTTP 404 |
| 63 | googledomains.com | HTTP 301 | Published · non-conformant |
| 64 | cloudflare.net | Present | Not probed |
| 65 | spotify.com | Present | Published · non-conformant |
| 66 | adobe.com | Present | Published · conformant |
| 67 | windows.net | Not measured | Not probed |
| 68 | chatgpt.com | Present | Not probed |
| 69 | ntp.org | Present | HTTP 404 |
| 70 | wa.me | Present | Published · non-conformant |
| 71 | gvt1.com | Not measured | Not probed |
| 72 | vimeo.com | Present | HTTP 404 |
| 73 | myfritz.net | Present | Published · non-conformant |
| 74 | tiktokcdn.com | Not measured | Not probed |
| 75 | akam.net | Not measured | Not probed |
| 76 | gvt2.com | Not measured | Not probed |
| 77 | gwfb.net | Not measured | Not probed |
| 78 | zoom.us | Present | HTTP 404 |
| 79 | cdninstagram.com | Fetch failed · connect error | Not probed |
| 80 | edgekey.net | Not measured | Not probed |
| 81 | pv-cdn.net | Not measured | Not probed |
| 82 | nginx.org | Present | HTTP 404 |
| 83 | baidu.com | Present | HTTP 404 |
| 84 | qq.com | Present | Published · non-conformant |
| 85 | windows.com | HTTP 301 | Published · non-conformant |
| 86 | tiktokv.com | Present | Not probed |
| 87 | ytimg.com | Not measured | Not probed |
| 88 | yandex.net | HTTP 404 | HTTP 404 |
| 89 | cloudflare-dns.com | HTTP 301 | HTTP 404 |
| 90 | mozilla.org | Present | HTTP 404 |
| 91 | workers.dev | HTTP 301 | Published · non-conformant |
| 92 | opera.com | Present | Published · conformant |
| 93 | nic.ru | Present | Published · non-conformant |
| 94 | samsung.com | Present | Published · conformant |
| 95 | yandex.ru | Present | Not probed |
| 96 | nginx.com | HTTP 301 | Published · non-conformant |
| 97 | reddit.com | Present | Published · non-conformant |
| 98 | edgesuite.net | Not measured | Not probed |
| 99 | wordpress.com | HTTP 429 | Published · conformant |
| 100 | sentry.io | Present | Published · conformant |
These are source-native views, not claims of one universal popularity truth. Bot For You stores only the bounded rows used here. See ranking methodology, normalization, and attribution.