Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations through each source's available ranking or membership view.
Research consensus
Tranco Top 100
A 30-day, multi-provider ranking designed for stable web research.
| Tranco rank | Source domain | robots.txt | llms.txt |
|---|---|---|---|
| 1 | google.com | Present | HTTP 404 |
| 2 | cloudflare.com | Present | Published · conformant |
| 3 | gstatic.com | Present | Not probed |
| 4 | facebook.com | Present | Published · non-conformant |
| 5 | microsoft.com | Present | HTTP 404 |
| 6 | googleapis.com | HTTP 404 | Not probed |
| 7 | amazonaws.com | HTTP 301 | HTTP 404 |
| 8 | youtube.com | Present | HTTP 404 |
| 9 | akamai.net | Fetch failed · dns error | Not probed |
| 10 | apple.com | Present | HTTP 404 |
| 11 | instagram.com | Present | Published · non-conformant |
| 12 | mail.ru | Present | Published · non-conformant |
| 13 | ezviz7.com | Fetch failed · dns error | Not probed |
| 14 | fbcdn.net | Present | Published · non-conformant |
| 15 | dzen.ru | Present | HTTP 404 |
| 16 | twitter.com | Present | Published · non-conformant |
| 17 | linkedin.com | Present | Published · non-conformant |
| 18 | gtld-servers.net | Fetch failed · dns error | Not probed |
| 19 | domaincontrol.com | Fetch failed · connect error | Not probed |
| 20 | googlevideo.com | Fetch failed · connect error | Not probed |
| 21 | office.com | HTTP 302 | Published · non-conformant |
| 22 | googletagmanager.com | HTTP 404 | Not probed |
| 23 | hicloudcam.com | Fetch failed · dns error | Not probed |
| 24 | live.com | HTTP 301 | Published · non-conformant |
| 25 | akamaiedge.net | Fetch failed · dns error | Not probed |
| 26 | amazon.com | Present | HTTP 404 |
| 27 | akadns.net | Fetch failed · dns error | Not probed |
| 28 | azure.com | HTTP 301 | Published · conformant |
| 29 | bing.com | Present | HTTP 404 |
| 30 | github.com | Present | Published · conformant |
| 31 | wikipedia.org | HTTP 301 | HTTP 404 |
| 32 | whatsapp.net | HTTP 302 | HTTP 404 |
| 33 | apple-dns.net | Fetch failed · dns error | Not probed |
| 34 | googleusercontent.com | HTTP 404 | Not probed |
| 35 | fastly.net | HTTP 301 | Published · conformant |
| 36 | appsflyersdk.com | HTTP 403 | Not probed |
| 37 | doubleclick.net | Present | HTTP 404 |
| 38 | aaplimg.com | Fetch failed · dns error | Not probed |
| 39 | microsoftonline.com | Fetch failed · dns error | Not probed |
| 40 | office.net | Fetch failed · dns error | Not probed |
| 41 | netflix.com | Present | Not probed |
| 42 | trafficmanager.net | Fetch failed · dns error | Not probed |
| 43 | gandi.net | Present | HTTP 404 |
| 44 | sharepoint.com | HTTP 301 | Not probed |
| 45 | youtu.be | Present | Published · non-conformant |
| 46 | digicert.com | Present | Published · conformant |
| 47 | cloud.microsoft | HTTP 301 | Published · non-conformant |
| 48 | wordpress.org | Present | Published · conformant |
| 49 | skype.com | HTTP 301 | Published · non-conformant |
| 50 | x.com | Present | Published · non-conformant |
| 51 | goo.gl | HTTP 400 | Not probed |
| 52 | cloudfront.net | Fetch failed · dns error | Not probed |
| 53 | icloud.com | Present | HTTP 404 |
| 54 | googledomains.com | HTTP 301 | Published · non-conformant |
| 55 | tiktok.com | Present | Published · non-conformant |
| 56 | whatsapp.com | Present | HTTP 404 |
| 57 | pinterest.com | Present | Published · non-conformant |
| 58 | roblox.com | Present | HTTP 404 |
| 59 | cloudflare.net | HTTP 403 | Not probed |
| 60 | yahoo.com | Present | Not probed |
| 61 | gvt1.com | Fetch failed · dns error | Not probed |
| 62 | windowsupdate.com | Fetch failed · dns error | Not probed |
| 63 | googlesyndication.com | HTTP 404 | HTTP 404 |
| 64 | spotify.com | Present | Published · non-conformant |
| 65 | msn.com | Present | HTTP 404 |
| 66 | windows.net | Fetch failed · connect error | Not probed |
| 67 | ntp.org | Present | HTTP 404 |
| 68 | adobe.com | Present | Published · conformant |
| 69 | pv-cdn.net | Fetch failed · dns error | Not probed |
| 70 | myfritz.net | Present | Published · non-conformant |
| 71 | wa.me | Present | HTTP 404 |
| 72 | akam.net | Fetch failed · dns error | Not probed |
| 73 | nginx.org | Present | HTTP 404 |
| 74 | cdninstagram.com | Fetch failed · dns error | Not probed |
| 75 | workers.dev | Present | Published · non-conformant |
| 76 | chatgpt.com | Present | Not probed |
| 77 | tiktokcdn.com | Fetch failed · dns error | Not probed |
| 78 | zoom.us | Present | HTTP 404 |
| 79 | tiktokv.com | Present | Not probed |
| 80 | cloudflare-dns.com | HTTP 301 | HTTP 404 |
| 81 | edgekey.net | Fetch failed · dns error | Not probed |
| 82 | vimeo.com | Present | HTTP 404 |
| 83 | baidu.com | Present | HTTP 404 |
| 84 | openai.com | Present | Not probed |
| 85 | gvt2.com | Fetch failed · dns error | Not probed |
| 86 | qq.com | Present | Published · non-conformant |
| 87 | yandex.net | HTTP 404 | HTTP 404 |
| 88 | root-servers.net | Fetch failed · dns error | Not probed |
| 89 | windows.com | HTTP 301 | Published · non-conformant |
| 90 | ytimg.com | Fetch failed · dns error | Not probed |
| 91 | opera.com | Present | Published · conformant |
| 92 | blogspot.com | Present | HTTP 404 |
| 93 | gwfb.net | Fetch failed · dns error | Not probed |
| 94 | samsung.com | Present | Published · conformant |
| 95 | nginx.com | HTTP 301 | Published · non-conformant |
| 96 | yandex.ru | Present | Not probed |
| 97 | edgesuite.net | Fetch failed · dns error | Not probed |
| 98 | sentry.io | Present | Published · conformant |
| 99 | wordpress.com | HTTP 429 | Published · conformant |
| 100 | mozilla.org | Present | HTTP 404 |
These are source-grounded views, not claims of one universal popularity truth. CrUX is a membership slice ordered here by Tranco rank; vendor lists retain their native order. See ranking methodology, normalization, and attribution.