Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations through each source's available ranking or membership view.
Link authority
Majestic Million Top 100
Sites ordered by the number of referring network subnets.
| Majestic Million rank | Source domain | robots.txt | llms.txt |
|---|---|---|---|
| 1 | google.com | Present | HTTP 404 |
| 2 | facebook.com | Present | Published · non-conformant |
| 3 | youtube.com | Present | HTTP 404 |
| 4 | instagram.com | Present | Published · non-conformant |
| 5 | twitter.com | Present | Published · non-conformant |
| 6 | linkedin.com | Present | Published · non-conformant |
| 7 | googletagmanager.com | HTTP 404 | Not probed |
| 8 | apple.com | Present | HTTP 404 |
| 9 | microsoft.com | Present | HTTP 404 |
| 10 | github.com | Present | Published · conformant |
| 11 | wikipedia.org | HTTP 301 | HTTP 404 |
| 12 | wordpress.org | Present | Published · conformant |
| 13 | play.google.com Measured as google.com | Present | HTTP 404 |
| 14 | youtu.be | Present | Published · non-conformant |
| 15 | goo.gl | HTTP 400 | Not probed |
| 16 | en.wikipedia.org Measured as wikipedia.org | HTTP 301 | HTTP 404 |
| 17 | x.com | Present | Published · non-conformant |
| 18 | maps.google.com Measured as google.com | Present | HTTP 404 |
| 19 | pinterest.com | Present | Published · non-conformant |
| 20 | apps.apple.com Measured as apple.com | Present | HTTP 404 |
| 21 | wa.me | Present | HTTP 404 |
| 22 | tiktok.com | Present | Published · non-conformant |
| 23 | nginx.org | Present | HTTP 404 |
| 24 | docs.google.com Measured as google.com | Present | HTTP 404 |
| 25 | vimeo.com | Present | HTTP 404 |
| 26 | policies.google.com Measured as google.com | Present | HTTP 404 |
| 27 | whatsapp.com | Present | HTTP 404 |
| 28 | nginx.com | HTTP 301 | Published · non-conformant |
| 29 | amazon.com | Present | HTTP 404 |
| 30 | drive.google.com Measured as google.com | Present | HTTP 404 |
| 31 | api.whatsapp.com Measured as whatsapp.com | Present | HTTP 404 |
| 32 | mozilla.org | Present | HTTP 404 |
| 33 | adobe.com | Present | Published · conformant |
| 34 | maps.app.goo.gl Measured as goo.gl | HTTP 400 | Not probed |
| 35 | wordpress.com | HTTP 429 | Published · conformant |
| 36 | t.me | HTTP 301 | Published · non-conformant |
| 37 | bit.ly | Present | Published · non-conformant |
| 38 | support.google.com Measured as google.com | Present | HTTP 404 |
| 39 | europa.eu | Present | HTTP 404 |
| 40 | apache.org | Present | HTTP 404 |
| 41 | blogspot.com | Present | HTTP 404 |
| 42 | plus.google.com Measured as google.com | Present | HTTP 404 |
| 43 | github.io | HTTP 301 | Published · non-conformant |
| 44 | miit.gov.cn | HTTP 404 | Not probed |
| 45 | reddit.com | Present | Published · non-conformant |
| 46 | forms.gle | HTTP 400 | Not probed |
| 47 | yahoo.com | Present | Not probed |
| 48 | beian.miit.gov.cn Measured as miit.gov.cn | HTTP 404 | Not probed |
| 49 | itunes.apple.com Measured as apple.com | Present | HTTP 404 |
| 50 | nih.gov | Present | HTTP 404 |
| 51 | qq.com | Present | Published · non-conformant |
| 52 | gravatar.com | Present | Published · conformant |
| 53 | httpd.apache.org Measured as apache.org | Present | HTTP 404 |
| 54 | spotify.com | Present | Published · non-conformant |
| 55 | godaddy.com | Present | Not probed |
| 56 | tumblr.com | HTTP 403 | Not probed |
| 57 | macromedia.com | HTTP 301 | Not probed |
| 58 | open.spotify.com Measured as spotify.com | Present | Published · non-conformant |
| 59 | archive.org | Present | HTTP 404 |
| 60 | amazonaws.com | HTTP 301 | HTTP 404 |
| 61 | sites.google.com Measured as google.com | Present | HTTP 404 |
| 62 | office.com | HTTP 302 | Published · non-conformant |
| 63 | flickr.com | Present | Not probed |
| 64 | baidu.com | Present | HTTP 404 |
| 65 | nytimes.com | Present | HTTP 404 |
| 66 | w3.org | Present | HTTP 404 |
| 67 | creativecommons.org | Present | HTTP 404 |
| 68 | medium.com | Present | Not probed |
| 69 | soundcloud.com | Present | HTTP 404 |
| 70 | doi.org | HTTP 301 | HTTP 404 |
| 71 | forbes.com | HTTP 307 | HTTP 404 |
| 72 | cpanel.net | Present | HTTP 404 |
| 73 | live.com | HTTP 301 | Published · non-conformant |
| 74 | web.archive.org Measured as archive.org | Present | HTTP 404 |
| 75 | zoom.us | Present | HTTP 404 |
| 76 | developers.google.com Measured as google.com | Present | HTTP 404 |
| 77 | sciencedirect.com | HTTP 301 | Not probed |
| 78 | www.ncbi.nlm.nih.gov Measured as nih.gov | Present | HTTP 404 |
| 79 | sourceforge.net | HTTP 403 | Published · non-conformant |
| 80 | theguardian.com | Present | HTTP 404 |
| 81 | youtube-nocookie.com | Present | Not probed |
| 82 | paypal.com | Present | Published · conformant |
| 83 | who.int | Present | HTTP 404 |
| 84 | oracle.com | Present | Not probed |
| 85 | accounts.google.com Measured as google.com | Present | HTTP 404 |
| 86 | shopify.com | Present | Published · conformant |
| 87 | wixsite.com | Fetch failed · dns error | Not probed |
| 88 | opera.com | Present | Published · conformant |
| 89 | t.co | Present | HTTP 404 |
| 90 | launchpad.net | Present | HTTP 404 |
| 91 | dropbox.com | Present | Published · conformant |
| 92 | bbc.com | Present | HTTP 404 |
| 93 | cnn.com | Present | Not probed |
| 94 | vk.com | Present | HTTP 404 |
| 95 | tinyurl.com | Present | HTTP 404 |
| 96 | researchgate.net | Present | Not probed |
| 97 | springer.com | Present | Published · non-conformant |
| 98 | googleblog.com | HTTP 302 | Published · non-conformant |
| 99 | reuters.com | Present | Not probed |
| 100 | mit.edu | Present | HTTP 404 |
These are source-grounded views, not claims of one universal popularity truth. CrUX is a membership slice ordered here by Tranco rank; vendor lists retain their native order. See ranking methodology, normalization, and attribution.