robots.txt & llms.txt across the Tranco top 1M
Data as of 2026-07-25 · robots.txt via Common Crawl CC-MAIN-2026-25 + our polite crawl · llms.txt via our polite crawl · Tranco top-1,000,000 687,794 robots.txt fetches · 491,025 parsed · 54,000 llms.txt probes

Popularity lenses

Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations in each source's native Top-100 order.

Link authority

Majestic Million Top 100

Sites ordered by the number of referring network subnets.

Retrieved 2026-07-25 View source ↗
Majestic Million rank Source domain Tranco rank robots.txt llms.txt
1 google.com 1 Present HTTP 404
2 facebook.com 5 Present Published · non-conformant
3 youtube.com 9 Present HTTP 404
4 instagram.com 11 Present Published · non-conformant
5 twitter.com 16 Present Published · non-conformant
6 linkedin.com 18 Present HTTP 404
7 googletagmanager.com 19 Not measured Not probed
8 apple.com 10 Present HTTP 404
9 microsoft.com 6 Present HTTP 404
10 github.com 30 Present Published · conformant
11 wikipedia.org 29 HTTP 301 HTTP 404
12 wordpress.org 43 Present Published · conformant
13 play.google.com Measured as google.com 1 Present HTTP 404
14 youtu.be 46 Present Published · non-conformant
15 goo.gl 52 HTTP 301 Not probed
16 en.wikipedia.org Measured as wikipedia.org 29 HTTP 301 HTTP 404
17 x.com 53 Present Published · non-conformant
18 maps.google.com Measured as google.com 1 Present HTTP 404
19 pinterest.com 51 Present Published · non-conformant
20 apps.apple.com Measured as apple.com 10 Present HTTP 404
21 wa.me 70 Present Published · non-conformant
22 tiktok.com 60 Present Published · non-conformant
23 nginx.org 82 Present HTTP 404
24 docs.google.com Measured as google.com 1 Present HTTP 404
25 vimeo.com 72 Present HTTP 404
26 policies.google.com Measured as google.com 1 Present HTTP 404
27 whatsapp.com 56 Present HTTP 404
28 nginx.com 96 HTTP 301 Published · non-conformant
29 amazon.com 25 Present HTTP 404
30 drive.google.com Measured as google.com 1 Present HTTP 404
31 api.whatsapp.com Measured as whatsapp.com 56 Present HTTP 404
32 mozilla.org 90 Present HTTP 404
33 adobe.com 66 Present Published · conformant
34 maps.app.goo.gl Measured as goo.gl 52 HTTP 301 Not probed
35 wordpress.com 99 HTTP 429 Published · conformant
36 t.me 105 HTTP 404 Published · non-conformant
37 bit.ly 103 Present Published · non-conformant
38 support.google.com Measured as google.com 1 Present HTTP 404
39 europa.eu 111 Present HTTP 404
40 apache.org 117 Present HTTP 404
41 blogspot.com 110 Present HTTP 404
42 plus.google.com Measured as google.com 1 Present HTTP 404
43 github.io 115 HTTP 301 Published · non-conformant
44 miit.gov.cn 143 HTTP 404 Not probed
45 reddit.com 97 Present Published · non-conformant
46 forms.gle 133 HTTP 400 Not probed
47 yahoo.com 59 Present Not probed
48 beian.miit.gov.cn Measured as miit.gov.cn 143 HTTP 404 Not probed
49 itunes.apple.com Measured as apple.com 10 Present HTTP 404
50 nih.gov 127 HTTP 403 HTTP 404
51 qq.com 84 Present Published · non-conformant
52 gravatar.com 136 Present Published · conformant
53 httpd.apache.org Measured as apache.org 117 Present HTTP 404
54 spotify.com 65 Present Published · non-conformant
55 godaddy.com 134 Present Not probed
56 tumblr.com 151 HTTP 403 Not probed
57 macromedia.com 170 HTTP 301 Not probed
58 open.spotify.com Measured as spotify.com 65 Present Published · non-conformant
59 archive.org 145 Present HTTP 404
60 amazonaws.com 8 Present HTTP 404
61 sites.google.com Measured as google.com 1 Present HTTP 404
62 office.com 21 HTTP 302 Published · non-conformant
63 flickr.com 173 Not measured Not probed
64 baidu.com 83 Present HTTP 404
65 nytimes.com 153 Present HTTP 404
66 w3.org 181 Present HTTP 404
67 creativecommons.org 200 Present HTTP 404
68 medium.com 184 Present Not probed
69 soundcloud.com 178 Present HTTP 404
70 doi.org 203 Present HTTP 404
71 forbes.com 198 Present HTTP 404
72 cpanel.net 210 Present HTTP 404
73 live.com 22 HTTP 301 Published · non-conformant
74 web.archive.org Measured as archive.org 145 Present HTTP 404
75 zoom.us 78 Present HTTP 404
76 developers.google.com Measured as google.com 1 Present HTTP 404
77 sciencedirect.com 216 HTTP 301 Not probed
78 www.ncbi.nlm.nih.gov Measured as nih.gov 127 HTTP 403 HTTP 404
79 sourceforge.net 228 HTTP 403 Published · non-conformant
80 theguardian.com 188 Present HTTP 404
81 youtube-nocookie.com 217 Present Not probed
82 paypal.com 159 Present Published · conformant
83 who.int 252 Present HTTP 404
84 oracle.com 196 HTTP 403 Not probed
85 accounts.google.com Measured as google.com 1 Present HTTP 404
86 shopify.com 168 Present Published · conformant
87 wixsite.com 261 Fetch failed · connect error Not probed
88 opera.com 92 Present Published · conformant
89 t.co 223 HTTP 530 HTTP 404
90 launchpad.net 269 Present HTTP 404
91 dropbox.com 137 Present Published · conformant
92 bbc.com 208 Not measured HTTP 404
93 cnn.com 191 Present Not probed
94 vk.com 114 Present HTTP 404
95 tinyurl.com 282 Present HTTP 404
96 researchgate.net 219 Present Not probed
97 springer.com 275 Present Published · non-conformant
98 googleblog.com 298 HTTP 302 Published · non-conformant
99 reuters.com 283 Present Not probed
100 mit.edu 222 Present HTTP 404

These are source-native views, not claims of one universal popularity truth. Bot For You stores only the bounded rows used here. See ranking methodology, normalization, and attribution.