Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
Popularity lenses
Different lists measure different things. Browse the same Bot For You robots.txt and llms.txt observations through each source's available ranking or membership view.
Real-user traffic
CrUX member domains
The first 100 domains with sufficient real-user Chrome traffic to appear in the monthly Chrome UX Report, shown in Tranco order because CrUX publishes rank buckets rather than a unique ordinal rank.
| CrUX rank | Source domain | robots.txt | llms.txt |
|---|---|---|---|
| 1 | google.com | Present | HTTP 404 |
| 2 | cloudflare.com | Present | Published · conformant |
| 3 | gstatic.com | Present | Not probed |
| 4 | facebook.com | Present | Published · non-conformant |
| 5 | microsoft.com | Present | HTTP 404 |
| 6 | googleapis.com | HTTP 404 | Not probed |
| 7 | amazonaws.com | HTTP 301 | HTTP 404 |
| 8 | youtube.com | Present | HTTP 404 |
| 10 | apple.com | Present | HTTP 404 |
| 11 | instagram.com | Present | Published · non-conformant |
| 12 | mail.ru | Present | Published · non-conformant |
| 15 | dzen.ru | Present | HTTP 404 |
| 16 | twitter.com | Present | Published · non-conformant |
| 17 | linkedin.com | Present | Published · non-conformant |
| 21 | office.com | HTTP 302 | Published · non-conformant |
| 24 | live.com | HTTP 301 | Published · non-conformant |
| 26 | amazon.com | Present | HTTP 404 |
| 28 | azure.com | HTTP 301 | Published · conformant |
| 29 | bing.com | Present | HTTP 404 |
| 30 | github.com | Present | Published · conformant |
| 31 | wikipedia.org | HTTP 301 | HTTP 404 |
| 34 | googleusercontent.com | HTTP 404 | Not probed |
| 35 | fastly.net | HTTP 301 | Published · conformant |
| 36 | appsflyersdk.com | HTTP 403 | Not probed |
| 40 | office.net | Fetch failed · dns error | Not probed |
| 41 | netflix.com | Present | Not probed |
| 42 | trafficmanager.net | Fetch failed · dns error | Not probed |
| 43 | gandi.net | Present | HTTP 404 |
| 44 | sharepoint.com | HTTP 301 | Not probed |
| 46 | digicert.com | Present | Published · conformant |
| 47 | cloud.microsoft | HTTP 301 | Published · non-conformant |
| 48 | wordpress.org | Present | Published · conformant |
| 49 | skype.com | HTTP 301 | Published · non-conformant |
| 50 | x.com | Present | Published · non-conformant |
| 51 | goo.gl | HTTP 400 | Not probed |
| 52 | cloudfront.net | Fetch failed · dns error | Not probed |
| 53 | icloud.com | Present | HTTP 404 |
| 55 | tiktok.com | Present | Published · non-conformant |
| 56 | whatsapp.com | Present | HTTP 404 |
| 57 | pinterest.com | Present | Published · non-conformant |
| 58 | roblox.com | Present | HTTP 404 |
| 60 | yahoo.com | Present | Not probed |
| 63 | googlesyndication.com | HTTP 404 | HTTP 404 |
| 64 | spotify.com | Present | Published · non-conformant |
| 65 | msn.com | Present | HTTP 404 |
| 66 | windows.net | Fetch failed · connect error | Not probed |
| 68 | adobe.com | Present | Published · conformant |
| 71 | wa.me | Present | HTTP 404 |
| 73 | nginx.org | Present | HTTP 404 |
| 75 | workers.dev | Present | Published · non-conformant |
| 76 | chatgpt.com | Present | Not probed |
| 77 | tiktokcdn.com | Fetch failed · dns error | Not probed |
| 78 | zoom.us | Present | HTTP 404 |
| 79 | tiktokv.com | Present | Not probed |
| 82 | vimeo.com | Present | HTTP 404 |
| 83 | baidu.com | Present | HTTP 404 |
| 84 | openai.com | Present | Not probed |
| 86 | qq.com | Present | Published · non-conformant |
| 87 | yandex.net | HTTP 404 | HTTP 404 |
| 89 | windows.com | HTTP 301 | Published · non-conformant |
| 91 | opera.com | Present | Published · conformant |
| 92 | blogspot.com | Present | HTTP 404 |
| 94 | samsung.com | Present | Published · conformant |
| 96 | yandex.ru | Present | Not probed |
| 97 | edgesuite.net | Fetch failed · dns error | Not probed |
| 98 | sentry.io | Present | Published · conformant |
| 99 | wordpress.com | HTTP 429 | Published · conformant |
| 100 | mozilla.org | Present | HTTP 404 |
| 101 | capgemini.com | Present | Published · conformant |
| 103 | nic.ru | Present | Published · non-conformant |
| 104 | reddit.com | Present | Published · non-conformant |
| 105 | t.me | HTTP 301 | Published · non-conformant |
| 106 | ui.com | Present | HTTP 404 |
| 107 | a2z.com | Fetch failed · dns error | Not probed |
| 108 | b-cdn.net | Present | Published · non-conformant |
| 109 | f5.com | Present | HTTP 404 |
| 110 | office365.com | HTTP 403 | Not probed |
| 111 | bit.ly | Present | Published · non-conformant |
| 112 | europa.eu | Present | HTTP 404 |
| 116 | outlook.com | HTTP 302 | Published · non-conformant |
| 118 | afternic.com | Present | HTTP 404 |
| 119 | vk.com | Present | HTTP 404 |
| 121 | github.io | HTTP 301 | Published · non-conformant |
| 122 | criteo.com | Present | Not probed |
| 127 | apache.org | Present | HTTP 404 |
| 129 | snapchat.com | Present | HTTP 404 |
| 130 | unity3d.com | HTTP 302 | Published · conformant |
| 131 | amazon-adsystem.com | HTTP 404 | Not probed |
| 134 | amazon.dev | Fetch failed · dns error | Not probed |
| 135 | nih.gov | Present | HTTP 404 |
| 136 | dns.google | HTTP 404 | HTTP 404 |
| 137 | kaspersky.com | Present | Published · conformant |
| 140 | dropbox.com | Present | Published · conformant |
| 141 | miit.gov.cn | HTTP 404 | Not probed |
| 142 | static.microsoft | Fetch failed · dns error | Not probed |
| 144 | reg.ru | Present | Published · conformant |
| 145 | intuit.com | Present | HTTP 400 |
| 146 | forms.gle | HTTP 400 | Not probed |
| 147 | archive.org | Present | HTTP 404 |
| 148 | epicgames.com | HTTP 301 | Not probed |
These are source-grounded views, not claims of one universal popularity truth. CrUX is a membership slice ordered here by Tranco rank; vendor lists retain their native order. See ranking methodology, normalization, and attribution.