Platforms
What each platform ships
Many sites never write their own robots.txt: their platform does. We infer each site's platform from its robots.txt rules and its server's response headers, then compare what those sites say with every site Common Crawl saved this month.
A platform here is an informed guess, not a fact. Platforms whose guesses fail checks against the server's own headers too often are withheld.
| Platform | Sites | Block GPTBot | Sitemap | llms.txt | Checked out |
|---|---|---|---|---|---|
| Every site Common Crawl saved | 453,286 | 10.1% | 66.7% | 13.7% | — |
| WordPress | 62,854 | 7.9% | 85.0% | 21.4% | 96.9% |
| Shopify | 24,891 | 1.5% | 98.3% | 93.8% | 88.4% |
| Drupal withheld | Only 33.6% of 3,521 checkable guesses were right, below our 80.0% bar, so its figures would mostly describe other platforms. | 33.6% | |||
| Cloudflare | 8,057 | 11.6% | 82.2% | 25.9% | 100.0% |
| Joomla | 5,588 | 7.0% | 28.3% | 6.8% | 82.7% |
| Wix | 1,218 | 0.8% | 99.2% | 91.8% | 100.0% |
| Squarespace | 1,073 | 3.1% | 99.8% | 7.1% | 96.5% |
| Next.js | 973 | 8.5% | 96.2% | 45.3% | 99.6% |
| Magento | 522 | 15.1% | 87.2% | 7.3% | 97.2% |
| Ghost | 12 | 33.3% | 83.3% | — | 100.0% |
"llms.txt" is the share of live sites in our llms.txt sample that publish one. "Checked out" is the share of guesses a server header or generator tag could confirm or contradict that it confirmed. A site counts as a platform when that guess reaches 80.0% confidence. llms.txt shows "—" below 30 live sampled sites. Robots.txt files Common Crawl saved for this month's Tranco top 1M that we could read; a platform is inferred from each file's rules and the server's response headers.