BotForYou Updated 9 October 2026 · Common Crawl CC-MAIN-2026-39

Platforms

Magento

522 sites on this month's Tranco top 1M whose robots.txt looks like Magento's, compared with all 453,286 sites Common Crawl saved.

How sure: of the 865 Magento guesses a server header or generator tag could check, 97.2% were right. A platform here is an informed guess, not a fact.

Magento sites against every site

Block GPTBot by name15.1%
every site10.1%
List a sitemap87.2%
every site66.7%
Name any AI crawler38.1%
every site20.1%
Use Cloudflare's ready-made file8.0%
every site5.9%
Tell every crawler to stay out0.6%
every site2.7%

7.3% of live Magento sites in our llms.txt sample publish one (3 of 41), against 13.7% of all live sampled sites.

Cloudflare stopped adding its ready-made file on 16 September 2026, so this month's Cloudflare row mixes files from before and after (what happened).

The biggest Magento sites we found

  1. cellphones.com.vn #8,471
  2. tokmanni.fi #19,260
  3. seasaltcornwall.com #62,622
  4. nen.nl #63,631
  5. jysk.ca #73,748

Highest-ranked sites whose Magento guess a server header or generator tag confirmed, skipping sites we don't showcase. Compare them side by side →