WordPress

62,854 sites on this month's Tranco top 1M whose robots.txt looks like WordPress's, compared with all 453,286 sites Common Crawl saved.

How sure: of the 9,119 WordPress guesses a server header or generator tag could check, 96.9% were right. A platform here is an informed guess, not a fact.

WordPress sites against every site

Block GPTBot by name7.9%

every site10.1%

List a sitemap85.0%

every site66.7%

Name any AI crawler14.5%

every site20.1%

Use Cloudflare's ready-made file4.6%

every site5.9%

Tell every crawler to stay out0.3%

every site2.7%

21.4% of live WordPress sites in our llms.txt sample publish one (1,128 of 5,282), against 13.7% of all live sampled sites.

Cloudflare stopped adding its ready-made file on 16 September 2026, so this month's Cloudflare row mixes files from before and after (what happened).

The biggest WordPress sites we found

Highest-ranked sites whose WordPress guess a server header or generator tag confirmed, skipping sites we don't showcase. Compare them side by side →