Cloudflare

8,057 sites on this month's Tranco top 1M whose robots.txt looks like Cloudflare's, compared with all 453,286 sites Common Crawl saved.

How sure: of the 193,827 Cloudflare guesses a server header or generator tag could check, 100.0% were right. A platform here is an informed guess, not a fact.

Cloudflare sites against every site

Block GPTBot by name11.6%

every site10.1%

List a sitemap82.2%

every site66.7%

Name any AI crawler29.7%

every site20.1%

Use Cloudflare's ready-made file3.5%

every site5.9%

Tell every crawler to stay out0.2%

every site2.7%

25.9% of live Cloudflare sites in our llms.txt sample publish one (297 of 1,146), against 13.7% of all live sampled sites.

Cloudflare stopped adding its ready-made file on 16 September 2026, so this month's Cloudflare row mixes files from before and after (what happened).

The biggest Cloudflare sites we found

Highest-ranked sites whose Cloudflare guess a server header or generator tag confirmed, skipping sites we don't showcase. Compare them side by side →