Ghost

12 sites on this month's Tranco top 1M whose robots.txt looks like Ghost's, compared with all 453,286 sites Common Crawl saved.

How sure: of the 35 Ghost guesses a server header or generator tag could check, 100.0% were right. A platform here is an informed guess, not a fact.

Ghost sites against every site

Block GPTBot by name33.3%

every site10.1%

List a sitemap83.3%

every site66.7%

Name any AI crawler50.0%

every site20.1%

Use Cloudflare's ready-made file8.3%

every site5.9%

Tell every crawler to stay out0.0%

every site2.7%

Too few Ghost sites in this month's llms.txt sample to give a rate (3 live).

Cloudflare stopped adding its ready-made file on 16 September 2026, so this month's Cloudflare row mixes files from before and after (what happened).

Highest-ranked sites whose Ghost guess a server header or generator tag confirmed, skipping sites we don't showcase. Compare them side by side →