BotForYou Updated 9 October 2026 · Common Crawl CC-MAIN-2026-39

Platforms

Next.js

973 sites on this month's Tranco top 1M whose robots.txt looks like Next.js's, compared with all 453,286 sites Common Crawl saved.

How sure: of the 6,287 Next.js guesses a server header or generator tag could check, 99.6% were right. A platform here is an informed guess, not a fact.

Next.js sites against every site

Block GPTBot by name8.5%
every site10.1%
List a sitemap96.2%
every site66.7%
Name any AI crawler33.8%
every site20.1%
Use Cloudflare's ready-made file1.6%
every site5.9%
Tell every crawler to stay out0.9%
every site2.7%

45.3% of live Next.js sites in our llms.txt sample publish one (62 of 137), against 13.7% of all live sampled sites.

Cloudflare stopped adding its ready-made file on 16 September 2026, so this month's Cloudflare row mixes files from before and after (what happened).

The biggest Next.js sites we found

  1. here.com #2,396
  2. venturebeat.com #2,887
  3. weightwatchers.com #10,827
  4. modelcontextprotocol.io #12,612
  5. hellomagazine.com #13,279

Highest-ranked sites whose Next.js guess a server header or generator tag confirmed, skipping sites we don't showcase. Compare them side by side →