GPTBot

OpenAI's crawler that collects web pages to train AI models.

In robots.txt: User-agent: GPTBot

Named by13.0%of readable robots.txt files on the top 1M (78,270 of 600,979)

Of those, block it71.9%shut it out of the whole site (56,270 of 78,270)

Same sites, since CC-MAIN-2026-34−0.6 ptsshare of the same 394,690 sites that block it by name: 10.3% → 9.7%

Via Cloudflare's file47.9%of its blocks come from Cloudflare's ready-made robots.txt (26,935 of 56,270)Cloudflare stopped adding that file on 16 September 2026, so this month mixes before and after. What happened

71.9%

13.5%

14.6%

Block the whole site: 56,270Block some paths: 10,534Name it but allow it: 11,466

Every robots.txt file on the top 1M that names GPTBot, by what the rules that apply to it say.

OpenAI's other bots

Sites often treat one company's bots differently, depending on what each is for.

Bot	Purpose	Files naming it	Of those, block it
ChatGPT-User	Fetches a page for a chatbot user	34,242	48.3%
OAI-SearchBot	Builds an AI search index	23,100	25.7%

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

Block some paths

Name it and allow it

Lists skip sites we don't showcase (adult content).

What this means for you: blocking GPTBot asks OpenAI not to collect your pages for AI training from now on. robots.txt only affects future visits, and it doesn't touch OpenAI's other bots. Decide each bot on what it's for, not who runs it.