Every AI bot · Allen Institute

ai2bot

Allen Institute's crawler that collects web pages to train AI models.

In robots.txt: User-agent: ai2bot

Named by1.7%of readable robots.txt files on the top 1M (10,135 of 600,979)

Of those, block it61.2%shut it out of the whole site (6,198 of 10,135)

Same sites, since CC-MAIN-2026-34+0.0 ptsshare of the same 394,690 sites that block it by name: 0.9% → 0.9%

Via Cloudflare's file1.7%of its blocks come from Cloudflare's ready-made robots.txt (108 of 6,198)

61.2%

33.9%

4.9%

Block the whole site: 6,198Block some paths: 3,438Name it but allow it: 499

Every robots.txt file on the top 1M that names ai2bot, by what the rules that apply to it say.

Allen Institute's other bots

Sites often treat one company's bots differently, depending on what each is for.

Bot	Purpose	Files naming it	Of those, block it
ai2bot-dolma	Collects pages to train AI	7,022	55.0%

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

Block some paths

Name it and allow it

mindbox.ru #1,932
vedomosti.ru #4,658
lachainemeteo.com #8,132
jasper.ai #12,259
belk.com #12,286

Lists skip sites we don't showcase (adult content).

What this means for you: blocking ai2bot asks Allen Institute not to collect your pages for AI training from now on. robots.txt only affects future visits, and it doesn't touch Allen Institute's other bots. Decide each bot on what it's for, not who runs it.