Every AI bot · Meta

Meta-ExternalFetcher

Meta's agent that fetches a page when someone asks its AI assistant about it.

In robots.txt: User-agent: Meta-ExternalFetcher

Named by1.3%of readable robots.txt files on the top 1M (7,983 of 600,979)

Of those, block it51.6%shut it out of the whole site (4,117 of 7,983)

Same sites, since CC-MAIN-2026-34+0.0 ptsshare of the same 394,690 sites that block it by name: 0.6% → 0.6%

Via Cloudflare's file3.1%of its blocks come from Cloudflare's ready-made robots.txt (128 of 4,117)

51.6%

34.3%

14.1%

Block the whole site: 4,117Block some paths: 2,739Name it but allow it: 1,127

Every robots.txt file on the top 1M that names Meta-ExternalFetcher, by what the rules that apply to it say.

Meta's other bots

Sites often treat one company's bots differently, depending on what each is for.

Bot	Purpose	Files naming it	Of those, block it
Meta-ExternalAgent	Collects pages to train AI	54,207	76.5%
facebookbot	Collects pages to train AI	16,505	60.2%

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

Block some paths

Name it and allow it

jotform.com #642
mindbox.ru #1,932
crunchbase.com #3,135
salon.com #3,819
udn.com #4,091

Lists skip sites we don't showcase (adult content).

What this means for you: blocking Meta-ExternalFetcher asks Meta's assistant not to fetch your pages when someone asks about them. Decide each bot on what it's for, not who runs it.