Every AI bot · Webz.io

omgili

Webz.io's crawler that collects web pages to train AI models.

In robots.txt: User-agent: omgili

Named by2.9%of readable robots.txt files on the top 1M (17,288 of 600,979)

Of those, block it79.6%shut it out of the whole site (13,763 of 17,288)

Same sites, since CC-MAIN-2026-34+0.1 ptsshare of the same 394,690 sites that block it by name: 1.5% → 1.5%

Via Cloudflare's file1.6%of its blocks come from Cloudflare's ready-made robots.txt (223 of 13,763)

79.6%

18.5%

1.9%

Block the whole site: 13,763Block some paths: 3,198Name it but allow it: 327

Every robots.txt file on the top 1M that names omgili, by what the rules that apply to it say.

Webz.io's other bots

Sites often treat one company's bots differently, depending on what each is for.

Bot	Purpose	Files naming it	Of those, block it
omgilibot	Collects pages to train AI	16,565	77.7%
webzio-extended	Collects pages to train AI	6,507	68.4%

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

Block some paths

squarespace.com #544
jimdo.com #1,131
deezer.com #1,828
volvocars.com #3,936
20min.ch #4,977

Name it and allow it

mgid.com #2,112
vedomosti.ru #4,658
amarujala.com #7,234
adskeeper.com #10,676
belk.com #12,286

Lists skip sites we don't showcase (adult content).

What this means for you: blocking omgili asks Webz.io not to collect your pages for AI training from now on. robots.txt only affects future visits, and it doesn't touch Webz.io's other bots. Decide each bot on what it's for, not who runs it.