BotForYou Updated 9 October 2026 · Common Crawl CC-MAIN-2026-39

Every AI bot · img2dataset

img2dataset

img2dataset's crawler that collects web pages to train AI models.

In robots.txt: User-agent: img2dataset

The numbers

Named by1.2%of readable robots.txt files on the top 1M (7,369 of 600,979)
Of those, block it57.4%shut it out of the whole site (4,229 of 7,369)
Same sites, since CC-MAIN-2026-34+0.0 ptsshare of the same 394,690 sites that block it by name: 0.7% → 0.8%
Via Cloudflare's file1.9%of its blocks come from Cloudflare's ready-made robots.txt (79 of 4,229)
Block the whole site: 4,229Block some paths: 3,038Name it but allow it: 102
Every robots.txt file on the top 1M that names img2dataset, by what the rules that apply to it say.

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

  1. amazon.com #24
  2. yahoo.com #59
  3. msn.com #69
  4. chatgpt.com #75
  5. amazonvideo.com #123

Block some paths

  1. indeed.com #371
  2. squarespace.com #544
  3. deezer.com #1,828
  4. web.de #2,479
  5. gmx.net #2,498

Name it and allow it

  1. lachainemeteo.com #8,132
  2. zinio.com #16,277
  3. kling.ai #17,121
  4. biu.ac.il #29,152
  5. meteoconsult.fr #33,558

Lists skip sites we don't showcase (adult content).

What this means for you: blocking img2dataset asks img2dataset not to collect your pages for AI training from now on. robots.txt only affects future visits, and it doesn't touch img2dataset's other bots. Decide each bot on what it's for, not who runs it.

Counts use one robots.txt per site on this month's Tranco top 1M (600,979 files we could read). Bot list: registry 2026-07-14.1, kept in sync with Dark Visitors and ai.robots.txt.