BotForYou Updated 9 October 2026 · Common Crawl CC-MAIN-2026-39

Every AI bot · Amazon

Amazonbot

Amazon's agent that fetches a page when someone asks its AI assistant about it.

In robots.txt: User-agent: Amazonbot

The numbers

Named by10.0%of readable robots.txt files on the top 1M (60,353 of 600,979)
Of those, block it81.6%shut it out of the whole site (49,240 of 60,353)
Same sites, since CC-MAIN-2026-34−0.6 ptsshare of the same 394,690 sites that block it by name: 9.0% → 8.4%
Via Cloudflare's file54.8%of its blocks come from Cloudflare's ready-made robots.txt (26,968 of 49,240)Cloudflare stopped adding that file on 16 September 2026, so this month mixes before and after. What happened
Block the whole site: 49,240Block some paths: 6,959Name it but allow it: 4,154
Every robots.txt file on the top 1M that names Amazonbot, by what the rules that apply to it say.

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

  1. instagram.com #11
  2. fbcdn.net #13
  3. whatsapp.com #55
  4. msn.com #69
  5. snapchat.com #130

Block some paths

  1. facebook.com #3
  2. nytimes.com #159
  3. oracle.com #205
  4. imdb.com #265
  5. linktr.ee #272

Name it and allow it

  1. twitch.tv #221
  2. forter.com #270
  3. sophos.com #714
  4. theverge.com #846
  5. repubblica.it #965

Lists skip sites we don't showcase (adult content).

What this means for you: blocking Amazonbot asks Amazon's assistant not to fetch your pages when someone asks about them. Decide each bot on what it's for, not who runs it.

Counts use one robots.txt per site on this month's Tranco top 1M (600,979 files we could read). Bot list: registry 2026-07-14.1, kept in sync with Dark Visitors and ai.robots.txt.