BotForYou Updated 9 October 2026 · Common Crawl CC-MAIN-2026-39

Every AI bot · Huawei

petalbot

Huawei's crawler that builds an index for AI search answers.

In robots.txt: User-agent: petalbot

The numbers

Named by7.8%of readable robots.txt files on the top 1M (46,881 of 600,979)
Of those, block it40.4%shut it out of the whole site (18,917 of 46,881)
Same sites, since CC-MAIN-2026-34+0.1 ptsshare of the same 394,690 sites that block it by name: 3.1% → 3.2%
Via Cloudflare's file2.9%of its blocks come from Cloudflare's ready-made robots.txt (551 of 18,917)
Block the whole site: 18,917Block some paths: 2,699Name it but allow it: 25,265
Every robots.txt file on the top 1M that names petalbot, by what the rules that apply to it say.

Who does what

The highest-ranked sites on the top million in each group. Click one to see its whole robots.txt verdict.

Block it

  1. facebook.com #3
  2. instagram.com #11
  3. fbcdn.net #13
  4. amazon.com #24
  5. tiktok.com #54

Block some paths

  1. flickr.com #189
  2. indeed.com #371
  3. ted.com #795
  4. disneyplus.com #809
  5. jimdo.com #1,131

Name it and allow it

  1. patreon.com #622
  2. 18comic.vip #1,517
  3. photobucket.com #2,281
  4. france24.com #2,905
  5. spglobal.com #3,490

Lists skip sites we don't showcase (adult content).

What this means for you: blocking petalbot can keep your pages out of Huawei's AI search answers, including any links back to you. Decide each bot on what it's for, not who runs it.

Counts use one robots.txt per site on this month's Tranco top 1M (600,979 files we could read). Bot list: registry 2026-07-14.1, kept in sync with Dark Visitors and ai.robots.txt.