Which sites let AI crawlers in but block Google?

1,314 domains in the Tranco top 1,000,000 shut Googlebot out while explicitly allowing at least one AI crawler, measured on 2026-09-07. That is the number worth quoting — and it is 52,131 before the deflation below.

Why the raw number is 52,131 and the real one is 1,314

The obvious query — Google blocked, an AI crawler explicitly allowed — returns 52,131 domains. 50,817 of them (97%) serve one of a small number of byte-identical files. Same group count, same row count, same disallow count, same allow-set, across tens of thousands of hosts. Those are generated networks, not crawler-policy decisions, and counting them would build a headline almost entirely out of spam infrastructure.

The exclusion is a stated mechanical rule, not a judgement about which sites look disreputable: a file signature shared by at least 1,000 distinct hosts is one template, counted once. A rule can be argued with. It also generalises — the second cluster below was found by the rule, not by anyone noticing it.

Is this just the tail of the web?

No. The independent cohort reaches the top of the panel, which is what makes it a decision rather than an artefact of abandoned domains.

Which sites are they?

The best-ranked 100 of the 1,314, each linking to its full measured profile. "Allows" lists only crawlers the file names with an explicit Allow — a bot the file never mentions is not counted as allowed, because under RFC 9309 §2.2.1 it falls to the * group, and that group is exactly what blocks Google here.

What this means if you run a site