Data as of 2026-07-25 ·
robots.txt via Common Crawl CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
· crawls:
2026-07-25 · 2026-07-17
687,794 robots.txt fetches ·
491,025 parsed ·
54,000 llms.txt probes
Outliers & Hall of Fame
The famous, the weird, and the extreme. Largest files (johnmu.com class), most directives, and content-type mismatches — where the served MIME type disagrees with the sniffed true type (a real conformance anomaly, free from Common Crawl's dual MIME columns).
Largest robots.txt files
| Host | Tranco rank | size | rows | Content-Type | >500 KiB? |
|---|---|---|---|---|---|
| chem-jet.co.uk | 619,540 | ≥ 5.0 MiB truncated | 8,768 | text/plain | yes |
| magsstore.com | 406,971 | ≥ 5.0 MiB truncated | 7,242 | text/plain | yes |
| matahukum.net | 553,915 | ≥ 5.0 MiB truncated | 9,195 | text/plain | yes |
| idealbimbo.it | 785,864 | ≥ 5.0 MiB truncated | 9,401 | text/plain | yes |
| solox.gg | 903,137 | ≥ 5.0 MiB truncated | 9,977 | text/plain; charset=UTF-8 | yes |
| 716lavistacomplex.com | 921,487 | ≥ 5.0 MiB truncated | 7,969 | text/plain | yes |
| myavne.co.il | 803,798 | 4.4 MiB | 4,919 | text/plain | yes |
| greatwuyi.com | 907,722 | 3.8 MiB | 6,139 | text/plain | yes |
| hangzhou.com.cn | 46,573 | 3.6 MiB | 6,966 | text/plain | yes |
| xuexi.cn | 13,970 | 3.5 MiB | 6,482 | text/plain | yes |
| academylinknetwork.com | 982,250 | 2.9 MiB | 8,396 | text/plain; charset=utf-8 | yes |
| maditakbooks.com | 906,555 | 2.8 MiB | 8,916 | text/plain | yes |
| primetimepremier.com | 984,156 | 2.6 MiB | 8,005 | text/plain | yes |
| energycity.vn | 685,572 | 2.5 MiB | 9,142 | text/plain; charset=UTF-8 | yes |
| premiumtownshipinjabalpur.in | 918,482 | 2.2 MiB | 7,112 | text/plain | yes |
| enviroent.in | 919,286 | 2.0 MiB | 9,145 | text/plain | yes |
| motorola.com.br | 57,623 | 2.0 MiB | 235 | text/html; charset=utf-8 | no |
| bigjoy.com.tr | 160,919 | 2.0 MiB | 10 | text/html;charset=utf-8 | yes |
| kipling.com.br | 470,022 | 2.0 MiB | 210 | text/html; charset=utf-8 | yes |
| ebates.com | 78,159 | 1.9 MiB | 1 | text/html; charset=utf-8 | yes |
| qpr.co.uk | 173,320 | 1.8 MiB | 19 | text/html | yes |
| vikasnistore.in | 628,680 | 1.8 MiB | 8,667 | text/plain | yes |
| islandrecords.co.uk | 695,050 | 1.7 MiB | 7,119 | text/html; charset=utf-8 | yes |
| canadasolar.com.vn | 960,253 | 1.7 MiB | 8,389 | text/plain; charset=UTF-8 | yes |
| netscape.com | 2,413 | 1.7 MiB | 99 | text/html; charset=utf-8 | yes |
RFC 9309 requires parsers honor at least 500 KiB; files above that are truncated by Google. Our canonical size outlier is johnmu.com (~1.65 MB, simultaneously a valid WAV file and a valid robots.txt). Rows marked truncated hit Common Crawl's 5 MiB capture cap — the actual file is larger.
Most directives
| Host | Tranco rank | disallow | allow | total rows |
|---|---|---|---|---|
| eventer.co.il | 176,232 | 27,653 | 0 | 27,657 |
| steepster.com | 102,435 | 23,568 | 0 | 23,588 |
| steepster.com | 102,435 | 23,568 | 0 | 23,588 |
| monlapinrose.com | 556,594 | 19,734 | 8 | 19,835 |
| modneduzerozmiary.pl | 212,205 | 18,571 | 8 | 18,620 |
| urwatools.com | 86,891 | 17,601 | 1 | 17,624 |
| acompanhantesrosa.com | 751,405 | 0 | 17,373 | 17,374 |
| idrugstore.jp | 193,853 | 16,901 | 0 | 16,913 |
| climadiff.com | 463,859 | 15,970 | 9 | 16,094 |
| climadiff.com | 463,859 | 15,970 | 9 | 16,095 |
| smashrun.com | 513,682 | 15,978 | 0 | 15,992 |
| millemercismariage.com | 631,282 | 15,528 | 0 | 15,529 |
| lclark.edu | 46,862 | 14,698 | 0 | 14,701 |
| f71.com | 388,495 | 14,192 | 0 | 14,193 |
| hostedredmine.com | 720,938 | 13,440 | 0 | 13,487 |
| hostedredmine.com | 720,938 | 13,440 | 0 | 13,487 |
| matbao.net | 155,204 | 13,391 | 1 | 13,393 |
| eporncam.com | 563,890 | 13,199 | 0 | 13,200 |
| eightsleep.com | 76,036 | 13,053 | 0 | 13,057 |
| fucolle.com | 22,840 | 13,032 | 0 | 13,034 |
| glantier.com | 223,641 | 12,894 | 8 | 12,930 |
| boe.es | 4,171 | 11,989 | 0 | 13,717 |
| localplace.jp | 83,187 | 11,957 | 0 | 11,965 |
| teyuchiller.com | 587,123 | 11,626 | 0 | 11,628 |
| miltor.ru | 203,824 | 11,153 | 7 | 11,207 |
Most user-agents addressed
Files naming the most distinct crawler user-agents — a signal of operator sophistication (or blocklist maximalism). A robots.txt addressing hundreds of bots is making a deliberate statement about who may crawl.
| Host | Tranco rank | user-agents | groups |
|---|---|---|---|
| fitnessfirst.com | 492,707 | 2,142 | 2 |
| fitnessfirst.com | 492,707 | 2,142 | 2 |
| pneumat.com.pl | 381,036 | 1,946 | 1,970 |
| pneumat.com.pl | 381,036 | 1,940 | 1,964 |
| insertface.com | 632,555 | 1,837 | 1,894 |
| insertface.com | 632,555 | 1,837 | 1,894 |
| rolepages.com | 261,371 | 1,784 | 1,788 |
| rolepages.com | 261,371 | 1,784 | 1,788 |
| spectrumculture.com | 439,780 | 1,783 | 1,785 |
| retailcustomerexperience.com | 145,806 | 1,778 | 1,775 |
| qsrweb.com | 428,755 | 1,778 | 1,775 |
| fastcasual.com | 233,673 | 1,778 | 1,775 |
| digitalsignagetoday.com | 343,145 | 1,778 | 1,775 |
| vendingtimes.com | 599,396 | 1,778 | 1,775 |
| athensauthenticmarathon.gr | 601,480 | 1,778 | 1,778 |
| kioskmarketplace.com | 451,243 | 1,778 | 1,775 |
| bantoa.com | 157,995 | 1,778 | 1,778 |
| pizzamarketplace.com | 696,026 | 1,778 | 1,775 |
| atmmarketplace.com | 331,406 | 1,778 | 1,775 |
| retailcustomerexperience.com | 145,806 | 1,778 | 1,775 |
| bantoa.com | 157,995 | 1,778 | 1,778 |
| fastcasual.com | 233,673 | 1,778 | 1,775 |
| atmmarketplace.com | 331,406 | 1,778 | 1,775 |
| digitalsignagetoday.com | 343,145 | 1,778 | 1,775 |
| socialtradia.com | 374,384 | 1,778 | 1,780 |
Content-Type mismatches
Body parses as a valid robots.txt, but the served Content-Type isn't text/plain.
| Host | Tranco rank | served type | detected type |
|---|---|---|---|
| ghcr.io | 4,925 | application/json | text/plain |
| turkishairlines.com | 6,098 | document | text/x-robots |
| footmercato.net | 6,128 | txt | text/x-robots |
| liputan6.com | 6,320 | text/txt; charset=UTF-8 | text/x-robots |
| forexfactory.com | 6,361 | text; charset=UTF-8 | text/x-robots |
| midasbuy.com | 6,808 | application/javascript | text/x-robots |
| hornbach.de | 7,263 | text/plain,text/plain | text/x-robots |
| clips4sale.com | 8,264 | text-plain | text/x-robots |
| ilbe.com | 8,942 | application/json | text/x-robots |
| regway.com | 11,029 | text | text/x-robots |
| xiaomiwear.com | 11,896 | application/octet-stream | text/plain |
| publishersweekly.com | 13,788 | text/text | text/x-robots |
| nutaku.net | 14,886 | text | text/x-robots |
| ces.tech | 15,439 | text/text; charset=utf-8 | text/x-robots |
| hostgator.in | 16,408 | text | text/x-robots |
| productmadness.com | 16,873 | application/json | text/x-robots |
| smrtb.com | 17,429 | application/octet-stream | text/plain |
| alta.ru | 18,311 | application/download | text/x-robots |
| jugantor.com | 18,864 | text/txt; charset=UTF-8 | text/x-robots |
| eatright.org | 18,895 | text; charset=utf-8 | text/x-robots |
| teamsystem.com | 19,052 | text/txt; charset=utf-8 | text/x-robots |
| bola.com | 19,078 | text/txt; charset=UTF-8 | text/x-robots |
| virgool.io | 19,411 | document | text/x-robots |
| hornbach.nl | 19,652 | text/plain,text/plain | text/x-robots |
| mailfence.com | 19,963 | charset=utf-8;charset=UTF-8 | text/x-robots |