Data as of 2026-08-02 ·
robots.txt via Common Crawl
CC-MAIN-2026-30
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list L5684
(2026-07-14)
· 1,000,000 panel domains
· crawls:
2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17
1,000,000 robots.txt fetches ·
608,834 parsed ·
55,770 llms.txt probes
The biggest and strangest robots.txt files on the web
The famous, the weird, and the extreme. Largest files (johnmu.com class), most directives, and content-type mismatches — where the served MIME type disagrees with the sniffed true type (a real conformance anomaly, free from Common Crawl's dual MIME columns).
What is the biggest robots.txt file?
| Host | Tranco rank | size | rows | Content-Type | >500 KiB? |
|---|---|---|---|---|---|
| 716lavistacomplex.com | 921,487 | ≥ 5.0 MiB truncated | 7,969 | text/plain | yes |
| solox.gg | 903,137 | ≥ 5.0 MiB truncated | 9,977 | text/plain; charset=UTF-8 | yes |
| chem-jet.co.uk | 619,540 | ≥ 5.0 MiB truncated | 8,768 | text/plain | yes |
| magsstore.com | 406,971 | ≥ 5.0 MiB truncated | 7,242 | text/plain | yes |
| idealbimbo.it | 785,864 | ≥ 5.0 MiB truncated | 9,401 | text/plain | yes |
| myavne.co.il | 803,798 | 4.4 MiB | 4,919 | text/plain | yes |
| greatwuyi.com | 907,722 | 3.8 MiB | 6,139 | text/plain | yes |
| hangzhou.com.cn | 46,573 | 3.6 MiB | 6,966 | text/plain | yes |
| xuexi.cn | 13,970 | 3.5 MiB | 6,482 | text/plain | yes |
| academylinknetwork.com | 982,250 | 3.2 MiB | 8,396 | text/plain; charset=utf-8 | yes |
| maditakbooks.com | 906,555 | 2.8 MiB | 8,916 | text/plain | yes |
| primetimepremier.com | 984,156 | 2.6 MiB | 8,005 | text/plain | yes |
| energycity.vn | 685,572 | 2.5 MiB | 9,142 | text/plain; charset=UTF-8 | yes |
| premiumtownshipinjabalpur.in | 918,482 | 2.2 MiB | 7,112 | text/plain | yes |
| enviroent.in | 919,286 | 2.0 MiB | 9,145 | text/plain | yes |
| squarepegbrewerks.com | 898,222 | 2.0 MiB | 324 | text/html; charset=utf-8 | yes |
| wielkopolskiebilety.pl | 838,254 | 2.0 MiB | 929 | text/html;charset=UTF-8 | yes |
| exoarena.com | 227,806 | 2.0 MiB | 314 | text/html | yes |
| assentcompliance.eu | 972,640 | 1.9 MiB | 2,592 | text/html; charset=UTF-8 | yes |
| bcsgsrv.com | 943,194 | 1.9 MiB | 2,572 | text/html; charset=UTF-8 | yes |
| univerusrec.com | 594,767 | 1.9 MiB | 1,632 | text/html; charset=UTF-8 | yes |
| hboespana.com | 740,941 | 1.9 MiB | 3,445 | text/html; charset=utf-8 | yes |
| multichannelacd.de | 293,216 | 1.9 MiB | 7 | text/html; charset=utf-8 | yes |
| shopware.store | 203,456 | 1.9 MiB | 10 | text/html;charset=utf-8 | yes |
| plexvideos.com | 64,709 | 1.9 MiB | 1 | text/html; charset=utf-8 | yes |
RFC 9309 requires parsers honor at least 500 KiB; files above that are truncated by Google. Our canonical size outlier is johnmu.com (~1.65 MB, simultaneously a valid WAV file and a valid robots.txt). Rows marked truncated hit Common Crawl's 5 MiB capture cap — the actual file is larger.
Which robots.txt has the most rules?
| Host | Tranco rank | disallow | allow | total rows |
|---|---|---|---|---|
| eventer.co.il | 176,232 | 27,653 | 0 | 27,657 |
| steepster.com | 102,435 | 23,568 | 0 | 23,588 |
| bricksdirect.at | 534,058 | 23,011 | 5 | 23,067 |
| monlapinrose.com | 556,594 | 19,734 | 8 | 19,835 |
| urwatools.com | 86,891 | 19,114 | 0 | 19,119 |
| modneduzerozmiary.pl | 212,205 | 18,571 | 8 | 18,620 |
| acompanhantesrosa.com | 751,405 | 0 | 17,367 | 17,368 |
| idrugstore.jp | 193,853 | 16,874 | 0 | 16,886 |
| smashrun.com | 513,682 | 15,976 | 0 | 15,990 |
| millemercismariage.com | 631,282 | 15,927 | 0 | 15,928 |
| eightsleep.com | 76,036 | 14,701 | 0 | 14,705 |
| psyh.ru | 938,747 | 14,694 | 0 | 14,700 |
| f71.com | 388,495 | 14,192 | 0 | 14,193 |
| hostedredmine.com | 720,938 | 13,440 | 0 | 13,487 |
| matbao.net | 155,204 | 13,391 | 1 | 13,393 |
| eporncam.com | 563,890 | 13,199 | 0 | 13,200 |
| glantier.com | 223,641 | 12,894 | 8 | 12,930 |
| fucolle.com | 22,840 | 12,226 | 0 | 12,228 |
| boe.es | 4,171 | 12,065 | 0 | 13,804 |
| localplace.jp | 83,187 | 11,957 | 0 | 11,965 |
| teyuchiller.com | 587,123 | 11,626 | 0 | 11,628 |
| miltor.ru | 203,824 | 11,155 | 7 | 11,209 |
| edilivre.com | 413,802 | 11,064 | 2 | 11,083 |
| bigmat.it | 598,127 | 11,001 | 5 | 11,204 |
| cagri.com | 409,374 | 10,749 | 1 | 10,751 |
Which site addresses the most crawlers?
Files naming the most distinct crawler user-agents — a signal of operator sophistication (or blocklist maximalism). A robots.txt addressing hundreds of bots is making a deliberate statement about who may crawl.
| Host | Tranco rank | user-agents | groups |
|---|---|---|---|
| gopplay.com | 501,933 | 2,241 | 2,241 |
| fitnessfirst.com | 492,707 | 2,142 | 2 |
| pneumat.com.pl | 381,036 | 1,941 | 1,965 |
| insertface.com | 632,555 | 1,837 | 1,894 |
| atmmarketplace.com | 331,406 | 1,778 | 1,775 |
| fastcasual.com | 233,673 | 1,778 | 1,775 |
| vendingtimes.com | 599,396 | 1,778 | 1,775 |
| athensauthenticmarathon.gr | 601,480 | 1,778 | 1,778 |
| retailcustomerexperience.com | 145,806 | 1,778 | 1,775 |
| pizzamarketplace.com | 696,026 | 1,778 | 1,775 |
| bantoa.com | 157,995 | 1,778 | 1,778 |
| socialtradia.com | 374,384 | 1,778 | 1,780 |
| digitalsignagetoday.com | 343,145 | 1,778 | 1,775 |
| qsrweb.com | 428,755 | 1,778 | 1,775 |
| kioskmarketplace.com | 451,243 | 1,778 | 1,775 |
| midepositodental.com | 392,703 | 1,777 | 1,779 |
| aristeguinoticias.com | 39,823 | 1,775 | 1,774 |
| spitogatos.gr | 38,948 | 1,775 | 1,781 |
| cyberfoxdigital.co.uk | 386,806 | 1,775 | 1,775 |
| picanto.com | 640,046 | 1,775 | 1,775 |
| walletinvestor.com | 51,478 | 1,774 | 1,774 |
| spiti24.gr | 61,377 | 1,774 | 1,774 |
| mediabrief.com | 262,722 | 1,774 | 1,774 |
| tospitimou.gr | 174,785 | 1,774 | 1,774 |
| crozilla.com | 558,453 | 1,774 | 1,777 |
How many sites serve robots.txt with the wrong content type?
Body parses as a valid robots.txt, but the served Content-Type isn't text/plain.
| Host | Tranco rank | served type | detected type |
|---|---|---|---|
| naver.com | 351 | application/octet-stream | text/x-robots |
| kpmg.com | 2,890 | text/javascript | text/x-robots |
| micpn.com | 4,836 | application/octet-stream | text/x-robots |
| ghcr.io | 4,925 | application/json | text/plain |
| turkishairlines.com | 6,098 | document | text/x-robots |
| invisalign.com | 6,108 | text | text/x-robots |
| footmercato.net | 6,128 | txt | text/x-robots |
| liputan6.com | 6,320 | text/txt; charset=UTF-8 | text/x-robots |
| forexfactory.com | 6,361 | text; charset=UTF-8 | text/x-robots |
| funda.nl | 6,465 | application/json | text/x-robots |
| hornbach.de | 7,263 | text/plain,text/plain | text/x-robots |
| clips4sale.com | 8,264 | text-plain | text/x-robots |
| ilbe.com | 8,942 | application/json | text/x-robots |
| regway.com | 11,029 | text | text/x-robots |
| librarything.com | 11,035 | text/text;charset=UTF-8 | text/x-robots |
| jalbum.net | 13,466 | text/txt; charset=UTF-8 | text/x-robots |
| ua.es | 13,606 | text/css;charset=UTF-8 | text/x-robots |
| publishersweekly.com | 13,788 | text/text | text/x-robots |
| xcancel.com | 14,436 | application/octet-stream | text/x-robots |
| ip138.com | 14,503 | application/octet-stream | text/x-robots |
| nutaku.net | 14,886 | text | text/x-robots |
| ces.tech | 15,439 | text/text; charset=utf-8 | text/x-robots |
| cntd.ru | 15,904 | document | text/x-robots |
| bigrock.com | 16,034 | text | text/x-robots |
| nysed.gov | 16,278 | application/octet-stream | text/x-robots |