Data as of 2026-09-07 ·
robots.txt via Common Crawl
CC-MAIN-2026-34
+ our polite crawl · llms.txt via our polite crawl ·
Tranco list XN67N
(2026-09-03T22:00:02.533886)
· 1,000,000 panel domains
· crawls:
2026-09-07 · 2026-08-02 · 2026-07-25 · 2026-07-17
1,212,578 robots.txt fetches ·
740,113 parsed ·
56,498 llms.txt probes
The biggest and strangest robots.txt files on the web
The famous, the weird, and the extreme. Largest files (johnmu.com class), most directives, and content-type mismatches — where the served MIME type disagrees with the sniffed true type (a real conformance anomaly, free from Common Crawl's dual MIME columns).
What is the biggest robots.txt file?
| Host | Tranco rank | size | rows | Content-Type | >500 KiB? |
|---|---|---|---|---|---|
| matahukum.net | 553,915 | ≥ 5.0 MiB truncated | 9,195 | text/plain | yes |
| magsstore.com | 442,058 | ≥ 5.0 MiB truncated | 7,242 | text/plain | yes |
| idealbimbo.it | 796,802 | ≥ 5.0 MiB truncated | 9,401 | text/plain | yes |
| 716lavistacomplex.com | 921,487 | ≥ 5.0 MiB truncated | 7,969 | text/plain | yes |
| chem-jet.co.uk | 863,643 | ≥ 5.0 MiB truncated | 8,768 | text/plain | yes |
| myavne.co.il | 815,789 | 4.4 MiB | 4,919 | text/plain | yes |
| greatwuyi.com | 907,722 | 3.8 MiB | 6,139 | text/plain | yes |
| hangzhou.com.cn | 45,091 | 3.6 MiB | 6,966 | text/plain | yes |
| xuexi.cn | 13,987 | 3.5 MiB | 6,482 | text/plain | yes |
| academylinknetwork.com | 982,250 | 3.2 MiB | 8,396 | text/plain; charset=utf-8 | yes |
| maditakbooks.com | 906,555 | 2.8 MiB | 8,916 | text/plain | yes |
| primetimepremier.com | 984,156 | 2.6 MiB | 8,005 | text/plain | yes |
| energycity.vn | 685,572 | 2.5 MiB | 9,142 | text/plain; charset=UTF-8 | yes |
| premiumtownshipinjabalpur.in | 918,482 | 2.2 MiB | 7,112 | text/plain | yes |
| enviroent.in | 919,286 | 2.0 MiB | 9,145 | text/plain | yes |
| bhcosmetics.com | 525,310 | 2.0 MiB | 6,825 | text/html; charset=utf-8 | yes |
| astra-net.com | 656,142 | 2.0 MiB | 327 | text/html; charset=utf-8 | yes |
| jnedemure.com | 34,562 | 2.0 MiB | 367 | text/html; charset=UTF-8 | yes |
| wielkopolskiebilety.pl | 852,130 | 2.0 MiB | 925 | text/html;charset=UTF-8 | yes |
| exoarena.com | 227,806 | 2.0 MiB | 313 | text/html | yes |
| assentcompliance.eu | 899,039 | 2.0 MiB | 2,764 | text/html; charset=UTF-8 | yes |
| bcsgsrv.com | 939,909 | 1.9 MiB | 2,575 | text/html; charset=UTF-8 | yes |
| multichannelacd.de | 402,536 | 1.9 MiB | 6 | text/html; charset=utf-8 | yes |
| finom.de | 485,729 | 1.9 MiB | 2 | text/html; charset=utf-8 | yes |
| domainsatcost.ca | 38,492 | 1.9 MiB | 1 | text/html; charset=utf-8 | yes |
RFC 9309 requires parsers honor at least 500 KiB; files above that are truncated by Google. Our canonical size outlier is johnmu.com (~1.65 MB, simultaneously a valid WAV file and a valid robots.txt). Rows marked truncated hit Common Crawl's 5 MiB capture cap — the actual file is larger.
Which robots.txt has the most rules?
| Host | Tranco rank | disallow | allow | total rows |
|---|---|---|---|---|
| eventer.co.il | 381,089 | 27,653 | 0 | 27,657 |
| steepster.com | 118,806 | 23,568 | 0 | 23,588 |
| bricksdirect.at | 534,058 | 23,011 | 5 | 23,067 |
| bricksdirect.com | 477,198 | 23,011 | 5 | 23,067 |
| monlapinrose.com | 678,683 | 19,734 | 8 | 19,835 |
| boystore.com | 538,648 | 19,734 | 8 | 19,835 |
| modneduzerozmiary.pl | 514,447 | 18,571 | 8 | 18,620 |
| acompanhantesrosa.com | 760,503 | 0 | 17,258 | 17,259 |
| idrugstore.jp | 195,295 | 16,859 | 0 | 16,871 |
| millemercismariage.com | 655,342 | 16,324 | 0 | 16,325 |
| miniaturetoyshop.com | 813,086 | 16,198 | 1 | 16,202 |
| smashrun.com | 547,457 | 15,972 | 0 | 15,986 |
| eightsleep.com | 107,701 | 15,541 | 0 | 15,545 |
| avintage.com | 900,370 | 15,269 | 8 | 15,340 |
| frigelux-shop.com | 184,838 | 15,266 | 8 | 15,331 |
| psyh.ru | 938,747 | 14,694 | 0 | 14,700 |
| f71.com | 950,408 | 14,192 | 0 | 14,193 |
| hostedredmine.com | 500,744 | 13,440 | 0 | 13,487 |
| matbao.net | 231,148 | 13,390 | 1 | 13,393 |
| eporncam.com | 596,106 | 13,199 | 0 | 13,200 |
| glantier.com | 791,271 | 12,894 | 8 | 12,930 |
| boe.es | 4,618 | 12,145 | 1 | 13,897 |
| localplace.jp | 141,928 | 11,893 | 0 | 11,965 |
| teyuchiller.com | 961,713 | 11,626 | 0 | 11,628 |
| miltor.ru | 374,604 | 11,156 | 7 | 11,210 |
Which site addresses the most crawlers?
Files naming the most distinct crawler user-agents — a signal of operator sophistication (or blocklist maximalism). A robots.txt addressing hundreds of bots is making a deliberate statement about who may crawl.
| Host | Tranco rank | user-agents | groups |
|---|---|---|---|
| gopplay.com | 501,933 | 2,279 | 2,279 |
| fitnessfirst.com | 487,318 | 2,142 | 2 |
| pneumat.com.pl | 416,399 | 1,940 | 1,964 |
| findwallpapers.com | 711,270 | 1,843 | 1,904 |
| insertface.com | 624,005 | 1,837 | 1,894 |
| rolepages.com | 262,819 | 1,784 | 1,788 |
| gruppo-mossialos.com | 939,432 | 1,780 | 1,778 |
| ergosimansi.gr | 692,265 | 1,779 | 1,777 |
| digitalsignagetoday.com | 345,136 | 1,778 | 1,775 |
| retailcustomerexperience.com | 156,018 | 1,778 | 1,775 |
| pizzamarketplace.com | 948,421 | 1,778 | 1,775 |
| vendingtimes.com | 667,451 | 1,778 | 1,775 |
| athensauthenticmarathon.gr | 577,369 | 1,778 | 1,778 |
| fastcasual.com | 250,790 | 1,778 | 1,775 |
| kioskmarketplace.com | 475,090 | 1,778 | 1,775 |
| socialtradia.com | 383,631 | 1,778 | 1,780 |
| bantoa.com | 177,171 | 1,778 | 1,778 |
| atmmarketplace.com | 318,626 | 1,778 | 1,775 |
| qsrweb.com | 396,081 | 1,778 | 1,775 |
| passanatomic.gr | 822,985 | 1,778 | 1,776 |
| manthosoptical.gr | 359,802 | 1,778 | 1,776 |
| midepositodental.com | 606,451 | 1,777 | 1,779 |
| cyberfoxdigital.co.uk | 584,146 | 1,776 | 1,776 |
| daaveedee.com | 381,341 | 1,775 | 1,776 |
| aristeguinoticias.com | 40,101 | 1,775 | 1,774 |
How many sites serve robots.txt with the wrong content type?
Body parses as a valid robots.txt, but the served Content-Type isn't text/plain.
| Host | Tranco rank | served type | detected type |
|---|---|---|---|
| naver.com | 345 | application/octet-stream | text/x-robots |
| footmercato.net | 1,462 | txt | text/x-robots |
| kpmg.com | 2,961 | text/javascript | text/x-robots |
| lenta.com | 3,466 | application/octet-stream | text/x-robots |
| invisalign.com | 5,940 | text | text/x-robots |
| liputan6.com | 6,353 | text/txt; charset=UTF-8 | text/x-robots |
| forexfactory.com | 6,394 | text; charset=UTF-8 | text/x-robots |
| turkishairlines.com | 6,604 | document | text/x-robots |
| librarything.com | 6,846 | text/text;charset=UTF-8 | text/x-robots |
| hornbach.de | 7,293 | text/plain,text/plain | text/x-robots |
| clips4sale.com | 7,868 | text-plain | text/x-robots |
| ilbe.com | 8,593 | application/json | text/x-robots |
| ring.net | 12,777 | application/json | text/plain |
| regway.com | 13,008 | text | text/x-robots |
| jalbum.net | 13,412 | text/txt; charset=UTF-8 | text/x-robots |
| hostgator.in | 13,683 | text | text/x-robots |
| publishersweekly.com | 13,809 | text/text | text/x-robots |
| ua.es | 14,017 | text/css;charset=UTF-8 | text/x-robots |
| xcancel.com | 14,072 | application/octet-stream | text/x-robots |
| ip138.com | 14,525 | application/octet-stream | text/x-robots |
| nutaku.net | 14,913 | text | text/x-robots |
| nysed.gov | 15,519 | application/octet-stream | text/x-robots |
| cntd.ru | 15,644 | document | text/x-robots |
| ces.tech | 15,872 | text/text; charset=utf-8 | text/x-robots |
| tuxfamily.org | 16,736 | application/xhtml+xml; charset=utf-8 | text/plain |