Data as of 2026-07-17 ·
robots.txt via Common Crawl CC-MAIN-2026-25
+ our polite crawl · llms.txt via our polite crawl ·
Tranco top-1,000,000
· crawls:
2026-07-25 · 2026-07-17
746,000 robots.txt fetches ·
574,664 parsed ·
54,000 llms.txt probes
Outliers & Hall of Fame
The famous, the weird, and the extreme. Largest files (johnmu.com class), most directives, and content-type mismatches — where the served MIME type disagrees with the sniffed true type (a real conformance anomaly, free from Common Crawl's dual MIME columns).
Largest robots.txt files
| Host | Tranco rank | size | rows | Content-Type | >500 KiB? |
|---|---|---|---|---|---|
| chem-jet.co.uk | 619,540 | ≥ 5.0 MiB truncated | 8,768 | text/plain | yes |
| sheetz.com | 105,771 | ≥ 5.0 MiB truncated | 4 | text/html; charset=utf-8 | yes |
| magsstore.com | 406,971 | ≥ 5.0 MiB truncated | 7,242 | text/plain | yes |
| matahukum.net | 553,915 | ≥ 5.0 MiB truncated | 9,195 | text/plain | yes |
| idealbimbo.it | 785,864 | ≥ 5.0 MiB truncated | 9,401 | text/plain | yes |
| architecture.org | 134,187 | ≥ 5.0 MiB truncated | 2 | text/html;charset=utf-8 | yes |
| solox.gg | 903,137 | ≥ 5.0 MiB truncated | 9,977 | text/plain; charset=UTF-8 | yes |
| getlightcurve.net | 967,545 | ≥ 5.0 MiB truncated | 6,650 | — | yes |
| 716lavistacomplex.com | 921,487 | ≥ 5.0 MiB truncated | 7,969 | text/plain | yes |
| myavne.co.il | 803,798 | 4.4 MiB | 4,919 | text/plain | yes |
| takdin.co.il | 659,215 | 4.2 MiB | 4,842 | text/plain; charset=UTF-8 | yes |
| greatwuyi.com | 907,722 | 3.8 MiB | 6,139 | text/plain | yes |
| cdn-surfline.com | 57,307 | 3.6 MiB | 20,949 | text/plain | yes |
| hs-emden-leer.de | 377,332 | 3.5 MiB | 5 | text/html; charset=utf-8 | yes |
| academylinknetwork.com | 982,250 | 2.9 MiB | 8,396 | text/plain; charset=utf-8 | yes |
| easy-feedback.com | 286,637 | 2.8 MiB | 19,651 | text/plain;charset=UTF-8 | yes |
| easy-feedback.de | 196,178 | 2.8 MiB | 19,651 | text/plain;charset=UTF-8 | yes |
| maditakbooks.com | 906,555 | 2.8 MiB | 8,916 | text/plain | yes |
| primetimepremier.com | 984,156 | 2.6 MiB | 8,005 | text/plain | yes |
| energycity.vn | 685,572 | 2.5 MiB | 9,142 | text/plain; charset=UTF-8 | yes |
| padigitale2026.gov.it | 991,630 | 2.5 MiB | 3 | text/html; charset=utf-8 | yes |
| yamahamotorsports.com | 87,810 | 2.5 MiB | 6 | text/html; charset=UTF-8 | yes |
| taostats.io | 145,002 | 2.3 MiB | 11 | text/html; charset=utf-8 | yes |
| premiumtownshipinjabalpur.in | 918,482 | 2.2 MiB | 7,112 | text/plain | yes |
| ocenture.com | 461,922 | 2.1 MiB | 8 | text/html; charset=UTF-8 | yes |
RFC 9309 requires parsers honor at least 500 KiB; files above that are truncated by Google. Our canonical size outlier is johnmu.com (~1.65 MB, simultaneously a valid WAV file and a valid robots.txt). Rows marked truncated hit Common Crawl's 5 MiB capture cap — the actual file is larger.
Most directives
| Host | Tranco rank | disallow | allow | total rows |
|---|---|---|---|---|
| eventer.co.il | 176,232 | 27,653 | 0 | 27,657 |
| steepster.com | 102,435 | 23,568 | 0 | 23,588 |
| steepster.com | 102,435 | 23,568 | 0 | 23,588 |
| cdn-surfline.com | 57,307 | 20,948 | 0 | 20,949 |
| easy-feedback.de | 196,178 | 0 | 19,646 | 19,651 |
| easy-feedback.com | 286,637 | 0 | 19,646 | 19,651 |
| lasommeliere.com | 404,813 | 19,424 | 8 | 19,504 |
| modneduzerozmiary.pl | 212,205 | 18,571 | 8 | 18,620 |
| mdr24.ro | 799,669 | 18,496 | 8 | 18,545 |
| mdr24.sk | 799,670 | 18,496 | 8 | 18,545 |
| mdr24.lt | 799,668 | 18,496 | 8 | 18,545 |
| mdr24.hu | 799,667 | 18,496 | 8 | 18,545 |
| mdr24.bg | 865,519 | 18,496 | 8 | 18,545 |
| urwatools.com | 86,891 | 17,601 | 1 | 17,624 |
| acompanhantesrosa.com | 751,405 | 0 | 17,373 | 17,374 |
| idrugstore.jp | 193,853 | 16,901 | 0 | 16,913 |
| sxdaily.com.cn | 116,836 | 16,409 | 0 | 16,410 |
| millemercismariage.com | 631,282 | 15,650 | 0 | 15,651 |
| cur8.com | 330,888 | 1 | 15,040 | 15,045 |
| lclark.edu | 46,862 | 14,698 | 0 | 14,701 |
| f71.com | 388,495 | 14,192 | 0 | 14,193 |
| hostedredmine.com | 720,938 | 13,440 | 0 | 13,487 |
| hostedredmine.com | 720,938 | 13,440 | 0 | 13,487 |
| matbao.net | 155,204 | 13,391 | 1 | 13,393 |
| eporncam.com | 563,890 | 13,199 | 0 | 13,200 |
Most user-agents addressed
Files naming the most distinct crawler user-agents — a signal of operator sophistication (or blocklist maximalism). A robots.txt addressing hundreds of bots is making a deliberate statement about who may crawl.
| Host | Tranco rank | user-agents | groups |
|---|---|---|---|
| fitnessfirst.com | 492,707 | 2,142 | 2 |
| fitnessfirst.com | 492,707 | 2,142 | 2 |
| pneumat.com.pl | 381,036 | 1,946 | 1,970 |
| pneumat.com.pl | 381,036 | 1,940 | 1,964 |
| insertface.com | 632,555 | 1,837 | 1,894 |
| insertface.com | 632,555 | 1,837 | 1,894 |
| state.mn.us | 41,198 | 1,787 | 1,795 |
| rolepages.com | 261,371 | 1,784 | 1,788 |
| rolepages.com | 261,371 | 1,784 | 1,788 |
| spectrumculture.com | 439,780 | 1,783 | 1,785 |
| retailcustomerexperience.com | 145,806 | 1,778 | 1,775 |
| qsrweb.com | 428,755 | 1,778 | 1,775 |
| fastcasual.com | 233,673 | 1,778 | 1,775 |
| digitalsignagetoday.com | 343,145 | 1,778 | 1,775 |
| vendingtimes.com | 599,396 | 1,778 | 1,775 |
| athensauthenticmarathon.gr | 601,480 | 1,778 | 1,778 |
| kioskmarketplace.com | 451,243 | 1,778 | 1,775 |
| bantoa.com | 157,995 | 1,778 | 1,778 |
| pizzamarketplace.com | 696,026 | 1,778 | 1,775 |
| atmmarketplace.com | 331,406 | 1,778 | 1,775 |
| retailcustomerexperience.com | 145,806 | 1,778 | 1,775 |
| bantoa.com | 157,995 | 1,778 | 1,778 |
| fastcasual.com | 233,673 | 1,778 | 1,775 |
| atmmarketplace.com | 331,406 | 1,778 | 1,775 |
| digitalsignagetoday.com | 343,145 | 1,778 | 1,775 |
Content-Type mismatches
Body parses as a valid robots.txt, but the served Content-Type isn't text/plain.
| Host | Tranco rank | served type | detected type |
|---|---|---|---|
| githubusercontent.com | 725 | application/octet-stream | text/x-robots |
| go-mpulse.net | 905 | application/javascript; charset=utf-8 | text/plain |
| firebaseio.com | 2,004 | application/octet-stream | text/x-robots |
| ingage.tech | 2,220 | image/gif | text/plain |
| comodoca.com | 2,224 | application/ocsp-response | text/plain |
| mi-img.com | 2,539 | application/octet-stream | text/plain |
| kpmg.com | 2,890 | text/javascript | text/x-robots |
| imgix.net | 2,922 | text/plain, charset=utf-8 | text/x-robots |
| symcb.com | 2,976 | application/octet-stream | text/plain |
| outbrainimg.com | 4,194 | text/plain, charset=utf-8 | text/x-robots |
| micpn.com | 4,836 | application/octet-stream | text/x-robots |
| blackspider.com | 4,897 | text | text/plain |
| ghcr.io | 4,925 | application/json | text/plain |
| flippingbook.com | 5,680 | application/octet-stream | text/x-robots |
| imedia.cz | 6,061 | text/plain,text/plain | text/x-robots |
| turkishairlines.com | 6,098 | document | text/x-robots |
| footmercato.net | 6,128 | txt | text/x-robots |
| liputan6.com | 6,320 | text/txt; charset=UTF-8 | text/x-robots |
| forexfactory.com | 6,361 | text; charset=UTF-8 | text/x-robots |
| brightcovecdn.com | 6,648 | application/json | text/plain |
| midasbuy.com | 6,808 | application/javascript | text/x-robots |
| paheal.net | 7,252 | image/jpeg | text/x-robots |
| hornbach.de | 7,263 | text/plain,text/plain | text/x-robots |
| stirileprotv.ro | 7,780 | text | text/x-robots |
| clips4sale.com | 8,264 | text-plain | text/x-robots |