robots.txt & llms.txt across the Tranco top 1M

Domain explorer

Look up a single domain's robots.txt & llms.txt metrics, effective AI policy, inferred tech stack (labelled hypothesis), and crawl history. Read-only; one indexed lookup per query.

Try: openai.com nytimes.com github.com sheetz.com

pneumat.com.pl

Tranco rank
381,036
popularity rank
TLD
com.pl
pneumat.com.pl
robots.txt records
2
across crawls
llms.txt records
0
across crawls

Live files on pneumat.com.pl

Open the file as it exists on the host right now — may differ from what we measured. We link out rather than mirror (we store derived metrics, not raw bodies).

robots.txt ↗ llms.txt ↗ llms-full.txt ↗ ai.txt ↗

Captured samples

Snapshots we captured for the showcase set (may differ from the live file above). Large files are truncated; the true size is noted.

/robots.txt · 78,329 bytes 
User-agent: *
Disallow: /shared-cgi/
Disallow: /shared-doc/
Disallow: /*op=*
Disallow: /*recoverPassword$
Disallow: /*displayLogin$
Disallow: *,p*,a*
Disallow: /szukaj?
Disallow: /admin/blog/
Disallow: /*.stp$
Disallow: /*.igs$
Disallow: /porownaj/get-info
Disallow: /basket/info 
Disallow: /sales?
Disallow: /user-account/
Disallow: /admin/
Disallow: /pl-PL/eirec/frontend/get/
Disallow: /eir/cookie.php
Disallow: /pl-PL/eirec/frontend/get/
User-agent: *
Sitemap: https://www.pneumat.com.pl/public/xml/pneumat-rwd/sitemap.xml
Allow: /public/xml/pneumat-rwd/sitemap.xml

User-agent: nibbler
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: sitebot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: SentiBot
Disallow: /
User-agent: Uptimebot
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: Spbot
Disallow: /
User-agent: linkdexbot
Disallow: /
User-agent: PrivacyAwareBot
Disallow: /
User-agent: SafeDNSBot
Disallow: /
User-agent: VoilaBot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: Charlotte/1.0b
Disallow: /
User-agent: boitho.com-dc
Disallow: /
User-agent: ajSitemap
Disallow: /
User-agent: Rankivabot
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: SapphireWebCrawler
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: Widow
Disallow: /
User-agent: discobot
Disallow: /
User-agent: FlashGet
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: panscient.com
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: Antenne Hatena
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot 
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: grub
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro 
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetAnts/1.24 
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gatherer
Disallow: /
User-agent: PerMan
Disallow: / 
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RMA
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TightTwatBot
Disallow: /
User-agent: Titan
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: turingos
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: VCI
Disallow: /
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Website.Quester
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: WebZip/4.0
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: ApocalXExplorerBot
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BlowFish/1.0 
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: CazoodleBot
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: Curious George
Disallow: /
User-agent: DISCo Pump 3.1
Disallow: /
User-agent: delta-search
Disallow: /
User-agent: linklooker
Disallow: /
User-agent: SafetyNet Robot 0.1
Disallow: /
User-agent: INGRID/0.1
Disallow: /
User-agent: EnigmaBot
Disallow: /
User-agent: Synthesio
Disallow: / 
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack 3.0
Disallow: /
User-agent: HTTrack [NC,OR]
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: MOMspider/1.00 libwww-perl/0.40 
Disallow: /
User-agent: BSpider/1.0 libwww-perl/0.40 
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: wget
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: psbot/0.1
Disallow: /
User-agent: TMCrawler
Disallow: /
User-agent: e-SocietyRobot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: NPBot-1/2.0
Disallow: /
User-agent: AboutUsBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: VoilaBot BETA 1.2.
Disallow: /
User-agent: bot/1.0
Disallow: /
User-agent: Java/1.6.0_11.
Disallow: /
User-agent: WebDataCentreBot/1.0.
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: FreeFind
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: dumbot
Disallow: /
User-agent: Hatena Antenna
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: WebmasterWorld Extractor
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: sootle
Disallow: /
User-agent: es
Disallow: /
User-agent: Enterprise_Search/1.0
Disallow: /
User-agent: ECCP/1.0
Disallow: /
User-agent: ECCP
Disallow: /
User-agent: linkdexbot/2.1 
Disallow: /
User-agent: proximic
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: voltron
Disallow: /
User-agent: Updownerbot
Disallow: /
User-agent: meanpathbot
Disallow: /
User-agent: SnapPreviewBot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: ZumBot
Disallow: /
User-agent: CrazyWebCrawler-Spider
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: musobot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: FRCrawler
Disallow: /
User-agent: ca-crawler
Disallow: /
User-agent: 007ac9
Disallow: /
User-agent: NerdyBot
Disallow: /
User-agent: MixrankBot
Disallow: /
User-agent: oBot
Disallow: /
User-agent: MemoryBot
Disallow: /
User-agent: LinkpadBot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: HaoSouSpider
Disallow: /
User-agent: AITCSRobot/1.1
Disallow: /
User-agent: Arachnophilia
Disallow: /
User-agent: AddSugarSpiderBot
Disallow: /
User-agent: ASpider/0.09
Disallow: /
User-agent: AURESYS/1.0
Disallow: /
User-agent: BackRub/.
Disallow: /
User-agent: BlackWidow
Disallow: /
User-agent: BizBot04 kirk.overleaf.com
Disallow: /
User-agent: CyberSpyder/2.1
Disallow: /
User-agent: Deweb/1.01
Disallow: /
User-agent: DISCoFinder
Disallow: /
User-agent: DISCo Pump 3.2
Disallow: /
User-agent: DISCo Pump 3.0
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: conceptbot/0.3
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: Bizbot003
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: CACTVS Chemistry Spider
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 
Disallow: /
User-agent: Download Demon/3.2.0.8
Disallow: /
User-agent: Download Demon
Disallow: /
User-agent: Download Demon/3.5.0.11
Disallow: /
User-agent: DISCo
Disallow: /
User-agent: eCatch/3.0
Disallow: /
User-agent: EyeNetIE
Disallow: /
User-agent: FelixIDE/1.0
Disallow: /
User-agent: fido/0.9 Harvest/1.4.pl2
Disallow: /
User-agent: Fish-Search-Robot
Disallow: /
User-agent: GetURL.rexx v1.05
Disallow: /
User-agent: GrabNet
Disallow: /
User-agent: GetWeb!
Disallow: /
User-agent: grapeshot
Disallow: /
User-agent: Gromit/1.0
Disallow: /
User-agent: Golem/1.1
Disallow: /
User-agent: Go-Ahead-Got-It block
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Checkbot/x.xx LWP/5.x
Disallow: /
User-agent: Bot mailto:craftbot@yahoo.com
Disallow: /
User-agent: Hazel's Ferret Web hopper
Disallow: /
User-agent: HMView
Disallow: /
User-agent: Huaweisymantecspider
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: HappyFunBot
Disallow: /
User-agent: Image Sucker
Disallow: /
User-agent: Internet Ninja
Disallow: /
User-agent: Internet Ninja 4.0
Disallow: /
User-agent: Internet Ninja 5.0
Disallow: /
User-agent: Internet Ninja 6.0
Disallow: /
User-agent: Image Stripper
Disallow: /
User-agent: inagist.com url crawler
Disallow: /
User-agent: IncyWincy/1.0b1
Disallow: /
User-agent: Indy Library
Disallow: /
User-agent: Indy Library [NC,OR]
Disallow: /
User-agent: Informant
Disallow: /
User-agent: IsraeliSearch/1.0
Disallow: /
User-agent: JetCar
Disallow: /
User-agent: JubiiRobot
Disallow: /
User-agent: Katipo/1.0
Disallow: /
User-agent: KIT-Fireball/2.0 libwww/5.0a
Disallow: /
User-agent: LabelGrab/1.1
Disallow: /
User-agent: LeechFTP
Disallow: /
User-agent: Mata.Hari
Disallow: /
User-agent: MerzScope
Disallow: /
User-agent: MIDown tool
Disallow: /
User-agent: NeoScioCrawler
Disallow: /
User-agent: Yasaklibot
Disallow: /
User-agent: YesupBot
Disallow: /
User-agent: WhoWhere Robot
Disallow: /
User-agent: wired-digital-newsbot/1.5
Disallow: /
User-agent: www.freeloader.com.
Disallow: /
User-agent: WebWatch
Disallow: /
User-agent: webwalk
Disallow: /
User-agent: WebWalker
Disallow: /
User-agent: WebWalker/1.10
Disallow: /
User-agent: WebVac
Disallow: /
User-agent: webvac/1.0
Disallow: /
User-agent: WWWOFFLE
Disallow: /
User-agent: WWWWanderer v3.0
Disallow: /
User-agent: Xaldon WebSpider
Disallow: /
User-agent: Xaldon WebSpider 2.5.b3
Disallow: /
User-agent: WebWhacker
Disallow: /
User-agent: Web Sucker
Disallow: /
User-agent: WebCapture 2.0
Disallow: /
User-agent: WebCopier v.2.2
Disallow: /
User-agent: WebCopier v2.5
Disallow: /
User-agent: WebCopier v2.6
Disallow: /
User-agent: WebCopier v2.7a
Disallow: /
User-agent: WebCopier v2.8
Disallow: /
User-agent: WebCopier v3.0
Disallow: /
User-agent: WebCopier v3.0.1
Disallow: /
User-agent: WebCopier v3.2
Disallow: /
User-agent: WebCopier v3.2a
Disallow: /
User-agent: WebFerret
Disallow: /
User-agent: WebFetch
Disallow: /
User-agent: WebCrawler/3.0 Robot libwww/5.0a
Disallow: /
User-agent: webfetch/2.1.0
Disallow: /
User-agent: WebFetcher/0.8,
Disallow: /
User-agent: weblayers/0.0
Disallow: /
User-agent: WebMoose/0.0.0000
Disallow: /
User-agent: WebLinker/0.0 libwww-perl/0.1
Disallow: /
User-agent: WebSauger 1.20b
Disallow: /
User-agent: WebSauger 1.20j
Disallow: /
User-agent: Website Quester - www.asona.org
Disallow: /
User-agent: Website Quester - www.esalesbiz.com/extra/
Disallow: /
User-agent: Website.Quester
Disallow: /
User-agent: WebGo IS
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebAuto/3.40 (Win98; I)
Disallow: /
User-agent: VoidEYE
Disallow: /
User-agent: Valkyrie/1.0 libwww-perl/0.40
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: urlck/1.2.3
Disallow: /
User-agent: URLy.Warning
Disallow: /
User-agent: tAkeOut
Disallow: /
User-agent: Teleport Pro/1.29
Disallow: /
User-agent: Teleport Pro/1.29.1590
Disallow: /
User-agent: Teleport Pro/1.29.1634
Disallow: /
User-agent: Teleport Pro/1.29.1718
Disallow: /
User-agent: Teleport Pro/1.29.1820
Disallow: /
User-agent: Teleport Pro/1.29.1847
Disallow: /
User-agent: The.Intraformant
Disallow: /
User-agent: SmartDownload
Disallow: /
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
Disallow: /
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
Disallow: /
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
Disallow: /
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
Disallow: /
User-agent: Snooper/b97_01
Disallow: /
User-agent: Solbot/1.0 LWP/5.07
Disallow: /
User-agent: Spanner/1.0 (Linux 2.0.27 i586)
Disallow: /
User-agent: spyder3.microsys.com
Disallow: /
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
Disallow: /
User-agent: NetSpider
Disallow: /
User-agent: Net Vampire
Disallow: /
User-agent: Net Vampire/3.0
Disallow: /
User-agent: NetAnts/1.10
Disallow: /
User-agent: NetAnts/1.23
Disallow: /
User-agent: NetAnts/1.25
Disallow: /
User-agent: NetCarta CyberPilot Pro
Disallow: /
User-agent: Occam/1.0
Disallow: /
User-agent: NHSEWalker/3.0
Disallow: /
User-agent: NetScoop/1.0 libwww/5.0a
Disallow: /
User-agent: Navroad
Disallow: /
User-agent: NearSite
Disallow: /
User-agent: Nomad-V2.x
Disallow: /
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
Disallow: /
User-agent: NetZIP
Disallow: /
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
Disallow: /
User-agent: NetZippy+(http://www.innerprise.net/usp-spider.asp)
Disallow: /
User-agent: Open Text Site Crawler V1.0
Disallow: /
User-agent: Offline Navigator
Disallow: /
User-agent: PageGrabber
Disallow: /
User-agent: Papa Foto
Disallow: /
User-agent: R6_CommentReader
Disallow: /
User-agent: RealDownload
Disallow: /
User-agent: RealDownload/4.0.0.40
Disallow: /
User-agent: RealDownload/4.0.0.41
Disallow: /
User-agent: RealDownload/4.0.0.42
Disallow: /
User-agent: Roverbot
Disallow: /
User-agent: Senrigan/xxxxxx
Disallow: /
User-agent: MediaFox/x.y
Disallow: /
User-agent: Mister.PiX
Disallow: /
User-agent: Mister Pix II 2.01
Disallow: /
User-agent: Mister Pix II 2.02a
Disallow: /
User-agent: Mister PiX version.dll
Disallow: /
User-agent: GetRight/2.11
Disallow: /
User-agent: GetRight/3.1
Disallow: /
User-agent: GetRight/3.2
Disallow: /
User-agent: GetRight/3.3
Disallow: /
User-agent: GetRight/3.3.3
Disallow: /
User-agent: GetRight/3.3.4
Disallow: /
User-agent: GetRight/4.0.0
Disallow: /
User-agent: GetRight/4.1.0
Disallow: /
User-agent: GetRight/4.1.1
Disallow: /
User-agent: GetRight/4.1.2
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: GetRight/4.2b (Portuguxeas)
Disallow: /
User-agent: GetRight/4.2c
Disallow: /
User-agent: GetRight/4.3
Disallow: /
User-agent: GetRight/4.5
Disallow: /
User-agent: GetRight/4.5a
Disallow: /
User-agent: GetRight/4.5b
Disallow: /
User-agent: GetRight/4.5b1
Disallow: /
User-agent: GetRight/4.5b2
Disallow: /
User-agent: GetRight/4.5b3
Disallow: /
User-agent: GetRight/4.5b6
Disallow: /
User-agent: GetRight/4.5b7
Disallow: /
User-agent: GetRight/4.5c
Disallow: /
User-agent: GetRight/4.5d
Disallow: /
User-agent: GetRight/5.0beta1
Disallow: /
User-agent: GetRight/4.5e
Disallow: /
User-agent: GetRight/5.0beta2
Disallow: /
User-agent: Go!Zilla
Disallow: /
User-agent: Go!Zilla (www.gozilla.com)
Disallow: /
User-agent: Go!Zilla 3.3 (www.gozilla.com)
Disallow: /
User-agent: Go!Zilla 3.5 (www.gozilla.com)
Disallow: /
User-agent: larbin (samualt9@bigfoot.com)
Disallow: /
User-agent: larbin samualt9@bigfoot.com
Disallow: /
User-agent: larbin_2.6.2 (kabura@sushi.com)
Disallow: /
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
Disallow: /
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
Disallow: /
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
Disallow: /
User-agent: larbin_2.6.2 kabura@sushi.com
Disallow: /
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
Disallow: /
User-agent: larbin_2.6.2 larbin@correa.org
Disallow: /
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
Disallow: /
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
Disallow: /
User-agent: Mass Downloader
Disallow: /
User-agent: Mass Downloader/2.2
Disallow: /
User-agent: Offline Explorer/1.2
Disallow: /
User-agent: Offline Explorer/1.4
Disallow: /
User-agent: Offline Explorer/1.6
Disallow: /
User-agent: Offline Explorer/1.7
Disallow: /
User-agent: Offline Explorer/1.9
Disallow: /
User-agent: Offline Explorer/2.0
Disallow: /
User-agent: Offline Explorer/2.1
Disallow: /
User-agent: Offline Explorer/2.3
Disallow: /
User-agent: Offline Explorer/2.4
Disallow: /
User-agent: Offline Explorer/2.5
Disallow: /
User-agent: Offline.Explorer
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: pcBrowser
Disallow: /
User-agent: SlySearch
Disallow: /
User-agent: SuperBot
Disallow: /
User-agent: SuperBot/3.0 (Win32)
Disallow: /
User-agent: SuperBot/3.1 (Win32)
Disallow: /
User-agent: SuperHTTP
Disallow: /
User-agent: SuperHTTP/1.0
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: Microsoft.URL
Disallow: /
User-agent: linkdexbot/2.0
Disallow: /
User-agent: linkdexbot/2.1
Disallow: /
User-agent: LinkScan/8.1a.Unix
Disallow: /
User-agent: FlashGet WebWasher 3.2
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Keyword.Density
Disallow: /
User-agent: WebCopy/
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: Webster.Pro
Disallow: /
User-agent: WebStripper/2.03
Disallow: /
User-agent: WebStripper/2.10
Disallow: /
User-agent: WebStripper/2.12
Disallow: /
User-agent: WebStripper/2.13
Disallow: /
User-agent: Website eXtractor
Disallow: /
User-agent: WebStripper/2.15
Disallow: /
User-agent: WebStripper/2.16
Disallow: /
User-agent: WebStripper/2.19
Disallow: /
User-agent: WebZIP/2.75 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/3.65 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/3.80 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/4.0 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/4.1 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/4.21
Disallow: /
User-agent: WebZIP/4.21 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/5.0
Disallow: /
User-agent: WebZIP/5.0 (http://www.spidersoft.com)
Disallow: /
User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com)
Disallow: /
User-agent: Wget/1.4.0
Disallow: /
User-agent: Wget/1.5.2
Disallow: /
User-agent: Wget/1.5.3
Disallow: /
User-agent: Wget/1.6
Disallow: /
User-agent: Wget/1.7
Disallow: /
User-agent: Wget/1.8
Disallow: /
User-agent: Wget/1.8.1
Disallow: /
User-agent: Wget/1.8.1+cvs
Disallow: /
User-agent: Wget/1.8.2
Disallow: /
User-agent: Wget/1.9-beta
Disallow: /
User-agent: WWW Collector
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: Web.Image.Collector
Disallow: /
User-agent: XGET/0.7
Disallow: /
User-agent: Zeus 11389 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 11652 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 18018 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 26378 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 30747 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 39206 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 41641 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 44238 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 51070 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 51674 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 51837 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 63567 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 6694 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 82016 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 82900 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 84842 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 90872 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 94934 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 95245 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 95351 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus 97371 Webster Pro V2.9 Win32
Disallow: /
User-agent: ZyBorg
Disallow: /
User-agent: WebEMailExtrac.*
Disallow: /
User-agent: vBSEO
Disallow: /
User-agent: Big Brother
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: combine/0.0
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: Custo
Disallow: /
User-agent: Express WebPictures
Disallow: /
User-agent: Express WebPictures (www.express-soft.com)
Disallow: /
User-agent: Freecrawl
Disallow: /
User-agent: gcreep/1.0
Disallow: /
User-agent: Grafula
Disallow: /
User-agent: InterGET
Disallow: /
User-agent: ITI Spider
Disallow: /
User-agent: JOC Web Spider
Disallow: /
User-agent: Motor/0.2
Disallow: /
User-agent: NorthStar
Disallow: /
User-agent: OGspider
Disallow: /
User-agent: pavuk
Disallow: /
User-agent: PGP-KA/1.2
Disallow: /
User-agent: ReGet
Disallow: /
User-agent: Resume Robot
Disallow: /
User-agent: Surfbot
Disallow: /
User-agent: UCSD-Crawler
Disallow: /
User-agent: URL Control 
Disallow: /
User-agent: URLSpiderPro
Disallow: /
User-agent: WebLeacher
Disallow: /
User-agent: 12soso
Disallow: /
User-agent: 192.comAgent
Disallow: /
User-agent: 1Noonbot
Disallow: /
User-agent: 3DE_SEARCH2
Disallow: /
User-agent: 3D_SEARCH
Disallow: /
User-agent: 3GSE
Disallow: /
User-agent: 50.nu
Disallow: /
User-agent: A1 Sitemap Generator
Disallow: /
User-agent: A1 Website Download
Disallow: /
User-agent: AASP
Disallow: /
User-agent: ABACHOBot
Disallow: /
User-agent: AChulkov.NET page walker
Disallow: /
User-agent: AF Knowledge Now Verity Spider
Disallow: /
User-agent: AIBOT
Disallow: /
User-agent: AISIID
Disallow: /
User-agent: AONDE-Spider
Disallow: /
User-agent: ASAHA Search Engine Turkey
Disallow: /
User-agent: ASPSeek
Disallow: /
User-agent: ASPseek
Disallow: /
User-agent: AURESYS
Disallow: /
User-agent: AbotEmailSearch
Disallow: /
User-agent: AccMonitor Compliance Server
Disallow: /
User-agent: Acme.Spider
Disallow: /
User-agent: ActiveTouristBot
Disallow: /
User-agent: Ad Muncher
Disallow: /
User-agent: AdamM Bot,
Disallow: /
User-agent: Advanced Email Extractor
Disallow: /
User-agent: Akamai-SiteSnapshot
Disallow: /
User-agent: AlexfDownload
Disallow: /
User-agent: AlkalineBOT
Disallow: /
User-agent: All Acronyms Bot
Disallow: /
User-agent: Alpha Search Agent
Disallow: /
User-agent: AmPmPPC.com
Disallow: /
User-agent: Amerla Search Bot
Disallow: /
User-agent: Amfibibot
Disallow: /
User-agent: Anonymous/3G bot
Disallow: /
User-agent: Anonymouse.org
Disallow: /
User-agent: AnotherBot
Disallow: /
User-agent: AnswerBus
Disallow: /
User-agent: AnswerChase PROve
Disallow: /
User-agent: AntBot
Disallow: / 
User-agent: AntiSantyWorm
Disallow: /
User-agent: Antro.Net
Disallow: /
User-agent: Aport
Disallow: /
User-agent: AraBot
Disallow: /
User-agent: Arachmo
Disallow: /
User-agent: Arikus_Spider
Disallow: /
User-agent: Art-Online.com
Disallow: /
User-agent: ArtavisBot
Disallow: /
User-agent: Artera
Disallow: /
User-agent: AstroFind
Disallow: /
User-agent: AtlocalBot
Disallow: /
User-agent: Atomic_Email_Hunter
Disallow: /
User-agent: Attributor.comBot
Disallow: /
User-agent: Attributor/Dejan-
Disallow: /
User-agent: AutoBaron crawler
Disallow: /
User-agent: Axonize-bot
Disallow: /
User-agent: Ayna
Disallow: /
User-agent: BDFetch
Disallow: /
User-agent: BIGLOTRON
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: BLT
Disallow: /
User-agent: BOIA-Scan-Agent
Disallow: /
User-agent: BOIA.ORG-Scan-Agent
Disallow: /
User-agent: BOI_crawl_00
Disallow: /
User-agent: BOT
Disallow: /
User-agent: BSearchR&D
Disallow: /
User-agent: BSpider
Disallow: /
User-agent: BYINDIA
Disallow: /
User-agent: BackStreet Browser
Disallow: /
User-agent: BackWeb
Disallow: /
User-agent: Bandit
Disallow: /
User-agent: BatchFTP
Disallow: /
User-agent: BecomeJPBot
Disallow: /
User-agent: BeetleBot
Disallow: /
User-agent: BeetleBot
Disallow: /
User-agent: Bender
Disallow: /
User-agent: Big Data
Disallow: /
User-agent: BigCliqueBOT
Disallow: /
User-agent: BigCliqueBot
Disallow: /
User-agent: Bigado.com
Disallow: /
User-agent: Bigfoot
Disallow: /
User-agent: Bilbo
Disallow: /
User-agent: BilgiBetaBot
Disallow: /
User-agent: BilgiBot
Disallow: /
User-agent: BitvoUserAgent
Disallow: /
User-agent: BizWorks Retriever
Disallow: /
User-agent: Black.Hole
Disallow: /
User-agent: BlackMask.Net Search Engine
Disallow: /
User-agent: Blackbird
Disallow: /
User-agent: Blaiz-Bee
Disallow: /
User-agent: Blinkx/DFS-Fetch
Disallow: /
User-agent: BlitzBOT
Disallow: /
User-agent: Blog Conversation Project
Disallow: /
User-agent: BlogMyWay
Disallow: /
User-agent: BlogPulseLive
Disallow: /
User-agent: BlogRefsBot
Disallow: /
User-agent: BlogScope
Disallow: /
User-agent: Blogslive
Disallow: /
User-agent: BloobyBot
Disallow: /
User-agent: BoaConstrictor
Disallow: /
User-agent: BoardReader-Image-Fetcher
Disallow: /
User-agent: Bookmark Buddy bookmark checker
Disallow: /
User-agent: Bot Apoena
Disallow: /
User-agent: Botswana
Disallow: /
User-agent: BpBot
Disallow: /
User-agent: BrowserEmulator
Disallow: /
User-agent: BrowserMob
Disallow: /
User-agent: BruinBot
Disallow: /
User-agent: Btsearch
Disallow: /
User-agent: Buddy
Disallow: /
User-agent: Buibui
Disallow: /
User-agent: BuildCMS crawler
Disallow: /
User-agent: BuscadorClarin
Disallow: /
User-agent: Buscaplus Robi
Disallow: /
User-agent: Butterfly
Disallow: /
User-agent: BuyHawaiiBot
Disallow: /
User-agent: BuzzBot
Disallow: /
User-agent: BySpider
Disallow: /
User-agent: CCGCrawl
Disallow: /
User-agent: CD-Preload
Disallow: /
User-agent: CE-Preload
Disallow: /
User-agent: CERT FigleafBot
Disallow: /
User-agent: CFNetwork
Disallow: /
User-agent: COAST WebMaster Pro
Disallow: /
User-agent: COAST scan engine
Disallow: /
User-agent: COMBINE
Disallow: /
User-agent: CRAWLER-ALTSE.VUNET.ORG-Lynx
Disallow: /
User-agent: CSHttpClient
Disallow: /
User-agent: CacheBlaster
Disallow: /
User-agent: Caddbot
Disallow: /
User-agent: Cafi
Disallow: /
User-agent: Camcrawler
Disallow: /
User-agent: CamelStampede
Disallow: /
User-agent: Canon-WebRecord
Disallow: /
User-agent: Canon-WebRecordPro
Disallow: /
User-agent: CareerBot
Disallow: /
User-agent: CatchBot
Disallow: /
User-agent: Cerberian Drtrs
Disallow: /
User-agent: Chameleon
Disallow: /
User-agent: Check&Get
Disallow: /
User-agent: CheckLinks
Disallow: /
User-agent: Checkbot
Disallow: /
User-agent: Checklinks
Disallow: /
User-agent: ChemieDE-NodeBot
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: Chilkat
Disallow: /
User-agent: CipinetBot
Disallow: /
User-agent: ClariaBot
Disallow: /
User-agent: Climate Ark
Disallow: /
User-agent: ClimateArk Spider
Disallow: /
User-agent: Clushbot
Disallow: /
User-agent: CollapsarWEB
Disallow: /
User-agent: Collector
Disallow: /
User-agent: Combine
Disallow: /
User-agent: Compatible
Disallow: /
User-agent: ConnectSearch
Disallow: /
User-agent: ContentSmartz
Disallow: /
User-agent: ContextAd Bot
Disallow: /
User-agent: CoolBot
Disallow: /
User-agent: CoolCheck
Disallow: /
User-agent: Copier
Disallow: /
User-agent: Covario-IDS
Disallow: /
User-agent: Cowbot 
Disallow: /
User-agent: Cowdog Bot
Disallow: /
User-agent: Crawl
Disallow: /
User-agent: CrawlConvera
Disallow: /
User-agent: Crawl_Application
Disallow: /
User-agent: Crawler
Disallow: /
User-agent: Crawly
Disallow: /
User-agent: CreativeCommons
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: Cuasarbot
Disallow: /
User-agent: Curl
Disallow: /
User-agent: CyberNavi_WebGet
Disallow: /
User-agent: CyberPatrol SiteCat Webbot
Disallow: /
User-agent: CyberSpyder
Disallow: /
User-agent: Cyberdog
Disallow: /
User-agent: CydralSpider
Disallow: /
User-agent: D1GArabicEngine
Disallow: /
User-agent: DA
Disallow: /
User-agent: DAUMOA-video
Disallow: /
User-agent: DAUMOA-web
Disallow: /
User-agent: DIIbot
Disallow: /
User-agent: DISCo Pump
Disallow: /
User-agent: DKIMRepBot
Disallow: /
User-agent: DOF-Verify
Disallow: /
User-agent: DOY
Disallow: /
User-agent: DTAAgent
Disallow: /
User-agent: DXSeeker
Disallow: /
User-agent: DataCha0s
Disallow: /
User-agent: DataFountains
Disallow: /
User-agent: DataFountains/DMOZ Downloader
Disallow: /
User-agent: DataSpearSpiderBot
Disallow: /
User-agent: DataSpider
Disallow: /
User-agent: DataparkSearch
Disallow: /
User-agent: Dattatec.com-Sitios-Top
Disallow: /
User-agent: Dattatec.com
Disallow: /
User-agent: Daumoa
Disallow: /
User-agent: Declumbot
Disallow: /
User-agent: DeepTrawl
Disallow: /
User-agent: Deepindex
Disallow: /
User-agent: DelvuBot
Disallow: /
User-agent: Der große BilderSauger
Disallow: /
User-agent: DiBot
Disallow: /
User-agent: DiGi-RSSBot
Disallow: /
User-agent: DiaGem
Disallow: /
User-agent: DigExt
Disallow: /
User-agent: DigOut4U
Disallow: /
User-agent: Digger
Disallow: /
User-agent: DigitalArchivesBot
Disallow: /
User-agent: Dillo
Disallow: /
User-agent: Dir_Snatch.exe
Disallow: /
User-agent: Distilled-Reputation-Monitor
Disallow: /
User-agent: Dit
Disallow: /
User-agent: DjangoTraineeBot
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: DomainScan
Disallow: /
User-agent: DomainWatcher Bot
Disallow: /
User-agent: Domaincrawler
Disallow: /
User-agent: DotSpotsBot
Disallow: /
User-agent: Dow Jones Searchbot
Disallow: /
User-agent: Download Devil
Disallow: /
User-agent: Download Master
Disallow: /
User-agent: Download Wonder
Disallow: /
User-agent: Download
Disallow: /
User-agent: Downloader
Disallow: /
User-agent: Drip
Disallow: /
User-agent: Dwaar
Disallow: /
User-agent: Dwaarbot
Disallow: /
User-agent: EAH
Disallow: /
User-agent: EC2LinkFinder
Disallow: /
User-agent: EMPAS_ROBOT
Disallow: /
User-agent: EMail Exractor
Disallow: /
User-agent: Earth Platform Indexer
Disallow: /
User-agent: Earth Science Educator  robot
Disallow: /
User-agent: EdisterBot (http://www.edister.com/bot.html)
Disallow: /
User-agent: EduGovSearch
Disallow: /
User-agent: Elblindo the Blind Bot
Disallow: /
User-agent: ElisaBot
Disallow: /
User-agent: EllerdaleBot
Disallow: /
User-agent: EmailLeach
Disallow: /
User-agent: EnaBot
Disallow: /
User-agent: Enswer Neuro Bot
Disallow: /
User-agent: EntityCubeBot
Disallow: /
User-agent: Eurosoft-Bot
Disallow: /
User-agent: Evaal
Disallow: /
User-agent: Eventware
Disallow: /
User-agent: Everest-Vulcan Inc.
Disallow: /
User-agent: ExactSearch
Disallow: /
User-agent: Exploder
Disallow: /
User-agent: FAST Data Search Document Retriever
Disallow: /
User-agent: FAST ESP
Disallow: /
User-agent: FDM 1.x
Disallow: /
User-agent: FDM 2.x
Disallow: /
User-agent: FDM 3
Disallow: /
User-agent: FDSE robot
Disallow: /
User-agent: FU-NBI/FU-NBI-
Disallow: /
User-agent: Factbot
Disallow: /
User-agent: Extractor
Disallow: /
User-agent: Falconsbot
Disallow: /
User-agent: FatBot
Disallow: /
User-agent: Favcollector
Disallow: /
User-agent: Faviconizer
Disallow: /
User-agent: Favorites Sweeper
Disallow: /
User-agent: FedContractorBot
Disallow: /
User-agent: Fetch API Request
Disallow: /
User-agent: Filangy
Disallow: /
User-agent: FileHound
Disallow: /
User-agent: FindAnISP.com_ISP_Finder
Disallow: /
User-agent: FindWeb
Disallow: /
User-agent: Firebat
Disallow: /
User-agent: FirstGov.gov Search
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: FlashCapture
Disallow: /
User-agent: FlickySearchBot
Disallow: /
User-agent: Fluffy the spider
Disallow: /
User-agent: FollowSite
Disallow: /
User-agent: Fopper
Disallow: /
User-agent: FormulaFinderBot
Disallow: /
User-agent: Forschungsportal
Disallow: /
User-agent: Francis
Disallow: /
User-agent: FreeWebMonitoring SiteChecker
Disallow: /
User-agent: FreshDownload
Disallow: /
User-agent: FriendFeedBot
Disallow: /
User-agent: FrontPage [NC,OR]
Disallow: /
User-agent: FrontPage
Disallow: /
User-agent: Froola Bot
Disallow: /
User-agent: FunnelBack
Disallow: /
User-agent: FurlBot
Disallow: /
User-agent: G10-Bot
Disallow: /
User-agent: GOFORITBOT
Disallow: /
User-agent: GPU p2p crawler
Disallow: /
User-agent: GVC BUSINESS crawler
Disallow: /
User-agent: GVC SEARCH BOT
Disallow: /
User-agent: GVC WEB crawler
Disallow: /
User-agent: GVC WORLD LINKS
Disallow: /
User-agent: GVC Weblink crawler
Disallow: /
User-agent: GVC crawler
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GalaxyBot
Disallow: /
User-agent: GenieBotRD_SmallCrawl
Disallow: /
User-agent: GeoVisu
Disallow: /
User-agent: Geomaxenginebot
Disallow: /
User-agent: GeonaBot
Disallow: /
User-agent: GermCrawler
Disallow: /
User-agent: GetHTMLContents
Disallow: /
User-agent: GetLeft
Disallow: /
User-agent: GetSmart
Disallow: /
User-agent: Getleft
Disallow: /
User-agent: Giant
Disallow: /
User-agent: Girafabot
Disallow: /
User-agent: GleameBot
Disallow: /
User-agent: Go-Ahead-Got-It
Disallow: /
User-agent: GoForIt.com
Disallow: /
User-agent: GoSeebot
Disallow: /
User-agent: Goldfire Server
Disallow: /
User-agent: GoodJelly
Disallow: /
User-agent: Gordon-College-Google-Mini
Disallow: /
User-agent: Govbot
Disallow: /
User-agent: Grabber
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: GreenYogi [ZSEBOT]
Disallow: /
User-agent: Gromit
Disallow: /
User-agent: GruBot
Disallow: /
User-agent: GrubNG
Disallow: /
User-agent: Grubclient-
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: GulperBot
Disallow: /
User-agent: GurujiBot
Disallow: /
User-agent: HD nutch agent
Disallow: /
User-agent: HLoader
Disallow: /
User-agent: HMSEbot
Disallow: /
User-agent: HSFT - LVU Scanner
Disallow: /
User-agent: HSFT - Link Scanner
Disallow: /
User-agent: HSlide
Disallow: /
User-agent: HTMLParser
Disallow: /
User-agent: HarvestMan
Disallow: /
User-agent: Hawler
Disallow: /
User-agent: Header_Test_Client
Disallow: /
User-agent: Helix
Disallow: /
User-agent: HiScan
Disallow: /
User-agent: HiSoftware AccMonitor Server
Disallow: /
User-agent: HiSoftware AccVerify
Disallow: /
User-agent: HomePageSearch
Disallow: /
User-agent: HooWWWer
Disallow: /
User-agent: Hooblybot-Image
Disallow: /
User-agent: Hostcrawler
Disallow: /
User-agent: Html Link Validator
Disallow: /
User-agent: HyperEstraier
Disallow: /
User-agent: Hyperix
Disallow: /
User-agent: IAArchiver
Disallow: /
User-agent: ICDS-Ingestion
Disallow: /
User-agent: IEAutoDiscovery
Disallow: /
User-agent: IECheck
Disallow: /
User-agent: IIITBOT
Disallow: /
User-agent: INFOMINE
Disallow: /
User-agent: INGRID
Disallow: /
User-agent: INTERNET RADIO crawler
Disallow: /
User-agent: IOI
Disallow: /
User-agent: IPAdd Bot
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: IU_CSCI_B659_class_crawler
Disallow: /
User-agent: IlTrovatore-Setaccio
Disallow: /
User-agent: IlTrovatore
Disallow: /
User-agent: IlseBot
Disallow: /
User-agent: Iltrovatore-Setaccio
Disallow: /
User-agent: ImageBot
Disallow: /
User-agent: ImageVisu
Disallow: /
User-agent: ImagesHereImagesThereImagesEverywhere
Disallow: /
User-agent: Industry Cortex Webcrawler
Disallow: /
User-agent: InelaBot
Disallow: /
User-agent: Inet32 Ctrl
Disallow: /
User-agent: InfoLink
Disallow: /
User-agent: InfoSeek Sidewinder
Disallow: /
User-agent: InfoTekies
Disallow: /
User-agent: InfoUSABot
Disallow: /
User-agent: Infoseek SideWinder
Disallow: /
User-agent: Infoseek Sidewinder
Disallow: /
User-agent: InsightsCollector
Disallow: /
User-agent: InsightsWorksBot
Disallow: /
User-agent: InspireBot
Disallow: /
User-agent: InsumaScout
Disallow: /
User-agent: Intelix
Disallow: /
User-agent: Intelliseek
Disallow: /
User-agent: InternetLinkAgent
Disallow: /
User-agent: IpselonBot
Disallow: /
User-agent: Ipselonbot
Disallow: /
User-agent: Iria
Disallow: /
User-agent: Isara/Isara-
Disallow: /
User-agent: IstellaBot
Disallow: /
User-agent: JOC
Disallow: /
User-agent: JSpyda
Disallow: /
User-agent: JadynAve
Disallow: /
User-agent: JadynAveBot
Disallow: /
User-agent: Jbot
Disallow: /
User-agent: JemmaTheTourist
Disallow: /
User-agent: JetBrains Omea Pro
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Jim
Disallow: /
User-agent: JoBo
Disallow: /
User-agent: JobSpider_BA
Disallow: /
User-agent: JoeDog
Disallow: /
User-agent: JoyScapeBot
Disallow: /
User-agent: Junut Bot
Disallow: /
User-agent: JustView
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: K.S.Bot
Disallow: /
User-agent: KATATUDO-Spider
Disallow: /
User-agent: KSbot
Disallow: /
User-agent: KakcleBot
Disallow: /
User-agent: KaloogaBot
Disallow: /
User-agent: KeepNI web site monitor
Disallow: /
User-agent: Kenjin.Spider
Disallow: /
User-agent: Keybot Translation-Search-Machine
Disallow: /
User-agent: KeywenBot
Disallow: /
User-agent: Kitenga-crawler-bot
Disallow: /
User-agent: KiwiStatus
Disallow: /
User-agent: Knight
Disallow: /
User-agent: KnowItAll
Disallow: /
User-agent: Knowledge Engine
Disallow: /
User-agent: Knowledge.com
Disallow: /
User-agent: KoepaBot
Disallow: /
User-agent: Koninklijke
Disallow: /
User-agent: KrOWLer
Disallow: /
User-agent: KummHttp
Disallow: /
User-agent: Kurzor
Disallow: /
User-agent: Kyluka crawl
Disallow: /
User-agent: L.webis
Disallow: /
User-agent: LARBIN-EXPERIMENTAL
Disallow: /
User-agent: LBot
Disallow: /
User-agent: LM Harvester
Disallow: /
User-agent: LMQueueBot
Disallow: /
User-agent: LT Scotland Checklink
Disallow: /
User-agent: LWP/natweb-bad-link-mailer
Disallow: /
User-agent: LWP::Simple
Disallow: /
User-agent: Labhoo
Disallow: /
User-agent: Lachesis
Disallow: /
User-agent: Lament
Disallow: /
User-agent: LamerExterminator
Disallow: /
User-agent: LapozzBot
Disallow: /
User-agent: LeapTag
Disallow: /
User-agent: LeechGet
Disallow: /
User-agent: LetsCrawl.com
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: Link Checker
Disallow: /
User-agent: Link Validator
Disallow: /
User-agent: LinkAlarm
Disallow: /
User-agent: LinkCheck Scanner
Disallow: /
User-agent: LinkCheck by Siteimprove.com
Disallow: /
User-agent: LinkChecker
Disallow: /
User-agent: LinkLint
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkSweeper
Disallow: /
User-agent: Linkbot
Disallow: /
User-agent: Linkman
Disallow: /
User-agent: Links SQL
Disallow: /
User-agent: LinksManager.com_bot
Disallow: /
User-agent: LiteFinder
Disallow: /
User-agent: LitlrBot
Disallow: /
User-agent: Little Grabber at Skanktale.com
Disallow: /
User-agent: Livelapbot
Disallow: /
User-agent: LoadTimeBot
Disallow: /
User-agent: LocalcomBot
Disallow: /
User-agent: LolongBot
Disallow: /
User-agent: LookBot
Disallow: /
User-agent: Lsearch/sondeur
Disallow: /
User-agent: Lycos_Spider
Disallow: /
User-agent: Lydia Entity Spider
Disallow: /
User-agent: LynnBot
Disallow: /
User-agent: Lytranslate
Disallow: /
User-agent: MAINSEEK_BOT
Disallow: /
User-agent: MFHttpScan
Disallow: /
User-agent: MFcrawler
Disallow: /
User-agent: MOSES
Disallow: /
User-agent: MQbot
Disallow: /
User-agent: MSE360
Disallow: /
User-agent: MSIndianWebcrawl
Disallow: /
User-agent: MSMOBOT
Disallow: /
User-agent: MSNPTC
Disallow: /
User-agent: MSRBOT
Disallow: /
User-agent: MT-Soft
Disallow: /
User-agent: MYCOMPANYBOT
Disallow: /
User-agent: MaSagool
Disallow: /
User-agent: Mag-Net
Disallow: /
User-agent: Magnet
Disallow: /
User-agent: Magus Bot
Disallow: /
User-agent: Mail.RU
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: Mail.Ru_Bot
Disallow: /
User-agent: Mail.Ru_Bot/2.0
Disallow: /
User-agent: Mammoth
Disallow: /
User-agent: Map robot
Disallow: /
User-agent: MarkWatch
Disallow: /
User-agent: McBot
Disallow: /
User-agent: Megite
Disallow: /
User-agent: MemacBot
Disallow: /
User-agent: Memo
Disallow: /
User-agent: MendeleyBot
Disallow: /
User-agent: Mercator-
Disallow: /
User-agent: MetaGloss
Disallow: /
User-agent: MetaURI API/2.0  metauri.com
Disallow: /
User-agent: MetaURI API/2.0 +metauri.com
Disallow: /
User-agent: Metaeuro Web Search
Disallow: /
User-agent: Metager2
Disallow: /
User-agent: Metaspinner
Disallow: /
User-agent: Mirror
Disallow: /
User-agent: Missigua Locator.
Disallow: /
User-agent: Miva
Disallow: /
User-agent: MojeekBot
Disallow: /
User-agent: MonkeyCrawl
Disallow: /
User-agent: MultiText
Disallow: /
User-agent: MyApp
Disallow: /
User-agent: MyEngines-US-Bot
Disallow: /
User-agent: MyFamilyBot
Disallow: /
User-agent: My_Little_SearchEngine_Project
Disallow: /
User-agent: Myra
Disallow: /
User-agent: NAMEPROTECT
Disallow: /
User-agent: NASA Search
Disallow: /
User-agent: NEC-MeshExplorer
Disallow: /
User-agent: NESSUS::SOAP
Disallow: /
User-agent: NG
Disallow: /
User-agent: NLESE USEPA
Disallow: /
User-agent: NLese
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: NRCan intranet crawler
Disallow: /
User-agent: NSDL_Search_Bot
Disallow: /
User-agent: NWSpider
Disallow: /
User-agent: Nambu
Disallow: /
User-agent: NatchCVS
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: NaverBot_dloader
Disallow: /
User-agent: NerdByNature.Bot
Disallow: /
User-agent: Nerima-crawl-
Disallow: /
User-agent: Nessus
Disallow: /
User-agent: Net::Trackback
Disallow: /
User-agent: NetID.com Bot
Disallow: /
User-agent: NetResearchServer
Disallow: /
User-agent: NetSeer crawler
Disallow: /
User-agent: NetSeer crawler/2.0
Disallow: /
User-agent: NetSeer
Disallow: /
User-agent: NetShift=
Disallow: /
User-agent: NetSongBot
Disallow: /
User-agent: NetSrcherP
Disallow: /
User-agent: Netcraft
Disallow: /
User-agent: Netprospector
Disallow: /
User-agent: Netsparker
Disallow: /
User-agent: NewMedhunt
Disallow: /
User-agent: NewsGatherer
Disallow: /
User-agent: NewsTroveBot
Disallow: /
User-agent: News_Search_App
Disallow: /
User-agent: Newsgroupreporter
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: NimbleCrawler
Disallow: /
User-agent: Ninja
Disallow: /
User-agent: NjuiceBot
Disallow: /
User-agent: Nogate
Disallow: /
User-agent: Norbert the Spider
Disallow: /
User-agent: NoteworthyBot
Disallow: /
User-agent: NuSearch Spider
Disallow: /
User-agent: Nusearch Spider
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: NutchCVS
Disallow: /
User-agent: NutchOrg
Disallow: /
User-agent: Nymesis
Disallow: /
User-agent: ODP entries t_st
Disallow: /
User-agent: OOZBOT
Disallow: /
User-agent: ORISBot
Disallow: /
User-agent: ORNL_Mercury
Disallow: /
User-agent: OWLER-BOT
Disallow: /
User-agent: ObjectsSearch
Disallow: /
User-agent: Obvius external linkcheck
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: Octopus
Disallow: /
User-agent: OmiExplorer_Bot
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: OmniWeb
Disallow: /
User-agent: OpenISearch
Disallow: /
User-agent: OpenIntelligenceData
Disallow: /
User-agent: OpenLink Virtuoso RDF crawler
Disallow: /
User-agent: OpenSearchServer_Bot
Disallow: /
User-agent: OpiDig
Disallow: /
User-agent: Oracle Secure Enterprise Search
Disallow: /
User-agent: OrangeBot
Disallow: /
User-agent: OsO
Disallow: /
User-agent: OutfoxBot
Disallow: /
User-agent: OutfoxMelonBot
Disallow: /
User-agent: P3P Client
Disallow: /
User-agent: PBBOT
Disallow: /
User-agent: PHP version tracker
Disallow: /
User-agent: PHPCrawl
Disallow: /
User-agent: PIENO robot
Disallow: /
User-agent: PROXY crawler
Disallow: /
User-agent: PSS-Bot
Disallow: /
User-agent: PWeBot
Disallow: /
User-agent: PageBitesHyperBot
Disallow: /
User-agent: PageDown
Disallow: /
User-agent: PageFetcher
Disallow: /
User-agent: PagePeeker
Disallow: /
User-agent: PageRank Monitor
Disallow: /
User-agent: Pagebull
Disallow: /
User-agent: Panopy Bot
Disallow: /
User-agent: Pansophica
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: PaperLiBot/2.1
Disallow: /
User-agent: Pathtraq
Disallow: /
User-agent: Pattern
Disallow: /
User-agent: Patwebbot
Disallow: /
User-agent: PaxleFramework
Disallow: /
User-agent: PerMan Surfer
Disallow: /
User-agent: PhpDig
Disallow: /
User-agent: PicoSearch
Disallow: /
User-agent: Pita
Disallow: /
User-agent: PiyushBot
Disallow: /
User-agent: Plucker
Disallow: /
User-agent: Plukkie
Disallow: /
User-agent: Plumtree 6.0
Disallow: /
User-agent: PoCoHTTP
Disallow: /
User-agent: Pockey-GetHTML
Disallow: /
User-agent: Pockey
Disallow: /
User-agent: Pogodak.co.yu
Disallow: /
User-agent: Poirot
Disallow: /
User-agent: Pompos
Disallow: /
User-agent: Poodle predictor
Disallow: /
User-agent: PopScreenBot
Disallow: /
User-agent: PostPost
Disallow: /
User-agent: PrivacyFinder
Disallow: /
User-agent: ProjectWF-java-test-crawler
Disallow: /
User-agent: Proxem WebSearch
Disallow: /
User-agent: PsBot
Disallow: /
User-agent: Pump
Disallow: /
User-agent: PythonWikipediaBot
Disallow: /
User-agent: QEAVis agent
Disallow: /
User-agent: QFKBot
Disallow: /
User-agent: Qualidator.com Bot
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: QueryN.Metasearch
Disallow: /
User-agent: Quintura-Crw
Disallow: /
User-agent: QunarBot
Disallow: /
User-agent: QweeryBot
Disallow: /
User-agent: Qweery_robot.txt_CheckBot
Disallow: /
User-agent: R6_FeedFetcher
Disallow: /
User-agent: R6_VoteReader
Disallow: /
User-agent: RAMPyBot
Disallow: /
User-agent: REBI-Shoveler
Disallow: /
User-agent: REBI-shoveler
Disallow: /
User-agent: RIIGHTBOT/RIIGHT-
Disallow: /
User-agent: RPT-HTTPClient
Disallow: /
User-agent: RaBot
Disallow: /
User-agent: Radian6
Disallow: /
User-agent: RankurBot/Rankur
Disallow: /
User-agent: Rational SiteCheck
Disallow: /
User-agent: RcStartBot
Disallow: /
User-agent: Reaper
Disallow: /
User-agent: Recorder
Disallow: /
User-agent: RedBot
Disallow: /
User-agent: RedCarpet
Disallow: /
User-agent: Riddler
Disallow: /
User-agent: RiseNetBot
Disallow: /
User-agent: RiverGlassScanner
Disallow: /
User-agent: RiverglassScanner
Disallow: /
User-agent: RoboPal
Disallow: /
User-agent: Robosourcer
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Rome Client
Disallow: /
User-agent: Rondello
Disallow: /
User-agent: Rotondo
Disallow: /
User-agent: Runnk RSS aggregator
Disallow: /
User-agent: Runnk online rss reader
Disallow: /
User-agent: S2Bot
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: SBL-BOT
Disallow: /
User-agent: SCFCrawler/Nutch-1.8
Disallow: /
User-agent: SCrawlTest
Disallow: /
User-agent: SEB Spider
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: SOFT411 Directory
Disallow: /
User-agent: SQ Webscanner
Disallow: /
User-agent: STEGMANN-Bot
Disallow: /
User-agent: SURF
Disallow: /
User-agent: SWEBot
Disallow: /
User-agent: SafariBookmarkChecker
Disallow: /
User-agent: SaladSpoon/ShopSalad
Disallow: /
User-agent: Sapienti/Indexer
Disallow: /
User-agent: Scich
Disallow: /
User-agent: ScientificCommons.org
Disallow: /
User-agent: ScollSpider
Disallow: /
User-agent: ScooperBot
Disallow: /
User-agent: Scooter
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Scumbot
Disallow: /
User-agent: SearQuBot/SearQuBot v1.0
Disallow: /
User-agent: Search Publisher
Disallow: /
User-agent: Search-Channel
Disallow: /
User-agent: Search-Engine-Studio
Disallow: /
User-agent: SearchBlox
Disallow: /
User-agent: SearchBot
Disallow: /
User-agent: SearchIt-Bot
Disallow: /
User-agent: SearchSpider.com
Disallow: /
User-agent: Searcharoo.NET
Disallow: /
User-agent: Searchmee! Spider
Disallow: /
User-agent: SearchnowBot_v1
Disallow: /
User-agent: Seekbot
Disallow: /
User-agent: Seeker.lookseek.com
Disallow: /
User-agent: Selflinkchecker
Disallow: /
User-agent: Semager
Disallow: /
User-agent: Semantifire1
Disallow: /
User-agent: ShablastBot
Disallow: /
User-agent: ShadowWebAnalyzer
Disallow: /
User-agent: Shareaza
Disallow: /
User-agent: Shelob
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShowLinks
Disallow: /
User-agent: ShowyouBot
Disallow: /
User-agent: Siphon
Disallow: /
User-agent: SiteArchive
Disallow: /
User-agent: SiteCheck-sitecrawl by Siteimprove.com
Disallow: /
User-agent: SiteFinder
Disallow: /
User-agent: SiteGuardBot
Disallow: /
User-agent: SiteOrbiter
Disallow: /
User-agent: SiteSucker
Disallow: /
User-agent: SiteSweeper
Disallow: /
User-agent: SiteXpert
Disallow: /
User-agent: SkimBot
Disallow: /
User-agent: SkimWordsBot
Disallow: /
User-agent: SkreemRBot
Disallow: /
User-agent: Skywalker
Disallow: /
User-agent: Sleipnir
Disallow: /
User-agent: Smarte Bot
Disallow: /
User-agent: Snake
Disallow: /
User-agent: Snappy
Disallow: /
User-agent: Snoopy
Disallow: /
User-agent: SocSciBot
Disallow: /
User-agent: Sokitomi crawl
Disallow: /
User-agent: Space Bison
Disallow: /
User-agent: Space Fung
Disallow: /
User-agent: SpaceBison
Disallow: /
User-agent: Spatineo Monitor Controller
Disallow: /
User-agent: Spatineo Serval Controller
Disallow: /
User-agent: Spatineo Serval GetMapBot
Disallow: /
User-agent: Sphere Scout&v
Disallow: /
User-agent: Sphere Scout
Disallow: /
User-agent: Sphider
Disallow: /
User-agent: Sphider2
Disallow: /
User-agent: Spider
Disallow: /
User-agent: Spider.TerraNautic.net
Disallow: /
User-agent: SpiderEngine
Disallow: /
User-agent: SpiderKU
Disallow: /
User-agent: SpiderMan
Disallow: /
User-agent: Spinn3r
Disallow: /
User-agent: Spinne
Disallow: /
User-agent: Squid-Prefetch
Disallow: /
User-agent: SquidClamAV_Redirector
Disallow: /
User-agent: Sqworm
Disallow: /
User-agent: SrevBot
Disallow: /
User-agent: StackRambler
Disallow: /
User-agent: StarDownloader
Disallow: /
User-agent: Steeler
Disallow: /
User-agent: Stripper
Disallow: /
User-agent: Stumbler
Disallow: /
User-agent: Sucker
Disallow: /
User-agent: SumeetBot
Disallow: /
User-agent: SumitBot
Disallow: /
User-agent: SummizeBot
Disallow: /
User-agent: SummizeFeedReader
Disallow: /
User-agent: Sunrise XP
Disallow: /
User-agent: SuperLumin Downloader
Disallow: /
User-agent: SuperPagesBot
Disallow: /
User-agent: Supybot
Disallow: /
User-agent: SurfControl
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: SygolBot
Disallow: /
User-agent: SynapticWalker
Disallow: /
User-agent: Syntryx ANT Scout Chassis Pheromone
Disallow: /
User-agent: SystemSearch-robot
Disallow: /
User-agent: T-H-U-N-D-E-R-S-T-O-N-E
Disallow: /
User-agent: TAMU_CRAWLER
Disallow: /
User-agent: TAUSDataBot
Disallow: /
User-agent: TERAGRAM_CRAWLER
Disallow: /
User-agent: TITAN
Disallow: /
User-agent: TPSystem
Disallow: /
User-agent: TRAAZI
Disallow: /
User-agent: TREX
Disallow: /
User-agent: TSM Translation-Search-Machine
Disallow: /
User-agent: Tailrank
Disallow: /
User-agent: Talkro Web-Shot
Disallow: /
User-agent: TapuzBot
Disallow: /
User-agent: Tarantula
Disallow: /
User-agent: TargetYourNews.com bot
Disallow: /
User-agent: TeamSoft WinInet Component
Disallow: /
User-agent: Tecomi Bot
Disallow: /
User-agent: TeezirBot
Disallow: /
User-agent: Teleport Pro
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Teradex Mapper
Disallow: /
User-agent: TerrawizBot
Disallow: /
User-agent: TestBot
Disallow: /
User-agent: TextBot
Disallow: /
User-agent: The Dyslexalizer
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: TheUsefulbot_
Disallow: /
User-agent: Theophrastus
Disallow: /
User-agent: ThumbBot
Disallow: /
User-agent: Thumbnail.CZ robot
Disallow: /
User-agent: TinEye
Disallow: /
User-agent: TopServer PHP
Disallow: /
User-agent: Topicalizer
Disallow: /
User-agent: Toplistbot
Disallow: /
User-agent: Touche
Disallow: /
User-agent: TourlentaScanner
Disallow: /
User-agent: TranSGeniKBot
Disallow: /
User-agent: TravelBot
Disallow: /
User-agent: TravelLazerBot
Disallow: /
User-agent: Treezy
Disallow: /
User-agent: TridentSpider
Disallow: /
User-agent: Trovator
Disallow: /
User-agent: TsWebBot
Disallow: /
User-agent: TulipChain
Disallow: /
User-agent: TurnitinBot/1.5
Disallow: /
User-agent: TutorGigBot
Disallow: /
User-agent: TweetedTimes Bot
Disallow: /
User-agent: TweetmemeBot
Disallow: /
User-agent: TwengaBot-Discover
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: Twikle
Disallow: /
User-agent: Twisted PageGetter
Disallow: /
User-agent: Twitturls
Disallow: /
User-agent: Twitturly
Disallow: /
User-agent: TygoBot
Disallow: /
User-agent: TygoProwler
Disallow: /
User-agent: Typhoeus
Disallow: /
User-agent: U.S. Government Printing Office
Disallow: /
User-agent: UFAM-crawler-
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URL-Checker
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: URLBlaze
Disallow: /
User-agent: USAF AFKN K2SPIDER
Disallow: /
User-agent: USS-Cosmix
Disallow: /
User-agent: USyd-NLP-Spider
Disallow: /
User-agent: UdmSearch
Disallow: /
User-agent: Ultraseek
Disallow: /
User-agent: UnChaos
Disallow: /
User-agent: Unitek UniEngine
Disallow: /
User-agent: UniversalSearch
Disallow: /
User-agent: UofTDB_experiment
Disallow: /
User-agent: UrlDispatcher
Disallow: /
User-agent: VIP
Disallow: /
User-agent: VIPr
Disallow: /
User-agent: VMBot
Disallow: /
User-agent: VSE
Disallow: /
User-agent: Vacobot
Disallow: /
User-agent: Vacuum
Disallow: /
User-agent: VadixBot
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: Validator
Disallow: /
User-agent: VerbstarBot
Disallow: /
User-agent: VeriCiteCrawler/Nutch-2.2.1
Disallow: /
User-agent: Verifactrola
Disallow: /
User-agent: Verity-URL-Gateway
Disallow: /
User-agent: Vermut
Disallow: /
User-agent: VisBot
Disallow: /
User-agent: VisWeb
Disallow: /
User-agent: Vishal For CLIA
Disallow: /
User-agent: Vital Search'n Urchin
Disallow: /
User-agent: VocusBot
Disallow: /
User-agent: Vortex
Disallow: /
User-agent: WEP Search
Disallow: /
User-agent: WEPA
Disallow: /
User-agent: WIRE
Disallow: /
User-agent: WISENutbot
Disallow: /
User-agent: WISEbot
Disallow: /
User-agent: WMCAI-robot
Disallow: /
User-agent: WMSBot
Disallow: /
User-agent: WWW-Mechanize
Disallow: /
User-agent: Water Conserve Portal
Disallow: /
User-agent: Water Conserve Spider
Disallow: /
User-agent: WatzBot
Disallow: /
User-agent: Wavefire
Disallow: /
User-agent: Waypath
Disallow: /
User-agent: Wazzup
Disallow: /
User-agent: Wazzup1.0.4800
Disallow: /
User-agent: WeBot
Disallow: /
User-agent: WeRelateBot
Disallow: /
User-agent: Web CEO Online robot
Disallow: /
User-agent: Web Downloader
Disallow: /
User-agent: Web Link Validator
Disallow: /
User-agent: Web Magnet
Disallow: /
User-agent: Web Site Downloader
Disallow: /
User-agent: Web-Sniffer
Disallow: /
User-agent: WebAuto/3.40
Disallow: /
User-agent: WebBOT
Disallow: /
User-agent: WebBot
Disallow: /
User-agent: WebCapture
Disallow: /
User-agent: WebCopy
Disallow: /
User-agent: WebCorp
Disallow: /
User-agent: WebCrawler
Disallow: /
User-agent: WebDownloader for X
Disallow: /
User-agent: WebGather
Disallow: /
User-agent: WebImages
Disallow: /
User-agent: WebIndex
Disallow: /
User-agent: WebIndexer
Disallow: /
User-agent: WebMiner
Disallow: /
User-agent: WebPix
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebRipper
Disallow: /
User-agent: WebSauger 1.20k
Disallow: /
User-agent: WebSite
Disallow: /
User-agent: WebSpear
Disallow: /
User-agent: WebSpider
Disallow: /
User-agent: WebTrafficExpress
Disallow: /
User-agent: WebTrends Link Analyzer
Disallow: /
User-agent: WebWasher
Disallow: /
User-agent: WebXM
Disallow: /
User-agent: WebaltBot
Disallow: /
User-agent: Webclipping.com
Disallow: /
User-agent: Webdup
Disallow: /
User-agent: Webinator-WBI
Disallow: /
User-agent: Webinator-search2.fasthealth.com
Disallow: /
User-agent: WebmasterCoffee
Disallow: /
User-agent: Webscan
Disallow: /
User-agent: Website Downloader
Disallow: /
User-agent: Website Explorer
Disallow: /
User-agent: Webster
Disallow: /
User-agent: Webwasher
Disallow: /
User-agent: Weddings.info Bot
Disallow: /
User-agent: Whacker
Disallow: /
User-agent: Whirlpool Web Engine
Disallow: /
User-agent: WikiaBot
Disallow: /
User-agent: Wikio
Disallow: /
User-agent: WinHTTP Example
Disallow: /
User-agent: WinHttp.WinHttpRequest.
Disallow: /
User-agent: WorQmada
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: Wysigot
Disallow: /
User-agent: X-clawler
Disallow: /
User-agent: X-crawler
Disallow: /
User-agent: Xaldon
Disallow: /
User-agent: Xaldon_WebSpider
Disallow: /
User-agent: Xerka MetaBot
Disallow: /
User-agent: Xerka WebBot
Disallow: /
User-agent: XmarksFetch
Disallow: /
User-agent: Y!TunnelPro
Disallow: /
User-agent: Yanga WorldSearch Bot
Disallow: /
User-agent: YebolBot
Disallow: /
User-agent: YioopBot
Disallow: /
User-agent: Yoono
Disallow: /
User-agent: Yoriwa
Disallow: /
User-agent: YottaCars_Bot
Disallow: /
User-agent: Z-Add Link Checker
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zing-BottaBot
Disallow: /
User-agent: ZipppBot
Disallow: /
User-agent: ZoomSpider
Disallow: /
User-agent: Zotag Search
Disallow: /
User-agent: ZuiBot
Disallow: /
User-agent: Zyte
Disallow: /
User-agent: abot
Disallow: /
User-agent: accoona
Disallow: /
User-agent: acquia-crawler
Disallow: /
User-agent: adbeat_bot
Disallow: /
User-agent: adminshop.com
Disallow: /
User-agent: aggregator:Vocus
Disallow: /
User-agent: ah-ha.com crawler
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: aipbot
Disallow: /
User-agent: antibot-
Disallow: /
User-agent: appid: s~stremor-crawler-
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: aria eQualizer
Disallow: /
User-agent: arianna.libero.it
Disallow: /
User-agent: athenusbot
Disallow: /
User-agent: attach
Disallow: /
User-agent: attrakt
Disallow: /
User-agent: attributor
Disallow: /
User-agent: augurfind
Disallow: /
User-agent: autoemailspider
Disallow: /
User-agent: autowebdir
Disallow: /
User-agent: axfeedsbot
Disallow: /
User-agent: b2w
Disallow: /
User-agent: baypup
Disallow: /
User-agent: besserscheitern-crawl
Disallow: /
User-agent: betaBot
Disallow: /
User-agent: bintellibot
Disallow: /
User-agent: bladder fusion
Disallow: /
User-agent: bnf.fr_bot
Disallow: /
User-agent: bosug
Disallow: /
User-agent: bot
Disallow: /
User-agent: bottybot
Disallow: /
User-agent: btbot
Disallow: /
User-agent: bumblebee
Disallow: /
User-agent: byindia
Disallow: /
User-agent: byteserver
Disallow: /
User-agent: bzBot
Disallow: /
User-agent: c r a w l 3 r
Disallow: /
User-agent: cataguru
Disallow: /
User-agent: ccubee
Disallow: /
User-agent: cfetch
Disallow: /
User-agent: cis455crawler
Disallow: /
User-agent: citeseerxbot
Disallow: /
User-agent: cizilla.com/Cizilla-
Disallow: /
User-agent: coccoc
Disallow: /
User-agent: combine
Disallow: /
User-agent: conpilot
Disallow: /
User-agent: contype
Disallow: /
User-agent: cookieNET
Disallow: /
User-agent: core-project
Disallow: /
User-agent: crabbyBot
Disallow: /
User-agent: crawl
Disallow: /
User-agent: crawl.UserAgent
Disallow: /
User-agent: crawler-upgrade-config
Disallow: /
User-agent: crawler
Disallow: /
User-agent: crawler.kpricorn.org
Disallow: /
User-agent: crawler43.ejupiter.com
Disallow: /
User-agent: crawler4j
Disallow: /
User-agent: crawler@
Disallow: /
User-agent: crawler_for_infomine
Disallow: /
User-agent: cs-crawler
Disallow: /
User-agent: culsearch/culs
Disallow: /
User-agent: cvaulev
Disallow: /
User-agent: datascape robot
Disallow: /
User-agent: deepnet crawler
Disallow: /
User-agent: dejan
Disallow: /
User-agent: del.icio.us-thumbnails
Disallow: /
User-agent: didaxusbot
Disallow: /
User-agent: domaincrawler
Disallow: /
User-agent: dragonfly
Disallow: /
User-agent: drone
Disallow: /
User-agent: dtSearchSpider
Disallow: /
User-agent: dumbot
Disallow: /
User-agent: eCairn-Grabber
Disallow: /
User-agent: eChooseBot
Disallow: /
User-agent: eStyleSearch
Disallow: /
User-agent: eSyndiCat Bot
Disallow: /
User-agent: ebingbong
Disallow: /
User-agent: egothor
Disallow: /
User-agent: eidetica.com/spider
Disallow: /
User-agent: endeca
Disallow: /
User-agent: exactseek.com
Disallow: /
User-agent: exooba/exooba crawler
Disallow: /
User-agent: ez-robot
Disallow: /
User-agent: f-bot test pilot
Disallow: /
User-agent: factbot
Disallow: /
User-agent: falcon
Disallow: /
User-agent: fast-search-engine
Disallow: /
User-agent: fastbot crawler
Disallow: /
User-agent: fastbot.de crawler
Disallow: /
User-agent: fembot
Disallow: /
User-agent: fetch_ici
Disallow: /
User-agent: fgcrawler
Disallow: /
User-agent: findlinks
Disallow: /
User-agent: flunky
Disallow: /
User-agent: focused_crawler
Disallow: /
User-agent: fr_crawler
Disallow: /
User-agent: freshlinks.exe
Disallow: /
User-agent: froGgle
Disallow: /
User-agent: frodo.at
Disallow: /
User-agent: full_breadth_crawler
Disallow: /
User-agent: gazz
Disallow: /
User-agent: generate_infomine_category_classifiers
Disallow: /
User-agent: genevabot
Disallow: /
User-agent: genieBot
Disallow: /
User-agent: geometabot
Disallow: /
User-agent: gnome-vfs
Disallow: /
User-agent: gold crawler
Disallow: /
User-agent: goroam
Disallow: /
User-agent: gotit
Disallow: /
User-agent: grapeFX
Disallow: /
User-agent: grbot
Disallow: /
User-agent: grub crawler
Disallow: /
User-agent: grub
Disallow: /
User-agent: gvcbot.com
Disallow: /
User-agent: hcat
Disallow: /
User-agent: hclsreport-crawler
Disallow: /
User-agent: healia/healia
Disallow: /
User-agent: here will be link to crawler site
Disallow: /
User-agent: heritrix bot
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: heritrix/3.1.0
Disallow: /
User-agent: heritrix/3.1.1
Disallow: /
User-agent: hijbul-heritrix-crawler
Disallow: /
User-agent: hitcrawler_
Disallow: /
User-agent: hoge
Disallow: /
User-agent: holmes
Disallow: /
User-agent: ht://check
Disallow: /
User-agent: htdig
Disallow: /
User-agent: hul-wax
Disallow: /
User-agent: iCab
Disallow: /
User-agent: iCopyright Conductor
Disallow: /
User-agent: iHWebChecker
Disallow: /
User-agent: iSearch
Disallow: /
User-agent: iSiloX
Disallow: /
User-agent: iVia Page Fetcher
Disallow: /
User-agent: iVia Site Checker
Disallow: /
User-agent: iVia
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ibuena
Disallow: /
User-agent: iim_405
Disallow: /
User-agent: imagefortress
Disallow: /
User-agent: imds_monitor
Disallow: /
User-agent: imo-google-robot-intelink
Disallow: /
User-agent: indylabs_marius
Disallow: /
User-agent: inetbot
Disallow: /
User-agent: info seeker
Disallow: /
User-agent: infomine.ucr.edu
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: its-learning crawler
Disallow: /
User-agent: kalooga
Disallow: /
User-agent: kbeta1
Disallow: /
User-agent: kinjabot
Disallow: /
User-agent: kmbot-
Disallow: /
User-agent: kmccrew Bot Search
Disallow: /
User-agent: kuloko-bot
Disallow: /
User-agent: kulturarw3
Disallow: /
User-agent: labourunions411
Disallow: /
User-agent: lachesis
Disallow: /
User-agent: lftp
Disallow: /
User-agent: libcrawl
Disallow: /
User-agent: libiViaCore
Disallow: /
User-agent: libwww-perl
Disallow: /
User-agent: likse
Disallow: /
User-agent: link_checker
Disallow: /
User-agent: linkbot
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: locust
Disallow: /
User-agent: lssbot
Disallow: /
User-agent: lwp-request
Disallow: /
User-agent: masidani_bot
Disallow: /
User-agent: matentzn at cs dot man dot ac dot uk
Disallow: /
User-agent: maxamine.com--robot
Disallow: /
User-agent: maxamine.com-robot
Disallow: /
User-agent: maxomobot
Disallow: /
User-agent: medrabbit
Disallow: /
User-agent: mercuryboard_user_agent_sql_injection.nasl
Disallow: /
User-agent: metacarta
Disallow: /
User-agent: metager2-verification-bot
Disallow: /
User-agent: metal crawler
Disallow: /
User-agent: metaquerier.cs.uiuc.edu
Disallow: /
User-agent: mini-robot
Disallow: /
User-agent: miniRank
Disallow: /
User-agent: minibot
Disallow: /
User-agent: mod_accessibility
Disallow: /
User-agent: moduna.com
Disallow: /
User-agent: mowserbot
Disallow: /
User-agent: my-heritrix-crawler
Disallow: /
User-agent: mycrawler
Disallow: /
User-agent: nabot
Disallow: /
User-agent: nabot_
Disallow: /
User-agent: naver
Disallow: /
User-agent: naverbookmarkcrawler
Disallow: /
User-agent: nestReader
Disallow: /
User-agent: news bot
Disallow: /
User-agent: nextthing.org
Disallow: /
User-agent: nicebot
Disallow: /
User-agent: niki-bot
Disallow: /
User-agent: nimbus-1
Disallow: /
User-agent: ninetowns
Disallow: /
User-agent: nuSearch Spider
Disallow: /
User-agent: nu_tch-princeton/Nu_tch
Disallow: /
User-agent: nuggetize.com BOT
Disallow: /
User-agent: nutch1/huntsman
Disallow: /
User-agent: nys-crawler
Disallow: /
User-agent: oegp v.
Disallow: /
User-agent: omnifind
Disallow: /
User-agent: online link validator
Disallow: /
User-agent: optidiscover
Disallow: /
User-agent: ornl_crawler_1
Disallow: /
User-agent: osis-project.jp
Disallow: /
User-agent: owsBot
Disallow: /
User-agent: ozelot
Disallow: /
User-agent: page_verifier
Disallow: /
User-agent: pamsnbot.htm
Disallow: /
User-agent: parasite
Disallow: /
User-agent: parsijoo
Disallow: /
User-agent: pd-crawler
Disallow: /
User-agent: penthesila
Disallow: /
User-agent: perform_crawl
Disallow: /
User-agent: personal ultimate crawler
Disallow: /
User-agent: pipBot
Disallow: /
User-agent: pipeLiner
Disallow: /
User-agent: pixfinder
Disallow: /
User-agent: plaNETWORK Bot Search
Disallow: /
User-agent: pogodak.ba
Disallow: /
User-agent: polybot
Disallow: /
User-agent: psbot-page
Disallow: /
User-agent: psycheclone
Disallow: /
User-agent: pub-crawler
Disallow: /
User-agent: pucl
Disallow: /
User-agent: pulseBot (pulse Web Miner)
Disallow: /
User-agent: pythonic-crawler
Disallow: /
User-agent: q1
Disallow: /
User-agent: qualidade
Disallow: /
User-agent: quest.durato
Disallow: /
User-agent: r2iBot
Disallow: /
User-agent: radian6_linkcheck
Disallow: /
User-agent: research robot
Disallow: /
User-agent: robotek
Disallow: /
User-agent: rtgibot
Disallow: /
User-agent: scSpider
Disallow: /
User-agent: search x-bot
Disallow: /
User-agent: search.KumKie.com
Disallow: /
User-agent: search.updated.com
Disallow: /
User-agent: search.usgs.gov
Disallow: /
User-agent: searchengine
Disallow: /
User-agent: searchhippo.com
Disallow: /
User-agent: searchmarking	
Disallow: /
User-agent: searchmarks
Disallow: /
User-agent: searchmee_v
Disallow: /
User-agent: searchmining
Disallow: /
User-agent: seeqpod-vertical-crawler
Disallow: /
User-agent: semanticdiscovery
Disallow: /
User-agent: semisearch
Disallow: /
User-agent: sherlock
Disallow: /
User-agent: siclab
Disallow: /
User-agent: silk
Disallow: /
User-agent: slow-crawler
Disallow: /
User-agent: smart-crawler
Disallow: /
User-agent: smartwit.com
Disallow: /
User-agent: snap.com beta crawler
Disallow: /
User-agent: snookit/Snookit
Disallow: /
User-agent: special_archiver
Disallow: /
User-agent: sportcrew-Bot
Disallow: /
User-agent: sqlmap
Disallow: /
User-agent: sslbot
Disallow: /
User-agent: statbot
Disallow: /
User-agent: statcrawler
Disallow: /
User-agent: statedept-crawler
Disallow: /
User-agent: stero
Disallow: /
User-agent: suchclip
Disallow: /
User-agent: superbot.com
Disallow: /
User-agent: swish-e
Disallow: /
User-agent: s~stremor-crawler
Disallow: /
User-agent: tScholarsBot
Disallow: /
User-agent: targetblaster.com
Disallow: /
User-agent: taxinomiabot
Disallow: /
User-agent: testBOT
Disallow: /
User-agent: testbot
Disallow: /
User-agent: testing of bot
Disallow: /
User-agent: thatrobotsite.com
Disallow: /
User-agent: theusefulbot
Disallow: /
User-agent: thumbshots-de-bot
Disallow: /
User-agent: tigerbot
Disallow: /
User-agent: to-dress_ru_bot_
Disallow: /
User-agent: to-night-Bot
Disallow: /
User-agent: topicblogs
Disallow: /
User-agent: topyx-crawler
Disallow: /
User-agent: travel-search
Disallow: /
User-agent: twinuffbot
Disallow: /
User-agent: uberbot
Disallow: /
User-agent: ucb-nutch
Disallow: /
User-agent: unchaos_crawler_
Disallow: /
User-agent: updated
Disallow: /
User-agent: url_gather
Disallow: /
User-agent: urlchecker
Disallow: /
User-agent: usasearch
Disallow: /
User-agent: vermut
Disallow: /
User-agent: versus crawler
Disallow: /
User-agent: versus.integis.ch
Disallow: /
User-agent: viasarchivinginformation.html
Disallow: /
User-agent: virus-detector
Disallow: /
User-agent: virus_detector
Disallow: /
User-agent: vlad
Disallow: /
User-agent: vlsearch
Disallow: /
User-agent: voyager-hc
Disallow: /
User-agent: voyager-partner-deep
Disallow: /
User-agent: voyager
Disallow: /
User-agent: vspider
Disallow: /
User-agent: w3m
Disallow: /
User-agent: w3search
Disallow: /
User-agent: wacbot
Disallow: /
User-agent: wastrix
Disallow: /
User-agent: wauuu engine/Wauuu
Disallow: /
User-agent: wbdbot
Disallow: /
User-agent: web-agent
Disallow: /
User-agent: webGobbler
Disallow: /
User-agent: webLyzard
Disallow: /
User-agent: webbandit
Disallow: /
User-agent: webbot
Disallow: /
User-agent: webbul-bot
Disallow: /
User-agent: webcheck 1
Disallow: /
User-agent: webcollage
Disallow: /
User-agent: webcrawl.net
Disallow: /
User-agent: webcrawler
Disallow: /
User-agent: webfetch
Disallow: /
User-agent: websearchbench
Disallow: /
User-agent: websitemirror
Disallow: /
User-agent: websphinx.test
Disallow: /
User-agent: wenbin/search
Disallow: /
User-agent: wget
Disallow: /
User-agent: wikiwix-bot-
Disallow: /
User-agent: wish-la
Disallow: /
User-agent: wish-project
Disallow: /
User-agent: wisponbot
Disallow: /
User-agent: wminer
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: worldshop
Disallow: /
User-agent: wume_crawler
Disallow: /
User-agent: wwwrobot
Disallow: /
User-agent: wwwster
Disallow: /
User-agent: wwwxref
Disallow: /
User-agent: xirq
Disallow: /
User-agent: xqrobot
Disallow: /
User-agent: yBot
Disallow: /
User-agent: yarienavoir.net
Disallow: /
User-agent: yellowJacket
Disallow: /
User-agent: yes
Disallow: /
User-agent: yolinkBot
Disallow: /
User-agent: yoogliFetchAgent
Disallow: /
User-agent: yoono
Disallow: /
User-agent: you-dir
Disallow: /
User-agent: zagrebin
Disallow: /
User-agent: zedzo.digest
Disallow: /
User-agent: zedzo.validate
Disallow: /
User-agent: zermelo
Disallow: /
User-agent: zibber-v
Disallow: /
User-agent: zimeno
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: 80legs
Disallow: /
User-agent: Barkrowler
Disallow: /
User-agent: BecomeBot
Disallow: /
User-agent: CLIPish
Disallow: /
User-agent: COMODO
Disallow: /
User-agent: Charlotte
Disallow: /
User-agent: Clickagy Intelligence Bot v2
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: Comodo-Certificates-Spider
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: Curious
Disallow: /
User-agent: DinoPing
Disallow: /
User-agent: Dow\ Jones\ Searchbot
Disallow: /
User-agent: Enterprise_Search
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: EyeMonIT
Disallow: /
User-agent: EzineArticlesLinkScanner
Disallow: /
User-agent: FTRF\:\ Friendly
Disallow: /
User-agent: Feedpresso Content Index Bot
Disallow: /
User-agent: HTTP_Request
Disallow: /
User-agent: HTTP_Request2
Disallow: /
User-agent: Indy\ Library
Disallow: /
User-agent: Jakarta\ Commons-HttpClient
Disallow: /
User-agent: Jeeves
Disallow: /
User-agent: Jetbot/1.0
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: Lynx
Disallow: /
User-agent: MSIE\ or\ Firefox\ mutant
Disallow: /
User-agent: Mattermost-Bot
Disallow: /
User-agent: Mozilla/4.0 (compatible  BullsEye  Windows 95)
Disallow: /
User-agent: NCBot
Disallow: /
User-agent: NetcraftSurveyAgent
Disallow: /
User-agent: Neticle Crawler
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: PEAR
Disallow: /
User-agent: PagesInventory
Disallow: /
User-agent: PeoplePal
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: ProCogSEOBot
Disallow: /
User-agent: PubMatic Crawler Bot
Disallow: /
User-agent: QuerySeekerSpider
Disallow: /
User-agent: SEO-CRAWLING
Disallow: /
User-agent: ScreenerBot
Disallow: /
User-agent: SocialSearcher
Disallow: /
User-agent: SolomonoBot
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: Stanford
Disallow: /
User-agent: Stanford Comp Sci
Disallow: /
User-agent: Synapse
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: The\ Incutio\ XML-RPC\ PHP\ Library
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: WEBSITEtheWEB\.COM
Disallow: /
User-agent: WebInDetail\.com
Disallow: /
User-agent: Wget
Disallow: /
User-agent: Wotbot
Disallow: /
User-agent: YamanaLab-Robot
Disallow: /
User-agent: Zend_Http_Client
Disallow: /
User-agent: ZmEu
Disallow: /
User-agent: accelobot
Disallow: /
User-agent: awcheckBot
Disallow: /
User-agent: brandwatch.net
Disallow: /
User-agent: dataprovider\.com
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: dotnetdotcom
Disallow: /
User-agent: http://MegaIndex.ru 
Disallow: /
User-agent: http://linkfluence.com 
Disallow: /
User-agent: http://mediacloud.org 
Disallow: /
User-agent: http://www.SearchEngineWorld.com  bot
Disallow: /
User-agent: http://www.WebmasterWorld.com  bot
Disallow: /
User-agent: http://www.trendiction.de/bot 
Disallow: /
User-agent: https://developers.snap.com/robots 
Disallow: /
User-agent: https://help.qwant.com/bot/ 
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: ip\-web\-crawler\.com
Disallow: /
User-agent: lindex\.com
Disallow: /
User-agent: linkdex\.com
Disallow: /
User-agent: lipperhey
Disallow: /
User-agent: looksmart
Disallow: /
User-agent: ltbot
Disallow: /
User-agent: magpie\-crawler
Disallow: /
User-agent: netEstate\ NE\ Crawler
Disallow: /
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: purebot
Disallow: /
User-agent: rojerbot
Disallow: /
User-agent: sentibot
Disallow: /
User-agent: sogou
Disallow: /
User-agent: uCrawler
Disallow: /
User-agent: visaduhoc\.info
Disallow: /
User-agent: www\.integromedb\.org
Disallow: /
User-agent: xpymep\.exe
Disallow: /

Effective AI / crawl policy (Google posture, §3.3)

CrawlPostureConsec. 5xxReasonAs of
CC-MAIN-2026-25 open 0 2xx: robots.txt present, no full disallow in effect 2026-06-07T01:41:18+00:00

robots.txt

CrawlStatusBytesGroupsAllowDisallow SitemapsWildcard *Disallow-allAI tokensAI verdicts
CC-MAIN-2026-25 200 78,329 1964 1 1980 1 yes no panscient.com, Scrapy, PetalBot, omgili, omgilibot panscient.com:block, scrapy:block, petalbot:block, omgili:block, omgilibot:block
CC-MAIN-2026-25 200 78,482 1970 1 1986 0 yes no panscient.com, Applebot, CCBot, Scrapy, Applebot-Extended, Bytespider, PetalBot, omgili, omgilibot —

llms.txt

Collected by our own polite crawler, not Common Crawl — Common Crawl doesn't capture /llms.txt. See Methodology.

No llms.txt records for this domain.

Inferred tech stack hypothesis

No tech inference for this domain.