French
protection
CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.





diable

Custom Search






Crawlers and spiders who can visit your website

Here is the list of crawlers used by CrawlTrack; that list is updated regularly. As you will see, it's already a very long list but there is still a lot of spiders crawling on the web not yet listed here.



Crawler User agent Owner
192.com 192.comAgent 192.com
4anything 4anything.com LinkChecker v2.0 4anything
80legs Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/200
8032620
80legs
A-Online A-Online Search Aon
ABCdatos ABCdatos BotLink/5.xx.xxx#BBL ABCdatos
AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) AOL
ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) ASAHA
ASPseek ASPSeek/1.2.xa ASPseek
ASPSeek/1.2.5
ASPSeek/1.2.xxpre
ASPseek/1.2.9d
ASPSeek/1.2.x
ASPseek/1.2.xx
AVSearch AVSearch-1.0(peter.turney@nrc.ca) National Research Council Canada
AbachoBot AbachoBOT Abacho
AbachoBOT (Mozilla compatible)
Abby Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html) Ellerdale
Aberja Checkoma Aberja Checkoma Aberja
Abot abot/0.1 (abot; http://www.abot.com; abot@abot.com) Abot.com
About About/0.1libwww-perl/5.47 About
AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) AboutUs
Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) Accelovation
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com)
Accoona accoona Accoona
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Acoi AcoiRobot Acoi
Acoon Robot Acoon Robot v1.50.001 Acoon
Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) Acoon
Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o
rg)
Isara
Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) Activtourist
AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html) Google
Aesop AESOP_com_SpiderMan Aesop
Agada Mozilla/4.0 (agadine3.0) www.agada.de Agada
Mozilla/4.0 (agadine3.0)
agadine/1.x.x (+http://www.agada.de)
AgentName AgentName/0.1 libwww-perl/5.48 Linkomatic
Aibot AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C
hina)
21seek
Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Accoona
Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) Aipbot
Alacra PortalBSpider/2.0 (spider@portalb.com) Alacra
Aladin.de Aladin/3.324 Abacho
Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/) Aleksika
Alexa ia_archiver Alexa
AlkalineBOT AlkalineBOT/1.3 Vestris
AlkalineBOT/1.4 (1.4.0326.0 RTM)
Allesklar.de Allesklar/0.1 libwww-perl/5.46 Allesklar
Allrati Allrati/1.1 (+) Unknown
Almaden http://www.almaden.ibm.com/cs/crawler [hc5] IBM
http://www.almaden.ibm.com/cs/crawler
Altavista Scooter-3.2.JT Altavista
Scooter-3.2.EX
Scooter-3.2.DIL
Scooter-3.2.BT
Scooter-3.2.NIV
Scooter-3.2.SF0
Scooter/3.3_SF
Scooter/3.3.QA.pczukor
Scooter/3.3
Scooter_trk3-3.0.3
Scooter-3.2
Scooter-3.0QI
Scooter/1.1 (custom)
Scooter/1.0 scooter@pa.dec.com
Scooter/2.0 G.R.A.B V1.0
Scooter/1.0
Scooter/2.0 G.R.A.B. V1.1.0
Scooter/2.0 G.R.A.B. X2.0
Scooter-3.0.VNS
Scooter-3.0.HD
Scooter-3.0.FS
Scooter-3.0.EU
Scooter/3.3.vscooter
Scooter-3.2.snippet
Scooter-W3.1.2
Scooter-3.3dev
Scooter_bh0-3.0.3
Scooter-W3-1.0
Scooter2_Mercator_x-x.0
scooter-venus-3.0.vns
Scooter-ARS-1.1
Scooter-ARS-1.1-ih
Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) Amfibi
Amidalla libwww-perl/5.65 Amidalla
Annomille AnnoMille spider 0.1 alpha - http://www.annomille.it Annomille
AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) Ansearch
AnswerBus AnswerBus (http://www.answerbus.com/) AnswerBus
Answerchase PROve AnswerBot 4.0 Answerchase
Antibot antibot-V1.3.3.1/debian-linux-sarge Antidot
Any Search Info Mozilla/4.0 (Sleek Spider/1.2) Search-Info
Anzwers Australia AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Anzwers Australia
Apexoo Spider Apexoo Spider 1.0 Apexoo
ApocalXExplorerBot ApocalXExplorerBot ApocalX
Aport Aport Aport
Appie appie 1.1 (www.walhello.com) Walhello
appie 1.1 (www.walhello.com)
Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) Arabul
ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby
.com;)
Araby
Arachnoidea Arachnoidea (arachnoidea@euroseek.com) Euroseek
ArchitextSpider ArchitextSpider Excite
Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht
tp://www.archive-it.org)
Archive.org
Arexera X-Crawler Arexera
TECOMAC-Crawler/0.x
Arianna www.arianna.it Libero
Arikus_Spider Arikus_Spider Arikus
Asahina Asahina-Antenna/1.x Asahina
Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
Ask 24x Info ask.24x.info Ask 24x
Ask Jeeves/Teoma Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Ask Jeeves
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr
awling.html)
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)
Mozilla/2.0 (compatible; Ask Jeeves)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_cra
wling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Asked asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) Asked
Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) Askpeter
Asterias asterias/2.0 Singing Fish
Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi
on: 3.10
Singingfish
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis
ion: 3.11
Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) Seeq
Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) @Local
Attentio Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@att
entio.com)
Attentio
Augurnet Swiss augurfind Augurnet Swiss
augurnfind V-1.x
Axada axadine/ (Axadine Crawler; http://www.axada.de/; ) Axada
Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) Axandra
Axmo AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine
.
Axmo
Ay-Up FastBug http://www.ay-up.com Ay-up
BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) Rawgrunt
Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be) BA
BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us) Baboum
BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) Babaloo
Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html) Backlink-Check
Baiduspider Baiduspider+(+http://www.baidu.com/search/spider.htm) Baidu.com
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
BaiDuSpider
BaiduImagespider+(+http://www.baidu.jp/search/s308.html)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.ht
ml)
Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots
meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Balihoo
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a
spx; tgautier at balihoo dot com)
BanBot BanBots/1.2 (spider@banbots.com) Banbot
BeamMachine BeamMachine/0.5 (dead link remover of www.beammachine.net) BeamMachine
Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) uCHOOSE
BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) Apassion4jazz
BecomeBot Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.
com/site_owners.html)
Become
Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become
.com/site_owners.html)
BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom
e.co.jp/site_owners.html)
Become
BeijingCrawler BeijingCrawler Unknown
BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) BigClique
Biglotron BIGLOTRON (Beta 2;GNU/Linux) Biglotron
Bigsearch Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
Bigsearch
Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk) Bilinfo
BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html
; nutch-agent@lucene.apache.org)
Bilgi
BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Bilgi
BinGet BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/) Bin-Co
Bingbot Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm) Bing
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Bisnisseek Custom Spider www.bisnisseek.com /1.0 Bisnisseek
Bitacle Robot Bitacle Robot (V:1.0;) (http://www.bitacle.com) Bitacle
Bitacle bot/1.1
Bitlybot bitlybot Bit.ly
Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net) Blaiz Enterprises
Blaiz-Bee Blaiz-Bee/2.00.5622 ( http://www.blaiz.net) Blaiz
Blaiz-Bee/2.00.5502 (+http://www.blaiz.net)
Blitzsuche Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) RP ONLINE
BlitzBOT@tricus.net
BlitzBOT@tricus.net (Mozilla compatible)
BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers
)
BlogRefs
BlogScope Mozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto
)
U of Toronto
BlogSearch BlogSearch/1.0 +http://www.icerocket.com/ IceRocket
BlogSearch/1.x +http://www.icerocket.com/
BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) Okumura Group
Blogbot Naamah 1.0a/Blogbot (http://blogbot.de/) Blogbot
Naamah 1.0.1/Blogbot (http://blogbot.de/)
Blogdex BlogBot/1.x Massachusetts Institute of Technology
Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com) Blogdimension
Bloglines Bloglines/3.1 (http://www.bloglines.com) Bloglines
Bloglines Title Fetch/1.0 (http://www.bloglines.com)
Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com) Bloglines
BlogzIce BlogzIce/1.0 +http://www.icerocket.com/ IceRocket
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
BnF bot Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/) BnF
Boitho boitho.com-robot/1.x (http://www.boitho.com/bot.html) Boitho
boitho.com-robot/1.x
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-dc
Bot bot/1.0 (bot; http://; bot@bot.bot) Unknown
BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) Penn State College of Information Sciences and Technology
Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi
nd.mobi/bot.html abuse@mtld.mobi)
Find.mobi
Brandwatch magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Brandwatch
BravoBrian bSTOP BStop.BravoBrian.it Agent Detector BravoBrian
BravoBrian SpiderEngine MarcoPolo
BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) Brightcloud
Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) University of California
Btbot BTbot/0.x (+http://www.btbot.com/btbot.html) Btbot
BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler) BuildCMS
BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) BuiltWith
BullsEye/Intelliseek BullsEye Intelliseek
BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl
er@burstfind.com)
Burstfind
Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Buscaplus
Butterfly Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) G
ecko/2009032608 Firefox/3.0.8
Topsy
CCBot CCBot/1.0 (+http://www.commoncrawl.org/bot.html) CommonCrawl
CEA larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) CEA
CMP libwww-perl/5.41 CMP United Business Media
CUPS PrivacyFinder Cache Bot v1.0 PrivacyBird
Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html) Sensation Internet Services
CamontSpider CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html STFC
CanadianContent Search RoboCrawl (www.canadiancontent.net) CanadianContent
RoboCrawl (http://www.canadiancontent.net)
Carleson carleson/1.0 Cosmix
Catall Spider Catall Spider Catall
Catall-Spider Catall-Spider/3.3.3(www.Catall.de) Catall
CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood
lebot; cazoodlebot@cazoodle.com)
Cazoodle
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.co
m)
Ccubee ccubee/x.0 Empyreum
Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect
ion.com/bot.html )
Changedetection
Charlotte Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) Searchme
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Chen Li Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com) Chen Li
Christcentral Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com) Christcentral
ChristCRAWLER 2.0
CipinetBot CipinetBot (http://www.cipinet.com/bot.html) Cipinet
CipinetBot/1.0 (http://www.cipinet.com/bot.html)
Cityreview Robot Cityreview Robot (+http://www.cityreview.org/crawler/) Cityreview
CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) CjLog
Claymont Search Claymont.com Claymont Search
CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/) Seznam
Clushbot Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) Clush
Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)
Clushbot/2.x (+http://www.clush.com/bot.html)
Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/) Search.com
CoBITSProbe CoBITSProbe Academia Sinica
Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) Cobion
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
oBot ((compatible;Win32))
Combine Combine/2.0 Lund University
Combine/2.0 http://combine.it.lth.se/
Combine/3 http://combine.it.lth.se/
Cometrics-bot cometrics-bot, http://www.cometrics.de Cometrics
Cometsystems Crawler (cometsearch@cometsystems.com) Cometsystems
Crawler (cometsearch@cometsystems.com)
Comperio FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Comperio
Compete.com larbin_2.2.0 (crawl@compete.com) Compete Inc
Computerorgs htdig/3.1.6 (http://computerorgs.com) Computerorgs.com
Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen
e.apache.org)
Comrite
ConveraCrawler ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) Convera
ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
Converas RetrievalWare ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) Convera
CrawlConvera0.1 (CrawlConvera@yahoo.com)
ConveraCrawler/0.2
Convera Internet Spider V6.x
CoolBot CoolBot SuchMaschine21
Copyright sheriff copyright sheriff (+http://www.copyrightsheriff.com/) Copyright sheriff
Cortina Vision Research Lab image spider at vision.ece.ucsb.edu Vision Research Lab
CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) CougarSearch
Cowbot Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver
Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
CrawlerBoy CrawlerBoy Pinpoint.com Motricity
Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) Yamana Laboratory - Waseda University Japan
Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Crawlly
Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Croccrawler.com
CsCrawler Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo
glespam/crawler.html RPT-HTTPClient/0.3-3
University of Kassel
Csci_b659/0.13 csci_b659/0.13 Indiana University School of Informatics
Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Cuasar
CurryGuide CurryGuide SiteScan 1.1 CurryGuide
CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) CyberAlerts
Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) Cydral
CydralSpider CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com) Cydral
CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com)
DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net
/aboutkr.html)
Daum
DAUM Web Robot Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Daum
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu
nications Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi
cations Corp., Korea; +http://ws.daum.net/aboutkr.html)
DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Dnsdigger
DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
DataFountains DataFountains/DMOZ Downloader University of California
DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents
.shtml)
DataSpear Spider Bot DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html
; dssb@dataspear.com)
DataSpear
DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co
m)
DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/) dpSearch
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DaviesBot DaviesBot/1.7 (www.wholeweb.net) Wholeweb
Daypop daypopbot/0.x Daypop
DbDig dbDig(http://www.prairielandconsulting.com) Connections
De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) De.com
DeepIndexer DeepIndexer.ca Deepindex
Deepak-USC/ISI deepak-USC/ISI University of Southern California
Deepindex DeepIndex Deepindex
Deepindex V2
DeepIndex (www.en.deepindex.com)
Denmex Websearch Denmex websearch (http://search.denmex.com) Denmex Websearch
DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) DepSpid
Dev-spider2 dev-spider2.searchpsider.com/1.3b Searchspider
DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) DiaGem Japan
Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de Die Kraehe
Diggit Digger/1.0 JDK/1.3.0rc3 Diggit
Direct Hit Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) Teoma
Disco-crawl disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc
overyengine.com)
Discoveryengine
disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@
discoveryengine.com)
Ditto DittoSpyder Ditto
DoCoMo DoCoMo/1.0/Nxxxi/c10/TB NTT DoCoMo
DoCoMo/2.0 P900iV(c100;TB;W24H11)
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.go
o.ne.jp/door/crawler.html)
DoCoMo/1.0/Nxxxi/c10
Dodgebot dodgebot/experimental Agmlab
DotBot Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne
tdotcom.org)
Dotnetdotcom
DotBot/1.0.1
Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com) Douban
Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/) Download-Tipp
EyeCatcher (Download-tipp.de)/1.0
Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html) Drecom Japan
DtSearchSpider dtSearchSpider dtSearch
DuckDuckBot DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) DuckDuckGo
Dumbot Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) DumbFind.com
Dumbot(version 0.1 beta - dumbfind.com)
Dumbot(version 0.1 beta)
E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Yamana Laboratory
E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) e-StyleISP
EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at
global-opto dot com)
EApollo
EMPAS_ROBOT EMPAS_ROBOT Empas
ESISmartSpider ESISmartSpider smart-spider.com
Earthcom EARTHCOM.info/1.x [www.earthcom.info] Earthcom.info
EARTHCOM.info/1.xbeta [www.earthcom.info]
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)
EasyDL EasyDL/3.xx http://keywen.com/Encyclopedia/Bot Keywen
EasyDL/3.xx
EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Echo.com
Echo.fr EchO!/2.0 Echo.fr
Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) Charles University in Prague
Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm) Egoto.com
Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) uCHOOSE
Elsop LinkScan/x.x Unix LinkScan
LinkScan/11.0beta2 Unix
LinkScan/9.0g Unix
EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) Emeraldshield
Enfish Tracker Enfish Tracker Enfish
Enoola enoola (http://www.enoola.com) Enoola
Enterprise Search Enterprise_Search/1.0 Innerprise
Search/1.0 (http://www.innerprise.net/es-spider.asp)
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
Enterprise_Search/1.0.xxx
Entireweb WorldLight Entireweb
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Speedy Spider (Beta/x.x; speedy@entireweb.com)
Speedy_Spider (http://www.entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) Envolk
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)
EroCrawler EroCrawler EroCrawler
Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com) Eruvo
EuripBot EuripBot/0.4 (+http://www.eurip.com) GetFile Eurip.com
EuripBot/0.4 (+http://www.eurip.com) PreCheck
EuripBot/0.5 (+http://www.eurip.com) PreCheck
EuripBot/0.2 (+http://www.eurip.com) GetRobots
Euro-spider Euro-Spider Shopping 1.0 Euro-spider
Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) Evaal
EvaalSE EvaalSE - bot@evaal.com Evaal
Eventax eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Eventax
Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra
wlerhelp)
Vulcan
Evrinid-RT Mozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid) Evri
Exabot NG/2.0 Exalead
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Exalead NG/MimeLive Client (convert/http/0.120)
ExaBotTest/3.0
Exabot-Test/1.0
Exabot/3.0
Exabot/2.0
ExaBotTest/2.0
Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) Exalead
Exabot-Images/1.0
NG/4.0.1229
ExactSEEK exactseek.com ExactSEEK
ExactSeek Crawler/0.1
exactseek-crawler-2.63 (crawler@exactseek.com)
exactseek-pagereaper-2.63 (crawler@exactseek.com)
eseek-larbin_2.6.2 (crawler@exactseek.com)
ExactSeek_Spider ExactSeek_Spider ExactSeek
Excalibur Excalibur Internet Spider V6.5.4 Convera
Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) Execrawl
FAST-WebCrawler FAST Enterprise Crawler/6.4.18 (crawler@fast.no) FAST
FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw
ebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch
/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/
faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
fastlwspider/1.0
FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)
FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa
qfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support
/crawler.asp)
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net)
FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net)
FDSE Mozilla/4.0 (compatible; FDSE robot) Abadoor
FaXobot Faxobot/1.0 Faxo
Factbot Factbot 1.09 (see http://www.factbites.com/webmasters.php) Factbites
factbot : http://www.factbites.com/robots
FairShare Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairsh
are.cc)
FairShare
Fast Search PycURL FAST
Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de) Fastbot
FauBot Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/) Buzzvolume
Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu) Favo
Feed24 Feed24.com Feed24
FeedChecker FeedChecker/0.01 University of Tokyo
Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Google
Feedfinder feedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/ Feedfinder
Feedster Crawler Feedster Crawler/3.0; Feedster, Inc. Feedster
Felix Felix - Mixcat Crawler (+http://mixcat.com) MixCat
Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi
langy-agent@filangy.com)
Filangy
FindLinks findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) University of Leipzig
findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/)
http://wortschatz.uni-leipzig.de/findlinks/
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) Findexa
FineBot FineBot Finesearch
Firefly Firefly/1.0 Fireball
Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
FirstGov FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov U.S.Government
Firstsbot firstsbot Firstsfind
Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio
d com)
Flaptor
Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.
flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
Flatland Industries
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan
dindustries.com/flatlandbot.php; jason@flatlandindustries.com)
flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu
stries.com/flatlandbot.php; jason@flatlandindustries.com)
FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3 DivX.com
Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se
archhippo.com)
Searchhippo
FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) Fnoole
Folkd.com Spider Folkd.com Spider/0.1 beta 1 (www.folkd.com) Folkd
ForAll.pl-Crawler ForAll.pl-Crawler/1.0 ForAll
Francis Francis/1.0 (francis@neomo.de http://www.neomo.de/) Neomo
FreshNotes crawler FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com FreshNotes
FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com
Freshmeat freshmeat.net URL validator/1.1 Freshmeat
FuchsBot FuchsBot +http://www.fuchsbot.tld FuchsBot
FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn
.furlbot@looksmart.net)
Furl
FuseBulb FuseBulb.Com FuseBulb
FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) FyberSearch
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
GAIS Robot GAIS Robot/1.0B2 Seed
GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co
m)
Unknown
GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin
e.php)
GPU
GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) GSiteCrawler
Gaaz gazz/x.x (gazz@nttrd.com) Infobee
Gaisbot Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gais
Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html) Galaxy
Mozilla/4.0 (compatible; www.galaxy.com)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) Uchoose
GammaSpider GammaSpider/1.0 Gammasite
GenieKnows Mozilla/5.0 wgao@genieknows.com GenieKnows
larbin_2.6.3 (wgao@genieknows.com)
geniebot wgao@genieknows.com
Mozilla/5.0 (wgao@genieknows.com)
GeonaBot GeonaBot 1.x; http://www.geona.com/ Geona
Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu) Georgia Institute of Technology
Geourl Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot) Geourl
GigaBaz Brainbot MicroBaz Gigabaz
gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
Gigabot Gigabot/2.0 Gigablast
Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gigabot/2.0att
Gigabot/2.0; http://www.gigablast.com/spider.html
Gigabot/2.0/gigablast.com/spider.html
Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf
a dot com; http://www.girafa.com)
Girafa
GlobalQueue Look.com Multi-mode
GnodSpider GNODSPIDER (www.gnod.net) Gnod
GoForIt GoForIt.com GoForIt
GOFORITBOT ( http://www.goforit.com/about/ )
Goblin Goblin/0.9.x (http://www.goguides.org/goblin-info.html) GoGuides
Goblin/0.9 (http://www.goguides.org/)
Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp T-info
Gonzo2 gonzo2[P] +http://www.suchen.de/faq.html T-info
gonzo2[P] mailto:crawleradmin.t-info@telekom.de
gonzo1[P] mailto:crawleradmin.t-info@telekom.de
Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Google Keyword Tool Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select
/KeywordToolExternal)
Google Keyword Tool
Google-Adsense Mediapartners-Google Google
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
Mediapartners-Google/2.1
Google-Image Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) Google
Googlebot-Image/1.0
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Google-Sitemaps Google-Sitemaps/1.0 Google
Google-WAP Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Google
Google WAP Proxy/1.0
GoogleBot Googlebot/2.0 beta (googlebot@googlebot.com) Google
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com)
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
no-user-agent-in-crawltrack-for-that-bot
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible;
Googlebot/2.1; http://www.google.com/bot.html)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/Test ( http://www.googlebot.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog
lebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
Googlebot/1.0
Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b
ot.html)
Google
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google
bot-Mobile/2.1; +http://www.google.com/bot.html)
KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) Greaterera
GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html) Grigor
Gromit Gromit/1.0 Australasian Legal Information Institute
Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru
b.org)
Grub
Gsa-crawler gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) IBM
gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw)
Gulliver Gulliver/1.2 Northernlight
Gulliver/1.3
GulperBot Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) University of New-York
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim
/cgi-bin/Link/GulperBot)
Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) Gungho
GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Guruji
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
Harvest-NG Harvest-NG/1.0.2 Harvest-NG
Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Hatena Antenna
HatenaScreenshot HatenaScreenshot/1.0 (checker) Hatena
HatenaScreenshot/1.0 (checker)
Hbtronix.spider hbtronix.spider.2 -- http://hbtronix.de/spider.php Hbtronix
HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Mirago
Helix Helix/1.x (+http://www.sitesearch.ca/helix/) SiteSearch
HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Mirago
HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) Mirago
Heritrix Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs
.washington.edu/research/networking/websys/)
University of Washington
mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org)
archive.org_bot
Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec
ts/crawling/)
L3S Research Center
Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) Chepi
Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) Hermits Search
Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider Hiigle
Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com Hitwise
Holmes holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl) Szukaj.onet
holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/x.x
HolmesBot HolmesBot (http://holmes.ge) Holmes
HomePageSearch HomePageSearch(hpsearch.uni-trier.de) HomePageSearch
Homerbot Homerbot: www.homerweb.com Homerweb
Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda
-search.com)
Honda-Search
Hoowwwer HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>
hiit.fi)
NGIR
HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:
crawler-info<at>hiit.fi)
Htdig htdig/3.1.x (root@localhost) ht:/dig
Htdig/3.1.6 htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) Acad?mie de Toulouse
I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search
.com)
I1search
ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c
ontact(at)ml(dot)nict(dot)go(dot)jp)
NICT
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a
t)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con
tact(at)ml(dot)nict(dot)go(dot)jp)
ICCrawler ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) ICCenter
ICRA_Label_spider ICRA_label_spider/x.0 Icra
IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Id-search
IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a
t iiit dot ac dot in)
Webkhoj
INGRID Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster.
ilse.nl/jsp/webmaster.jsp)
Ilse
IP2MapBot IP2MapBot/1.1 http://www.ip2map.com Ip2Map
IPiumBot IPiumBot laurion(dot)com Laurions
IRLbot IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) Texas A&M University
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IScraperBot IScraperBot/0.1 Unknown
IWAgent IWAgent/ 1.0 - www.brandprotect.com Brandprotect
Iaskspider2 iaskspider2 (iask@staff.sina.com.cn) Sina
Ichiro ichiro/1.0 (ichiro@nttr.co.jp) Goo
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/1.0 (ichiro@nttr.co.jp)
IconSurf IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) IconSurf
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
Icsbot icsbot-0.1 International Christian school of Seoul
Ideare ideare - SignSite/1.x Ideare
IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator
e.it)
IlTrovatore
Ilial/Nutch ilial/Nutch-0.9-dev University of California
ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo
r more information please visit http://www.ilial.com/crawler; http://www.ilial.c
om/crawler; crawl@ilial.com)
Ilse Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.
ilse.nl/?aanmeld_mode=webhints)
Ilse
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld
en.ilse.nl/?aanmeld_mode=webhints)
ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com) Bdbrandprotect
IncyWincy IncyWincy/2.1(loopimprovements.com/robot.html) LoopImprovements
NetResearchServer/x.x(loopimprovements.com/robot.html)
IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen
ts.com/robot.html)
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement
s.com/robot.html)
IncyWincy (Look) IncyWincy(http://www.look.com) Look
IndexTheWeb IndexTheWeb.com Crawler7 IndexTheWeb
Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia
Interactive
Indonesia Interactive
Indy Library Mozilla/3.0 (compatible; Indy Library) Unknown
InelaBot InelaBot/0.2 (+http://inelegant.org/bot) Inelegant
Inet Library Inet library Inet Library
InfoFly InfoFly/1.0 (http://www.versions-project.org/) Versions-project
InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Stanford University
InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht
ml; lhall@telegenetic.net)
Telegenetic
Infoseek InfoSeek Sidewinder/0.9 Go
Inria larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Inria
Insitor Search robot Insitor.com search and find world wide! Insitor
Insitornaut Insitornaut Insitor
Internet Ninja Internet Ninja x.0 Dream Train Internet
Internetseer InternetSeer.com Internetseer
Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) Iprospect
IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) Ipselon
Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot
com)
Iseek
Ishida Lab larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Kyoto University
It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt
rovatore.it/bot.html; bot@iltrovatore.it)
IlTrovatore
Jabot Jabot/7.x.x (http://odin.ingrid.org/) ODIN Directory
Jabot/6.x (http://odin.ingrid.org/)
Jambot Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com) Jambot
Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot
; crawler@jambot.com)
Jayde Crawler Jayde Crawler. http://www.jayde.com Jayde
Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) Sidedc
Jetbot Jetbot/1.0 JetEye
Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de)
( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge
n/search/robot.html )
Neofonie
Jomjaibot Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com) Jomrank
Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za; info@jonga.co.za) Jonga
Jyxobot Jyxobot/x Jyxo
Jyxobot/1
K2 Spider k2spider Verity
KAIST AITrc Crawler KAIST AITrc Crawler AITrc
KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de) KFSW
KIT_Fireball KIT_Fireball/2.0 Dino-online
KS Crawler KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net)
Kindsight
KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte
nt/crawler/index.html; crawleradmin@knowledgestorm.com)
Knowledgestorm
KakleBot KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bo
t.html; support@kakle.com)
akle
KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw
ler@kalooga.com)
kalooga
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com)
Kasparek Firefox_1.0.6 (kasparek@naparek.cz) Czech Technical University Prague
Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) Keegee
Kenjin Spider Kenjin Spider Kenjin
Kevin Kevin http://dznet.com/kevin/ Dznet.com
Kevin http://websitealert.net/kevin/
KicktooBot kicktooBotV1.1 kictooBot@kictoo.com Kicktoo
Kinja-imagebot kinja-imagebot (http://www.kinja.com/) Kinja
Kinjabot kinjabot (http://www.kinja.com) Kinja
KnowItAll KnowItAll(knowitall@cs.washington.edu) University of Washington
Knowledge.com Knowledge.com/0.x knowledge.com
Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info
.html; webcrawler@krugle.com)
Krugle
Kulokobot kuloko-bot/0.x Kuloko
kulokobot www.kuloko.com kuloko@backweave.com
Kulturarw kulturarw3/0.1 National Library of Sweden
Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez) Sanoma
Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k
yluka.com)
Kyluka
LECodeChecker LECodeChecker/3.0 libgetdoc/1.0 Linkexchange
LNSpiderguy LNSpiderguy Lexis-Nexis
LapozzBot LapozzBot/1.4 (+http://robot.lapozz.hu) Lapozz
LapozzBot/1.5 (+http://robot.lapozz.hu)
LapozzBot/1.4 ( http://robot.lapozz.com)
Larbin_2.6.3 larbin_2.6.3 marzia.polito@intel.com Unknown
larbin_2.6.3 larbin2.6.3@unspecified.mail
Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo
.com; webmaster@lawinfo.com)
Lawinfo
Lemur Consulting larbin_2.6.2 (tom@lemurconsulting.com) Lemur Consulting
Lexibot Mata Hari/2.00 BrightPlanet
LexiBot/1.00
Liafa larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Liafa
LibWeb libWeb/clsHTTP -- hiongun@kt.co.kr Korea Telecom
LibertyW LibertyW (+http://www.lw01.com) LibertyW
LibertyW (+http://www.libertyw.eu)
LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co
m)
Lijit
Linguee Bot Linguee Bot (http://www.linguee.com/bot; bot@linguee.com) Linguee
LinkWalker LinkWalker Seven TwentyFour
Linknzbot linknzbot LinkNZ
Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Links2Go
Links4US-Crawler Links4US-Crawler, (+http://links4us.com/) Links4US
LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck
er.html)
Unknown
Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) Universitat de les Illes Balears
Lmspider lmspider (lmspider@scansoft.com) Nuance
LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/) LocalBot
Lockstep Spider Lockstep Spider/1.0 Lockstep
Look.com NetResearchServer(http://www.look.com)
LookdirBot LookdirBot Lookdir
Lovel Lovel as 1.0 ( +http://www.everatom.com) Everatom
Ltaa_web_crawler larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Ecole Polytechnique F?d?rale de Lausanne
Luchs.at URL checker luchs.at URL checker Luchs
Lycos_Spider Lycos_Spider_(T-Rex) Lycos
Lycos_Spider_(modspider)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide
r_(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_
Spider_Beta2(T-Rex) )
MJ12bot MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+) Majestic 12
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.ui
uc.edu)
University of Illinois
MSN Bot msnbot/0.9 (+http://search.msn.com/msnbot.htm) MSN
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSN
MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot) Microsoft
MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/ Microsoft
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Sagool
Mail.Ru Mail.Ru/1.0 Mail.Ru
Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT) Mainseek
Mammoth mammoth/1.0 (+http://www.sli-systems.com/) SLI Systems
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
MantraAgent MantraAgent LookSmart
Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Kolibri
Martini Martini LookSmart
MARTINI
Marvin Marvin v0.3 Health On the Net Fondation
Masterseek MasterSeek Masterseek
Maxbot Spider/maxbot.com admin@maxbot.com Maxbot
Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html;
maxomobot@maxomo.com)
Maxomo
MediaCrawler MediaCrawler-1.0 (Experimental) Media Find
MediaSearch MediaSearch/0.1 WWW.FI
Mediater Rechercher libwww/5.3.2 Mediater
MegaSheep MegaSheep v1.0 (www.searchuk.com internet sheep) Search UK
Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) Megaglobe
Melbot WebSpider Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.) Melbot
Mercator Mercator-Scrub-1.1 Altavista
Mercator-1.x
Mercator-2.0
Merl.com larbin_2.1.1 larbin2.1.1@somewhere.com Mitsubishi Electrical Research Lab
MetaGer_PreChecker MetaGer_PreChecker0.1 MetaGer
Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) Metacarta
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)
Metadata Labs MLBot (www.metadatalabs.com/mlbot) Metadata Labs
Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta
euro.com; crawler at metaeuro dot com)
Metaeuro
Metager-Linkchecker MetaGer-LinkChecker Metager
MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Metager
Metaquerier MQbot metaquerier.cs.uiuc.edu/crawler University of Illinois
MQbot http://metaquerier.cs.uiuc.edu/crawler
Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner
.de/)
Metaspinner
Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/) Metatagsdir
Microsoft Small Business Indexer Microsoft Small Business Indexer Microsoft
Microsoft URL Control Microsoft URL Control - 6.00.8862 Unknown
Microsoft URL Control - 6.01.9782
Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/)
miggibot/2006.01.24
Micro-sys
Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s
ee getmirar.com)
Mirar
Missigua Locator 1.9 Missigua Locator 1.9 Unknown
Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot
.fr)
Misterbot
Miva Miva (AlgoFeedback@miva.com) Miva
MnoGoSearch MnogoSearch/3.2.xx MnoGoSearch
Mo College 1.9 Mo College 1.9 Unknown
Moget moget/x.x (moget@goo.ne.jp) Goo
Mogimogi mogimogi/1.0 Goo
Moi-même no-user-agent-in-crawltrack-for-that-bot Moi
MojeekBot MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) Mojeek
Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)
Monogol http://www.monogol.de Monogol
Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com) Moreover
Morris Morris - Mixcat Crawler (+http://mixcat.com) MixCat
Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.
com/bot)
Mowser
MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) MozDex
Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com) MozDex
MultiText MultiText/0.1 Virginia Polytechnic Institute and State University
Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) SWSE
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) Euro Ferret
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)
Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)
MusicWalker MusicWalker2.0 (+http://www.somusical.com) SoMusical!
Mxbot Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html) Chainn
My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info) My-byte
MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx
; SearchBot@MyFamilyInc.com)
Ancestry
Mylinea Mylinea.com Crawler 2.0 Mylinea
NASA Search 1.0 NASA Search 1.0 Unknown
NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) National Center for Supercomputing Applications
NG-SearchBot NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) NG-Search
NG-Search/0.9.8 (http://www.ng-search.com)
NII larbin_2.6.2 (hamasaki@grad.nii.ac.jp) National Institut of Informatics (Japan)
NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.
89 (like Gecko)
Northern Light
NMG Spider NMG Spider/0.3 (szukanko.com) Szukanko
NTT Directory nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) NTT Directory
NWSpider NWSpider 0.9 NWSpider
Nabot NABOT/5.0 Naver
nabot_1.0
Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x) Najdi
Nameprotect NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) Nameprotect
NPBot-1/2.0
NPBot (http://www.nameprotect.com/botinfo.html)
Nationaldirectory NationalDirectory-WebSpider/1.3 Nationaldirectory
NationalDirectoryAddURL/1.0
NatzanBot NatzanBot, http://www.natzan.com Natzan
NaverBot dloader(NaverRobot)/1.0 Naver
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.
jsp)
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
NaverBot_dloader/1.5
NavissoBot NavissoBot Navisso
NavissoBot/1.7 (+http://navisso.com/)
Nebulla Nebullabot/2.2 (http://bot.nebulla.info) Nebulla
Nebullabot/2.2 (http://bot.nebulla.de)
Nebulla/V1.0 Spider (http://spider.nebulla.de)
Nelian Pty Ltd - Spider Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com ) Pcaccessoriesparts
NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience:
http://www.netnose.com)
NetNose
NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html) Loopimprovements
NetSprint -- 2.0 NetSprint -- 2.0 Wirtualna Polska
NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com
; support@netwhat.com)
Netwhat
NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) NetinfoBot
Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) Netluchs
Netprospector Netprospector JavaCrawler Netprospector
Netscape Robozilla/1.0 DMOZ
NextGenSearchBot NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB
ot)
Zoominfo
NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe
nSearchBot.aspx)
NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot
)
NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1 Nextopia
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x
NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Healthline
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob
lems contact: crawler@healthline.com
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe
ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
NjuiceBot Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot Unknown
Noago Spider Noago Spider Noago
NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm) Nokodo
Norbert Norbert the Spider(Burf.com) Burf
Noxtrumbot noxtrumbot/1.0 (crawler@noxtrum.com) Noxtrum
Noyona noyona_0_1 Noyona
Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) Nsyght
NuSearch Spider nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible
; MSIE 4.01)
NuSearch
Nutch Nutch Lucene
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis
ts.sourceforge.net)
NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy
crawl@cs.washington.edu)
University of Washington
NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o
rg/nutch/bot.html; ec2test at lucene.com)
Amazon
OOZBOT OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) Setooz
Obidos-bot obidos-bot (just looking for books.) Weblog bookwatch
Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3
E
Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup
port@thesoftwareobjects.com)
SAIC
Ocelli Ocelli/1.x (http://www.globalspec.com/Ocelli) GlobalSpec
Ocelli/1.3 (http://www.globalspec.com/Ocelli)
Octora Octora Beta - www.octora.com Octora
Octora Beta Bot Octora Beta Bot - www.octora.com Octora
OmniExplorer OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler OmniExplorer
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer
OnetSzukaj Onet.pl SA, http://szukaj.onet.pl Szukaj
Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)
OpenISearch OpenISearch/1.x (www.openisearch.com) OpenISearch
OpenPortal4U DigOut4U Arisem
OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) OpenTagger
OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2 OpenText
OpenWebSpider OpenWebSpider/x OpenWebSpider
Openfind Openfind Robot/1.1A2 Openfind
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht
ml)
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.
openfind.com.tw/robot.html)
OpidooBot OpidooBOT (larbin2.6.3@unspecified.mail) Opidoo
Oracle Oracle Ultra Search Oracle
OrangeBot-Mobile Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19
.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
Orange
OrangeSpider OrangeSpider Orangeslicer
Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor
t/crawler.asp)
FAST
OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) OzMonitor
Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat
.de)
Flying-cat
PJspider PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Nextopia
PWeBot Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) Programacionweb
PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)
Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Page-store
PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/) PageBites
Page_verifier page_verifier http://www.securecomputing.com/goto/pv Securecomputing
Pagebull Pagebull http://www.pagebull.com/ Pagebull
Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) Pages Jaunes
FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com)
PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi
on 0.001
Links24h
Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/
crawl.html)
National Library of Australia
ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/) Ianett
PatchOneSpider Mozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se) PatchOne
Patwebbot Patwebbot (http://www.herz-power.de/technik.html) Patsearch
Peerbot PEERbot www.peerbot.com Peerbot
Petitsage www.petitsage.fr site detector 0.4 Petitsage
PicoSearch PicoSearch/1.0 Pico Search
PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org) Malfunction
Piffany Piffany_Web_Scraper_v0.x Piffany
Piffany_Web_Spider_v0.x
Pilot Hitlist Marketwave Hit List Pilot Hitlist
Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com) Pingdom
PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Pipeline
Pita Pita Stanford University
PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 dhkang@plantynet.com PlantyNet
PlantyNet_WebRobot_V1.9 yangsam@plantynet.com
PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1
subscribers)
Pluck
Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin
fo.htm)
Pmoz
Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) Podtech
Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x) Pogodak
Polybot polybot 1.0 (http://cis.poly.edu/polybot/) Polytechnic University Brooklyn
Pompos Pompos/1.2 http://dir.com/pompos.html Iliad (Free)
Pompos
Pompos/1.x http://dir.com/pompos.html
Pompos/1.x pompos@iliad.fr
Pompos/1.1 http://dir.com/pompos.html
Pompos/1.3 http://dir.com/pompos.html
PopJapanSearch Robot/www.pj-search.com PopJapanSearch
Popdexter Popdexter/1.0 Popdex
PrivacyFinder/1.1 PrivacyFinder/1.1 AT&T Privacy Bird Privacy Preferences
ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com) ProWebguide
Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi
a.com)
Probert Encyclopaedia
Psbot psbot/0.1 (+http://www.picsearch.com/bot.html) Picsearch
Psycheclone psycheclone Unknown
Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca Pubblisito
Pumpkin blogsearchbot-pumpkin-3 Artofcomputing
Purebot Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/) Puritysearch
Python-urllib Python-urllib/1.16 Unknown
QPCreep QPCreep Test Rig ( We are not indexing, just testing ) Quepasa
QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n
et)
Qihoo
Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) Quantcast
Qube qube/2.0 (+http://qube.qelix.com/v2) Qube
QuepasaCreep QuepasaCreep v0.9.1x Quepasa
QuepasaCreep ( crawler@quepasacorp.com )
QueryN Metasearch QueryN Metasearch QueryN
Qweerybot QweeryBot/3.01 ( http://qweerybot.qweery.nl) Qweery
QweeryBot/3.02 ( http://qweerybot.qweery.nl)
Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)
R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6
R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6
R6_FeedFetcher(www.radian6.com/crawler)
RAMPyBot RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveram
p.com/bot.html; support@giveRAMP.com)
Giveramp
RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu
cene.apache.org)
RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com) RSS One
Rambot ramBot xtreme x.x Intersearch
RankMeter Web Snooper SearchUtilities
Rational SiteCheck Rational SiteCheck (Windows NT) GSInnova
Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper) Marty Anstey
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) Rediff
RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Pronto
RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) Telegenetic
RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) RedKernel
RoboPal RoboPal (http://www.findpal.com/) FindPal
Rotondo Rotondo/3.1 libwww/5.3.1 QualiGo
RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) Webaroo
RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html) Aktienbetreuer
S2Bot S2Bot/1.0 (http://search2.net) Search2
SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Samsung
SBIder SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell
.com/contact-support.html)
SiteSell
SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site
sell.com/contact-support.html)
SandCrawler SandCrawler - Compatibility Testing Microsoft
Savvybot savvybot/0.2 WebSavvy
ScanWeb ScanWeb Eserver
Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/
/www.schibstedsok.no/bot/)
Schibsted
ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho
laruniverse.com)
ScholarUniverse
Scirus-crawler FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com
/srsapp/contactus/
Scirus
ScollSpider Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) Webwobot
ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)
ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots Fooky
ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ Ant
ScoutJet Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) ScoutJet
ScoutJet
Scrubby Scrubby/2.x (http://www.scrubtheweb.com/) Scrub the web
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache
.org/nutch/bot.html; researchbot@shopping.com)
Shopping
Search-Engine-Studio Search-Engine-Studio Xtreeme
Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Search-Info
Search.ch search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) Search CH
search.ch V1.4
Search17Bot Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php) Search17
Search4free lwp-trivial/1.34 Search4free
SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs
a.com)
Search4USA
SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
SearchEngineWorlds
SearchExpress Spider SearchExpress Spider0.99 SearchExpress
SearchMonkey Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/sea
rchmonkey/useragent)
Yahoo
SearchScout ClariaBot/1.0 SearchScout
DiamondBot
Diamond/1.0
SearchSight SearchSight/2.0 (http://SearchSight.com/) SearchSight
SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi
der.com)
SearchSpider
SearchSpider.com/1.1
SearchdayBot SearchdayBot Searchday
Searchdnabot searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at sear
chenginedna dot com)
Searchenginedna
Searchguild SearchGuild_DMOZ_Experiment (chris@searchguild.com) SearchGuild
Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Searchit-Now
Searchmee! Spider Searchmee! Spider v0.98a Searchmee!
Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 SeekPort
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2
Seeker.lookseek Seeker.lookseek.com Lookseek
Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/) Semager
Semager/1.0 (http://www.semager.de)
Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) Sensis
SeznamBot SeznamBot/1.0 (+http://fulltext.seznam.cz/) Seznam
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/1.0
SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) SharewarePlaza
SheenBot SheenBot/SheenBot-1.0.4 (Sheen web crawler) Unknown
Sherlock sherlock/1.0 Indiana University School of Informatics
Shim-Crawler Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra
wl@logos.ic.i.u-tokyo.ac.jp)
University of Tokyo
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c
rawl@logos.ic.i.u-tokyo.ac.jp)
Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl
er/; crawler@logos.ic.i.u-tokyo.ac.jp)
ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/) Littlewiki
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Shoula Shoula.com Crawler 2.0 Shoula! Search
SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz) Siets
Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com) Siigle
Silk/1.0 silk/1.0 Slider
silk/1.0 (+http://www.slider.com/silk.htm)/3.7
Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Sirketce
Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR American College of Radiology
SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) SiteBar
SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru) SiteBase
SiteSpider SiteSpider +(http://www.SiteSpider.com/) SiteSpider
SiteTruth SiteTruth.com site rating system SiteTruth
SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) SitiDi.net
Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] Skaffe
Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE
.com)
Skizzle
SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) SkreemR
Slider Slider_Search_v1-de Slider
Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)
Yahoo
Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Yahoo
Slurp Inktomi Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Hotbot-Lycos,NBCi etc.
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp Inktomi (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch
/slurp)
Yahoo
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu
rp)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE
cho/2.0.0.4
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071
3 Firefox/2.0.0.5
Slurpy Verifier Slurpy Verifier/1.0 Yahoo
SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0 Smiffysplace
SnapPreview Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 Unknown
Snapbot Snapbot/1.0 Hotlinks
Snoopy Snoopy v1.2 Unknown
SnykeBot SnykeBot/0.6 (http://www.snyke.com) Snyke
SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) SoftHypermarket
Sogou web spider Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou
Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sohu-search sohu-search Sohu
Somewhere Mozilla (Mozilla@somewhere.com) Somewhere
Sosospider Sosospider+(+http://help.soso.com/webspider.htm) Soso
Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Unknown
Spbot Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ ) Seoprofiler
Speedfind speedfind ramBot xtreme 8.1 Lotse
Speedy Spider Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Entireweb
Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Sphere Scout Sphere Scout&v4.0 - scout at sphere dot com Sphere
Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg
)
Singapore Press Holdings
SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) SpiderMonkey
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spinn3r Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.
1); http://spinn3r.com/robot) Gecko/20021130
Spinn3r
SplatSearch libwww-perl/5.45 Splat
Spock Crawler Spock Crawler (http://www.spock.com/crawler) Spock
Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr
oose.com)
Sproose
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c
om)
StackRambler StackRambler/x.x Rambler
StackRambler/2.0
StackRambler/2.0 (MSIE incompatible)
Steeler Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) University of Tokyo
Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com) Strategic Board
Suchbaer suchbaer.de Suchbaer
suchbaer.de (CrawlerAgent v0.103)
Suchpad suchpad/1.0 (+http://www.suchpad.de) Suchpad
SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Summize
SuperSnooper Robot@SuperSnooper.Com SuperSnooper
SurveyBot SurveyBot/2.3 (Whois Source) Domaintools
Susie !Susie (http://www.sync2it.com/susie) Sync2it
Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) University of Maryland
SycaBoT SycaBoT/1.0 SycaroX
SycaBoT-Audio SycaBoT-Audio SycaroX
SycaBoT-Image SycaBoT-Image SycaroX
SycaBoT-Programme SycaBoT-Programme SycaroX
SycaBoT-Video SycaBoT-Video SycaroX
Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr
awl@cs.washington.edu)
University of Washington
Sygol <http://www.sygol.com/> http://www.sygol.com Sygol
SygolBot SygolBot http://www.sygol.com Sygol
SygolBot http://www.sygol.net
SynooBot Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) Synoo
SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.
html)
Synthesio Crawler Synthesio Crawler release MonaLisa (contact at synthesio dot fr) Synthesio
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Syntryx
Szukacz Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) Szukacz
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl)
Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)
TAGword Tagword (http://tagword.com/dmoz_survey.php) TAGword DMOZ survey
TCDBOT TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot
IE)"
Trinity College Dublin
Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/) Tags2dir
TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta
ilrank.com/robot) Gecko/20021130
TailRank
Talkro Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w
ebshot)
DaumSoft
TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/
TargetSeek.html)
Targetgroups
Tasapspider Tasapspider/0.9 (http://www.tasap.com) Tasap
Technoratibot Technoratibot/0.7 Technorati
Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm) Tecomi
Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w
ww.netseer.com/crawler.html; crawler@netseer.com)
Netseer
TelenetDigger TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be) Telenet
Teoma Teoma MP Teoma
teoma_agent1
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
teomaagent crawler-admin@teoma.com
teomaagent1 [crawler-admin@teoma.com]
Teradex Mapper Teradex Mapper; mapper@teradex.com; http://www.teradex.com Teradex
TeragramCrawler TeragramCrawler Teragram
TeragramWebcrawler TeragramWebcrawler/1.0 Teragram
TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Terrawiz
TheSuBot TheSuBot/0.1 (www.thesubot.de) TheSuBot
Theme Spider Theme Spider (+http://www.themespider.com/spider.html) Themespider
Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th
eophrastus.php)
N.A.Smith
Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) Thumbshots
Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Thunderstones
TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html Breakingblogs
TinEye TinEye/1.1 (http://tineye.com/crawler.html) TinEye
TivraSpider tivraSpider/1.0 (crawler@tivra.com) AT&T
Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html) Tkensaku
ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 La Toile du Qu?bec
Topodia Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia
.com/; support@topodia.com)
Topodia
Toutatis Toutatis x-xx.x (hoppa.com) Hoppa
Toutatis x.x (hoppa.com)
Toutatis x.x-x
Traazibot traazibot/testengine (+http://www.traazi.de) Traazi
Trampelpfad Trampelpfad-Spider-v0.1 Trampelpfad
Trampelpfad-Spider
TrendTech http://www.trendtech.dk/spider.asp) TrendTech
Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) Truveo
Turnitin TurnitinBot Turnitin
TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) Turnitin
Turnpike Emporium Turnpike Emporium LinkChecker/0.1 Turnpike Emporium Directory
TutorGigBot Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) TutorGig
TutorGig/1.5 (+http://www.tutorgig.com/crawler)
TweetmemeBot Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0
.6 TweetmemeBot
TweetMeme
Mozilla/5.0 (compatible; TweetmemeBot/1.0; +http://www.tweetmeme.com/)
TwengaBot TwengaBot/2.0 (http://www.twenga.com/bot.html) Twenga
TwengaBot/1.1 (+http://www.twenga.com/bot.html)
Twiceler Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Cuill
Twiceler www.cuil.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
Twiceler-0.9 http://www.cuil.com/twiceler/robot.html
Twiceler www.cuill.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
TygoBot TygoProwler Tygo
TygoBot
Tzilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez
illa.de/t_st-odp-entries-agent.html)
Tzilla
UKWizz Mackster( http://www.ukwizz.com ) UKWizz
UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
UN OCHA FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA) UN OCHA
UOLCrawler UOLCrawler (soscrawler@uol.com.br) UOL
URL-Spider Mozilla/5.0 URL-Spider URL-Spider
URL_Spider_Pro URL Spider Pro/x.xx (innerprise.net) Innerprise
URL_Spider_Pro/x.x
URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp)
USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) University of Sydney
Ultraseek Ultraseek Go
UnChaosBot <b> UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc
haos.com </b> (info@unchaos.com)
UnCHAOS
<a href="http://www.unchaos.com/"> UnChaos Bot, Hybrid Web Search Engine. </a> (
vadim_gonchar@unchaos.com)
<a href="http://www.unchaos.com/"> UnChaos </a>, From Chaos To Order, Hybrid Web
Search Engine. (vadim_gonchar@unchaos.com)
Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt University of Koblenz
Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html University of Dortmund
Updated updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) Updated
Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
Updated
Uptimebot UptimeBot(www.uptimebot.com) Uptimebot
Uptimebot
Urlchecker1.0 Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) Euro Directory
Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html Urlfan
VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com) VBSEO
VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; info@vipsolutions.hu) Vipsolutions
VMBot VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) VerticalMatch
VMBot/0.9 (VMBot; http://www.verticalmatch.com; vmbot@tradedot.com)
VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu
c.edu
University of Illinois
Vacobot Vacobot; (+http://vaco.ws/bot.html) Vaco
Vagabondo Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http
://webagent.wise-guys.nl/)
WiseGuys
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt
p://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://
webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w
ebagent.wise-guys.nl/)
Vagabondo/3.0 (webagent at wise-guys dot nl)
Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/
)/1.0 Profile
WiseGuys
Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) Vakes
Verizon FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad
min@superpages.com)
Verizon Superpages
fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt
ers@fastsearch.com)
Vermut mozilla/5.0 (compatible; vermut +http://www.aol.com) AOL
Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com)
Mozilla/5.0 (compatible; vermut +http://vermut.aol.com)
Versus versus 0.2 (+http://versus.integis.ch) Hochschule f?r Technik Rapperswil
Versus Crawler versus crawler eda.baykan@epfl.ch Ecole Polytechnique F?d?rale de Lausanne
VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs VeryGoodSearch
Verzamelgids Verzamelgids/2.2 (http://www.verzamelgids.nl) Verzamelgids
verzamelgids.nl - Networking4all Bot/x.x
Vespa Crawler (Yahoo) Vespa Crawler Yahoo! Norway
Visbot VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Visvo
Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com)
Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) Freshnotes
VoilaBot KE_1.0/2.0 libwww/5.2.8 Voila.fr
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi
la.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor
t.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/
/www.voila.com/)
Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) Vonna
Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) Marty Anstey
Voyager voyager-hc/1.0 Kosmix
voyager/1.0
cfetch/1.0
Vspider vspider/3.x Verity
vspider
W3C-Validator W3C_Validator/1.432.2.10 W3C
W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803 W3C
W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de W3SiteSearch
W8net Mozilla/5.0 usww.com-Spider-for-w8.net usww
WEP Search WEP Search 00 Unknown
WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) CWR
WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)
WISEbot WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Koreawisenut
WWW.fi crawler www.fi crawler, contact crawler@www.fi WWW.FI
WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de) World Wide Weasel
Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/) Wadaino
WasaBot Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)
Wasalive
WasaBot 0.11 (+http://www.wasalive.com/)
WasaBot/2.4 (+ http://www.wasalive.com ) Java/1.6.0
Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) Wavefire
Waypath Waypath Scout v2.x - info at waypath dot com Waypath
Waypath development crawler - info at waypath dot com
WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer
elate.org)
WeRelate
WeatherBot WeatherBot v1.4 http://www.ezweather.net EZ Weather
WebAlta Crawler WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta
WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT
5.1; ru-RU)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot
/index.html
Carnegie Mellon University
WebCorp WebCorp/1.0 WebCorp
WebCorp/1.1
WebFindBot WebFindBot(http://www.web-find.com) web-find
WebGobbler webGobbler/1.x.x WebGobbler
WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) WebRankSpider
WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S
earch@WebSearch.COM.AU)
WebSearch
WebSearchBench WSB, http://websearchbench.cs.uni-dortmund.de University of Dortmund
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsitt Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de
WebSearchBench WebCrawler v0.1(Experimental)
WebStat WebStat/1.0 (Unix; beta; 20040314) University of South Carolina
WebVac WebVac (webmaster@pita.stanford.edu) Stanford University
WebarooBot WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) Webaroo
WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)
Webbandit webbandit/4.xx.0 SoftwareSolutions
Webbot webbot(+http://webbot.com/bot.htm) Webbot
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Webclipping Webclipping.com Webclipping
Webcrawl webcrawl.net Webcrawl
webcrawl.net
Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) Webdunia
Webglimpse Webglimpse 2.xx.x (http://webglimpse.net) Webglimpse
Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0 Los Alamos National Laboratory
Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de University of Leipzig
Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ University of Dortmund
WebsiteWorth WebsiteWorth v1.0 Sootle
Webspinne Webspinne/1.0 webmaster@webspinne.de Webspinne
Websquash Websquash.com (Add url robot) Websquash
Webster Webster v0.3 ( http://webster.healeys.net/ ) Webster.healeys
Webverzeichnis Webverzeichnis.de - Telefon: 01908 / 26005 Webverzeichnis
Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/] Axxus
Wikiwix-bot wikiwix-bot-3.0 Wikiwix
Willow Willow Internet Crawler by Twotrees V2.1 Twotrees
WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ?
WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB
ot; winkbot@wink.com)
Wink
WiseGuys Vagabondo/2.0 MT WiseGuys
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi
se-guys.nl/)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/1.x MT (webagent@wise-guys.nl)
Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.
nl/)
Worio Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) Worio
Mozilla/5.0 (compatible; woriobot support [at] worio [dot] com +http://worio.com
))
Worio bot Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com) Worio
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Worio heritrix bot worio heritrix bot (+http://worio.com/) Worio
WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) WorldWideWeb
Wotbox Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 Wotbox
Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)
Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Lehigh University
Wwlib/Linux Wwlib/Linux Wolverhampton Web Library
Wwwster wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) Centrum f?r Informations- und Sprachverarbeitung
XP5 Project XP5 [2.03.07-111203] Marty Anstey
Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html) Xdefine
Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) Xirq
Xyleme SA France cosmos/0.9_(robot@xyleme.com) Xyleme
cosmos/0.8_(robot@xyleme.com)
Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Yahoo Japan
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yacy yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/
en)
Yacy
Yacybot yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net Anomic Consulting
yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/
bot.html
yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net
yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_15; Europe/fr) http://yacy
.net/bot.html
Yahoo Japan robot Y!J-SRD/1.0 Yahoo Japan
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo Japan
Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) Yahoo
YahooYSMcm/2.0.0
Yahoo! Mindset Yahoo! Mindset Yahoo! Mindset
Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/
us/ysearch/crawling/crawling-02.html )
Yahoo
Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co
m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo
Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Yahoo
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Yahoo-Test/4.0 Yahoo-Test/4.0 Yahoo
Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com;
http://www.alltheweb.com/
Yahoo
YahooFeedSeeker YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu
blishers.html)
Yahoo
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c
om/rssguide)
YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web
master@yahoo-inc.com ) (KHTML, like Gecko)
Yahoo
YahooSeeker/M1A1-R2D2 SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch
/crawling/crawling-01.html)
Yahoo
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot
com)
Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo
-inc dot com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i
nc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo.
com/help/us/ysearch/crawling/crawling-01.html)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile
-search-customer-care AT yahoo-inc dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust
omer-care AT yahoo-inc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-
01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/
crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo
.com/help/us/ysearch/crawling/crawling-01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in
c dot com)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli
ng/crawling-01.html)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http:
//help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Yandex Yandex/1.01.001 (compatible; Win16; I) Yandex
Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Yandex
Yarienavoir yarienavoir.net/0.2 Yarienavoir
Yellopet spider.yellopet.com - www.yellopet.com Yellopet
YesupBot Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) Yesup
Yeti Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) Naver
Yeti
Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)
Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m
inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres
den dot de)
GoPubMed
YodaoBot Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide
r/; )
Yodao
Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0
/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/
; )
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu) YooW!
Yoogli yoogliFetchAgent/0.1 Yoogli
Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) Yoono
Yoono web-crawler yoono/1.0 web-crawler/1.0 Yoono
YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine YottaCars
YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine YottaShopping
Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) Z-add
Zao Crawler Zao/0.1 (http://www.kototoi.org/zao/) Kototoi
Zao-Crawler 0.2b
Zao-Crawler
ZeBot ZeBot_lseek.net (bot@ze.bz) Ze.bz
ZeBot_www.ze.bz (ze.bz@hotmail.com)
Zealbot Mozilla/4.0 (compatible; Zealbot 1.0) LookSmart
Zearchit Zearchit Zearchit
Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/) Zedzo
Zerxbot zerxbot/Version 0.6 libwww-perl/5.79 Zerx
Zetbot DeepIndex ( http://www.zetbot.com ) Zetbot
Zeus Zeus ThemeSite Viewer Webster Pro V2.9 Win32 Cyber-Robotics
ZBot/1.00 (icaulfield@zeus.com)
Zeus xxxxx Webster Pro V2.9 Win32
ZipppBot ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) Zipp
ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net)
Zippy Zippy v2.0 - Zippyfinder.com Zippyfinder
ZoomSpider ZoomSpider - wrensoft.com WrenSoft
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR
1.1.4322)
Zspider zspider/0.9-dev http://feedback.redkolibri.com/ Redkolibri
ZyBorg (LookSmart) ZyBorg LookSmart
ZyBorg (Wisenut) ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) Wisenut
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnu
tbot.com)
aiHitBot Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) Aihit
eCairn-Grabber eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15 eCairn