CrawlTrack, the spiders and crawlers tracking script.
protection
CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.


diable

CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.








Crawlers and spiders who can visit your website

Here is the list of crawlers used by CrawlTrack; that list is updated regularly. As you will see, it's already a very long list but there is still a lot of spiders crawling on the web not yet listed here.



Crawler User agent Owner
192.com 192.comAgent
4anything 4anything.com LinkChecker v2.0
80legs Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/200
8032620
A-Online A-Online Search
ABCdatos ABCdatos BotLink/5.xx.xxx#BBL
AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)
ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
ASPseek ASPseek/1.2.xx
ASPSeek/1.2.x
ASPSeek/1.2.xxpre
ASPseek/1.2.9d
ASPSeek/1.2.5
ASPSeek/1.2.xa
AVSearch AVSearch-1.0([email protected])
AbachoBot AbachoBOT (Mozilla compatible)
AbachoBOT
Abby Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Aberja Checkoma Aberja Checkoma
Abot abot/0.1 (abot; http://www.abot.com; [email protected])
About About/0.1libwww-perl/5.47
AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com)
Accoona accoona
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Acoi AcoiRobot
Acoon Robot Acoon Robot v1.50.001
Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com)
Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o
rg)
Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)
AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html)
Aesop AESOP_com_SpiderMan
Agada Mozilla/4.0 (agadine3.0) www.agada.de
agadine/1.x.x (+http://www.agada.de)
Mozilla/4.0 (agadine3.0)
AgentName AgentName/0.1 libwww-perl/5.48
Aibot AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C
hina)
Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com; [email protected])
Alacra PortalBSpider/2.0 ([email protected])
Aladin.de Aladin/3.324
Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/)
Alexa ia_archiver
AlkalineBOT AlkalineBOT/1.3
AlkalineBOT/1.4 (1.4.0326.0 RTM)
Allesklar.de Allesklar/0.1 libwww-perl/5.46
Allrati Allrati/1.1 (+)
Almaden http://www.almaden.ibm.com/cs/crawler
http://www.almaden.ibm.com/cs/crawler [hc5]
Altavista Scooter2_Mercator_x-x.0
Scooter-W3.1.2
Scooter-ARS-1.1-ih
Scooter-3.0.HD
Scooter-3.0.FS
Scooter-3.0.VNS
Scooter-3.0QI
Scooter-3.2.BT
Scooter-3.2
Scooter-3.0.EU
Scooter/2.0 G.R.A.B. X2.0
Scooter/2.0 G.R.A.B V1.0
Scooter/1.0
Scooter/1.0 [email protected]
Scooter/1.1 (custom)
Scooter/2.0 G.R.A.B. V1.1.0
Scooter-3.2.DIL
Scooter-3.2.EX
Scooter-3.3dev
Scooter/3.3.vscooter
Scooter-ARS-1.1
Scooter_bh0-3.0.3
scooter-venus-3.0.vns
Scooter_trk3-3.0.3
Scooter/3.3_SF
Scooter/3.3.QA.pczukor
Scooter-3.2.NIV
Scooter-3.2.JT
Scooter-3.2.SF0
Scooter-3.2.snippet
Scooter/3.3
Scooter-W3-1.0
Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; [email protected])
Amidalla libwww-perl/5.65
Annomille AnnoMille spider 0.1 alpha - http://www.annomille.it
AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/)
AnswerBus AnswerBus (http://www.answerbus.com/)
Answerchase PROve AnswerBot 4.0
Antibot antibot-V1.3.3.1/debian-linux-sarge
Any Search Info Mozilla/4.0 (Sleek Spider/1.2)
Anzwers Australia AnzwersCrawl/2.0 ([email protected];Engine)
Apexoo Spider Apexoo Spider 1.0
ApocalXExplorerBot ApocalXExplorerBot
Aport Aport
Appie appie 1.1 (www.walhello.com)
appie 1.1 (www.walhello.com)
Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html)
ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby
.com;)
Arachnoidea Arachnoidea ([email protected])
ArchitextSpider ArchitextSpider
Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht
tp://www.archive-it.org)
Arexera X-Crawler
TECOMAC-Crawler/0.x
Arianna www.arianna.it
Arikus_Spider Arikus_Spider
Asahina Asahina-Antenna/1.x
Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
Ask 24x Info ask.24x.info
Ask Jeeves/Teoma Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr
awling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_cra
wling.html)
Mozilla/2.0 (compatible; Ask Jeeves)
Asked asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info)
Asterias asterias/2.0
Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; [email protected]); SpiderThread Revis
ion: 3.11
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; [email protected]); SpiderThread Revisi
on: 3.10
Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)
Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)
Attentio Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@att
entio.com)
Augurnet Swiss augurfind
augurnfind V-1.x
Axada axadine/ (Axadine Crawler; http://www.axada.de/; )
Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)
Axmo AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine
.
Ay-Up FastBug http://www.ay-up.com
BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)
Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be)
BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us)
BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; [email protected])
Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html)
Baiduspider BaiduImagespider+(+http://www.baidu.jp/search/s308.html)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.ht
ml)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
BaiDuSpider
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots
meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a
spx; tgautier at balihoo dot com)
BanBot BanBots/1.2 ([email protected])
BeamMachine BeamMachine/0.5 (dead link remover of www.beammachine.net)
Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)
BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )
BecomeBot Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become
.com/site_owners.html)
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.
com/site_owners.html)
BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom
e.co.jp/site_owners.html)
BeijingCrawler BeijingCrawler
BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; [email protected])
Biglotron BIGLOTRON (Beta 2;GNU/Linux)
Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; [email protected])
Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; [email protected])
BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS ([email protected])
BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html
; [email protected])
BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
BinGet BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/)
Bingbot Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Bisnisseek Custom Spider www.bisnisseek.com /1.0 Bisnisseek
Bitacle Robot Bitacle bot/1.1
Bitacle Robot (V:1.0;) (http://www.bitacle.com)
Bitlybot bitlybot
Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net)
Blaiz-Bee Blaiz-Bee/2.00.5502 (+http://www.blaiz.net)
Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
Blitzsuche Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)
[email protected]
[email protected] (Mozilla compatible)
BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers
)
BlogScope Mozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto
)
BlogSearch BlogSearch/1.x +http://www.icerocket.com/
BlogSearch/1.0 +http://www.icerocket.com/
BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)
Blogbot Naamah 1.0.1/Blogbot (http://blogbot.de/)
Naamah 1.0a/Blogbot (http://blogbot.de/)
Blogdex BlogBot/1.x
Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com)
Bloglines Bloglines Title Fetch/1.0 (http://www.bloglines.com)
Bloglines/3.1 (http://www.bloglines.com)
Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com)
BlogzIce BlogzIce/1.0 +http://www.icerocket.com/
BlogzIce/1.0 (+http://icerocket.com; [email protected])
BnF bot Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/)
Boitho boitho.com-dc
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-robot/1.x (http://www.boitho.com/bot.html)
boitho.com-robot/1.x
Bot bot/1.0 (bot; http://; [email protected])
BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)
Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi
nd.mobi/bot.html [email protected])
Brandwatch magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
BravoBrian bSTOP BravoBrian SpiderEngine MarcoPolo
BStop.BravoBrian.it Agent Detector
BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
Btbot BTbot/0.x (+http://www.btbot.com/btbot.html)
BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler)
BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html)
BullsEye/Intelliseek BullsEye
BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl
[email protected])
Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/)
Butterfly Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) G
ecko/2009032608 Firefox/3.0.8
CCBot CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
CEA larbin_2.6_basileocaml ([email protected])
CMP libwww-perl/5.41
CUPS PrivacyFinder Cache Bot v1.0
Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html)
CamontSpider CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
CanadianContent Search RoboCrawl (www.canadiancontent.net)
RoboCrawl (http://www.canadiancontent.net)
Carleson carleson/1.0
Catall Spider Catall Spider
Catall-Spider Catall-Spider/3.3.3(www.Catall.de)
CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood
lebot; [email protected])
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; [email protected]
m)
Ccubee ccubee/x.0
Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect
ion.com/bot.html )
Charlotte Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; [email protected])
Chen Li Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; [email protected])
Christcentral ChristCRAWLER 2.0
Mozilla/4.0 (compatible; ChristCrawler.com, [email protected])
CipinetBot CipinetBot/1.0 (http://www.cipinet.com/bot.html) Cipinet
CipinetBot (http://www.cipinet.com/bot.html)
Cityreview Robot Cityreview Robot (+http://www.cityreview.org/crawler/)
CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot)
Claymont Search Claymont.com
CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/)
Clushbot Mozilla/5.0 (Clustered-Search-Bot/1.0; [email protected]; http://www.clush.com/)
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Clushbot/2.x (+http://www.clush.com/bot.html)
Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/)
CoBITSProbe CoBITSProbe
Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
oBot ((compatible;Win32))
Combine Combine/3 http://combine.it.lth.se/
Combine/2.0
Combine/2.0 http://combine.it.lth.se/
Cometrics-bot cometrics-bot, http://www.cometrics.de
Cometsystems Crawler ([email protected])
Crawler ([email protected])
Comperio FAST Enterprise Crawler 6 used by Comperio AS ([email protected])
Compete.com larbin_2.2.0 ([email protected])
Computerorgs htdig/3.1.6 (http://computerorgs.com)
Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen
e.apache.org)
ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Converas RetrievalWare Convera Internet Spider V6.x
ConveraCrawler/0.2
ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
CrawlConvera0.1 ([email protected])
CoolBot CoolBot
Copyright sheriff copyright sheriff (+http://www.copyrightsheriff.com/)
Cortina Vision Research Lab image spider at vision.ece.ucsb.edu
CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml)
Cowbot Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / [email protected])
Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / [email protected])
CrawlerBoy CrawlerBoy Pinpoint.com
Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp)
Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; [email protected])
Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
CsCrawler Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo
glespam/crawler.html RPT-HTTPClient/0.3-3
Csci_b659/0.13 csci_b659/0.13
Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/
CurryGuide CurryGuide SiteScan 1.1
CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)
Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)
CydralSpider CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com)
CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com)
DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net
/aboutkr.html)
DAUM Web Robot Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi
cations Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu
nications Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm
unications Corp., Korea)
DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
DataFountains DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents
.shtml)
DataFountains/DMOZ Downloader
DataSpear Spider Bot DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; [email protected]
m)
DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html
; [email protected])
DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/)
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DaviesBot DaviesBot/1.7 (www.wholeweb.net)
Daypop daypopbot/0.x
DbDig dbDig(http://www.prairielandconsulting.com)
De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)
DeepIndexer DeepIndexer.ca
Deepak-USC/ISI deepak-USC/ISI
Deepindex Deepindex V2
DeepIndex (www.en.deepindex.com)
DeepIndex
Denmex Websearch Denmex websearch (http://search.denmex.com)
DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net)
Dev-spider2 dev-spider2.searchpsider.com/1.3b
DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)
Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de
Diggit Digger/1.0 JDK/1.3.0rc3
Direct Hit Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)
Disco-crawl disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@
discoveryengine.com)
disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc
overyengine.com)
Ditto DittoSpyder
DoCoMo DoCoMo/1.0/Nxxxi/c10
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.go
o.ne.jp/door/crawler.html)
DoCoMo/1.0/Nxxxi/c10/TB
DoCoMo/2.0 P900iV(c100;TB;W24H11)
Dodgebot dodgebot/experimental
DotBot DotBot/1.0.1
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne
tdotcom.org)
Doubanbot Doubanbot/1.0 ([email protected] http://www.douban.com)
Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/)
EyeCatcher (Download-tipp.de)/1.0
Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html)
DtSearchSpider dtSearchSpider
DuckDuckBot DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Dumbot Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)
Dumbot(version 0.1 beta - dumbfind.com)
Dumbot(version 0.1 beta)
E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)
EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at
global-opto dot com)
EMPAS_ROBOT EMPAS_ROBOT
ESISmartSpider ESISmartSpider
Earthcom Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)
EARTHCOM.info/1.xbeta [www.earthcom.info]
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
EARTHCOM.info/1.x [www.earthcom.info]
EasyDL EasyDL/3.xx http://keywen.com/Encyclopedia/Bot
EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
EasyDL/3.xx
Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5
Echo.fr EchO!/2.0
Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/)
Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm)
Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)
Elsop LinkScan/11.0beta2 Unix
LinkScan/x.x Unix
LinkScan/9.0g Unix
EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)
Enfish Tracker Enfish Tracker
Enoola enoola (http://www.enoola.com)
Enterprise Search Search/1.0 (http://www.innerprise.net/es-spider.asp)
Enterprise_Search/1.0.xxx
Enterprise_Search/1.0
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
Entireweb Speedy_Spider (http://www.entireweb.com)
Speedy Spider (Beta/x.x; [email protected])
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
WorldLight
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
EroCrawler EroCrawler
Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com)
EuripBot EuripBot/0.4 (+http://www.eurip.com) PreCheck
EuripBot/0.4 (+http://www.eurip.com) GetFile
EuripBot/0.5 (+http://www.eurip.com) PreCheck
EuripBot/0.2 (+http://www.eurip.com) GetRobots
Euro-spider Euro-Spider Shopping 1.0
Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; [email protected])
EvaalSE EvaalSE - [email protected]
Eventax eventax/1.3 (eventax; http://www.eventax.de/; [email protected])
Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra
wlerhelp)
Evrinid-RT Mozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid)
Exabot Exalead NG/MimeLive Client (convert/http/0.120)
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
NG/2.0
Exabot/2.0
ExaBotTest/3.0
Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko)
Exabot-Test/1.0
Exabot/3.0
ExaBotTest/2.0
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
NG/4.0.1229
Exabot-Images/1.0
ExactSEEK eseek-larbin_2.6.2 ([email protected])
ExactSeek Crawler/0.1
exactseek-crawler-2.63 ([email protected])
exactseek.com
exactseek-pagereaper-2.63 ([email protected])
ExactSeek_Spider ExactSeek_Spider
Excalibur Excalibur Internet Spider V6.5.4
Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; [email protected])
FAST-WebCrawler FAST-WebCrawler/2.0.9 ([email protected]; http://www.fast.no/faq/faqfastwebsearch/
faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre5 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 ([email protected]; http://www.fast.no/faq/faqfastwebsearch
/faqfastwebcrawler.html)
FAST-SoccerCrawler/2.2-pre-cvs ([email protected]; http://www.fast.no/faq/faqfastw
ebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1.pre.2000-04-14.1 ([email protected])
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support
/crawler.asp)
FAST-WebCrawler/2.1-pre2 ([email protected])
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 ([email protected])
FAST-WebCrawler/2.1.pre.2000-04-18.1 ([email protected]; http://www.fast.no/faq/fa
qfastwebsearch/faqfastwebcrawler.html)
fastlwspider/1.0
FAST-WebCrawler/2.2-pre3 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 ([email protected]; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 ([email protected]; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST Enterprise Crawler/6.4.18 ([email protected])
FAST-WebCrawler/2.1-pre12 ([email protected]; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre13 ([email protected]; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 ([email protected]; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 ([email protected]; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FDSE Mozilla/4.0 (compatible; FDSE robot)
FaXobot Faxobot/1.0
Factbot Factbot 1.09 (see http://www.factbites.com/webmasters.php)
factbot : http://www.factbites.com/robots
FairShare Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairsh
are.cc)
Fast Search PycURL
Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de)
FauBot Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu)
Feed24 Feed24.com
FeedChecker FeedChecker/0.01
Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Feedfinder feedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/
Felix Felix - Mixcat Crawler (+http://mixcat.com)
Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi
[email protected])
FindLinks findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
http://wortschatz.uni-leipzig.de/findlinks/
findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
FineBot FineBot
Firefly Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
Firefly/1.0
FirstGov FirstGov.gov Search - POC:[email protected]
Firstsbot firstsbot
Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio
d com)
Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.
flatlandindustries.com/flatlandbot.php; [email protected])
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan
dindustries.com/flatlandbot.php; [email protected])
flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu
stries.com/flatlandbot.php; [email protected])
FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3
Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se
archhippo.com)
FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)
Folkd.com Spider Folkd.com Spider/0.1 beta 1 (www.folkd.com)
ForAll.pl-Crawler ForAll.pl-Crawler/1.0
Francis Francis/1.0 ([email protected] http://www.neomo.de/)
FreshNotes crawler FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com FreshNotes
FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com
Freshmeat freshmeat.net URL validator/1.1
FuchsBot FuchsBot +http://www.fuchsbot.tld
FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn
[email protected])
FuseBulb FuseBulb.Com
FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
GAIS Robot GAIS Robot/1.0B2
GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; [email protected],[email protected]
m)
GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin
e.php)
GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/)
Gaaz gazz/x.x ([email protected])
Gaisbot Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot Mozilla/4.0 (compatible; www.galaxy.com)
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/)
GammaSpider GammaSpider/1.0
GenieKnows Mozilla/5.0 ([email protected])
larbin_2.6.3 ([email protected])
Mozilla/5.0 [email protected]
geniebot [email protected]
GeonaBot GeonaBot 1.x; http://www.geona.com/
Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu)
Geourl Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot)
GigaBaz Brainbot MicroBaz
gigabaz/3.1x ([email protected]; http://gigabaz.com/gigabaz/)
Gigabot Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gigabot/2.0
Gigabot/2.0/gigablast.com/spider.html
Gigabot/2.0; http://www.gigablast.com/spider.html
Gigabot/2.0att
Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf
a dot com; http://www.girafa.com)
GlobalQueue Look.com
GnodSpider GNODSPIDER (www.gnod.net) Gnod
GoForIt GoForIt.com
GOFORITBOT ( http://www.goforit.com/about/ )
Goblin Goblin/0.9 (http://www.goguides.org/)
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)
Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp
Gonzo2 gonzo1[P] mailto:[email protected]
gonzo2[P] +http://www.suchen.de/faq.html
gonzo2[P] mailto:[email protected]
Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; [email protected]; http://www.inktomi.com/slurp.html)
Google Keyword Tool Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select
/KeywordToolExternal)
Google-Adsense Mediapartners-Google/2.1
Mediapartners-Google
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
Google-Image Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Googlebot-Image/1.0
Google-Sitemaps Google-Sitemaps/1.0
Google-WAP Google WAP Proxy/1.0
Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
GoogleBot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible;
Googlebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0 ([email protected] http://googlebot.com/)
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/1.0 ([email protected])
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog
lebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0
Googlebot/2.0 beta ([email protected])
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/Test ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
no-user-agent-in-crawltrack-for-that-bot
Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b
ot.html)
KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google
bot-Mobile/2.1; +http://www.google.com/bot.html)
Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html)
Gromit Gromit/1.0
Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru
b.org)
Gsa-crawler gsa-crawler (Enterprise; GIX-03519; [email protected])
gsa-crawler (Enterprise; GIX-04637; [email protected])
Gulliver Gulliver/1.2
Gulliver/1.3
GulperBot Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim
/cgi-bin/Link/GulperBot)
Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)
GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
Harvest-NG Harvest-NG/1.0.2
Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)
HatenaScreenshot HatenaScreenshot/1.0 (checker)
HatenaScreenshot/1.0 (checker)
Hbtronix.spider hbtronix.spider.2 -- http://hbtronix.de/spider.php
HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp)
Helix Helix/1.x (+http://www.sitesearch.ca/helix/)
HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp)
HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)
Heritrix Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs
.washington.edu/research/networking/websys/)
archive.org_bot
mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org)
Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec
ts/crawling/)
Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net)
Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)
Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider
Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com
Holmes holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/x.x
holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
HolmesBot HolmesBot (http://holmes.ge)
HomePageSearch HomePageSearch(hpsearch.uni-trier.de)
Homerbot Homerbot: www.homerweb.com
Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda
-search.com)
Hoowwwer HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:
crawler-info<at>hiit.fi)
HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>
hiit.fi)
Htdig htdig/3.1.x (root@localhost)
Htdig/3.1.6 htdig/3.1.6 ([email protected])
I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search
.com)
ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c
ontact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con
tact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a
t)ml(dot)nict(dot)go(dot)jp)
ICCrawler ICCrawler - ICjobs (http://www.icjobs.de/bot.htm)
ICRA_Label_spider ICRA_label_spider/x.0
IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a
t iiit dot ac dot in)
INGRID Mozilla/3.0 (INGRID/3.0 MT; [email protected]; http://webmaster.
ilse.nl/jsp/webmaster.jsp)
IP2MapBot IP2MapBot/1.1 http://www.ip2map.com
IPiumBot IPiumBot laurion(dot)com
IRLbot IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
IScraperBot IScraperBot/0.1
IWAgent IWAgent/ 1.0 - www.brandprotect.com
Iaskspider2 iaskspider2 ([email protected])
Ichiro ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/1.0 ([email protected])
ichiro/1.0 ([email protected])
IconSurf IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
Icsbot icsbot-0.1
Ideare ideare - SignSite/1.x
IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator
e.it)
Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo
r more information please visit http://www.ilial.com/crawler; http://www.ilial.c
om/crawler; [email protected])
ilial/Nutch-0.9-dev
Ilse Mozilla/3.0 (Vagabondo/2.0 MT; [email protected]; http://aanmeld
en.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (INGRID/3.0 MT; [email protected]; http://aanmelden.
ilse.nl/?aanmeld_mode=webhints)
ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com)
IncyWincy NetResearchServer/x.x(loopimprovements.com/robot.html)
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy/2.1(loopimprovements.com/robot.html)
IncyWincy page crawler([email protected],http://www.loopimprovement
s.com/robot.html)
IncyWincy data gatherer([email protected],http://www.loopimprovemen
ts.com/robot.html)
IncyWincy (Look) IncyWincy(http://www.look.com)
IndexTheWeb IndexTheWeb.com Crawler7
Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia
Interactive
Indy Library Mozilla/3.0 (compatible; Indy Library)
InelaBot InelaBot/0.2 (+http://inelegant.org/bot)
Inet Library Inet library
InfoFly InfoFly/1.0 (http://www.versions-project.org/)
InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/)
InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht
ml; [email protected])
Infoseek InfoSeek Sidewinder/0.9
Inria larbin_2.2.1_de_Viennot ([email protected])
Insitor Search robot Insitor.com search and find world wide!
Insitornaut Insitornaut
Internet Ninja Internet Ninja x.0
Internetseer InternetSeer.com
Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56)
IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; [email protected])
Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot
com)
Ishida Lab larbin_2.2.2 ([email protected])
It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt
rovatore.it/bot.html; [email protected])
Jabot Jabot/7.x.x (http://odin.ingrid.org/)
Jabot/6.x (http://odin.ingrid.org/)
Jambot Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot
; [email protected])
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; [email protected])
Jayde Crawler Jayde Crawler. http://www.jayde.com
Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51)
Jetbot Jetbot/1.0
Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; [email protected])
( newsexpress e-mail: [email protected] http://www.neofonie.de/loesunge
n/search/robot.html )
Jomjaibot Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com)
Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za; [email protected])
Jyxobot Jyxobot/x
Jyxobot/1
K2 Spider k2spider
KAIST AITrc Crawler KAIST AITrc Crawler
KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de)
KIT_Fireball KIT_Fireball/2.0
KS Crawler KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; [email protected])
KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte
nt/crawler/index.html; [email protected])
KakleBot KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bo
t.html; [email protected])
KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; [email protected])
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw
[email protected])
Kasparek Firefox_1.0.6 ([email protected])
Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html)
Kenjin Spider Kenjin Spider
Kevin Kevin http://dznet.com/kevin/
Kevin http://websitealert.net/kevin/
KicktooBot kicktooBotV1.1 [email protected]
Kinja-imagebot kinja-imagebot (http://www.kinja.com/)
Kinjabot kinjabot (http://www.kinja.com)
KnowItAll KnowItAll([email protected])
Knowledge.com Knowledge.com/0.x
Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info
.html; [email protected])
Kulokobot kuloko-bot/0.x
kulokobot www.kuloko.com [email protected]
Kulturarw kulturarw3/0.1
Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez)
Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k
yluka.com)
LECodeChecker LECodeChecker/3.0 libgetdoc/1.0
LNSpiderguy LNSpiderguy
LapozzBot LapozzBot/1.4 ( http://robot.lapozz.com)
LapozzBot/1.5 (+http://robot.lapozz.hu)
LapozzBot/1.4 (+http://robot.lapozz.hu)
Larbin_2.6.3 larbin_2.6.3 [email protected]
larbin_2.6.3 [email protected]
Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo
.com; [email protected])
Lemur Consulting larbin_2.6.2 ([email protected])
Lexibot Mata Hari/2.00
LexiBot/1.00
Liafa larbin_2.2.2_guillaume ([email protected])
LibWeb libWeb/clsHTTP -- [email protected]
LibertyW LibertyW (+http://www.lw01.com)
LibertyW (+http://www.libertyw.eu)
LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co
m)
Linguee Bot Linguee Bot (http://www.linguee.com/bot; [email protected])
LinkWalker LinkWalker
Linknzbot linknzbot
Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine)
Links4US-Crawler Links4US-Crawler, (+http://links4us.com/)
LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck
er.html)
Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)
Lmspider lmspider ([email protected])
LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/)
Lockstep Spider Lockstep Spider/1.0
Look.com NetResearchServer(http://www.look.com)
LookdirBot LookdirBot
Lovel Lovel as 1.0 ( +http://www.everatom.com)
Ltaa_web_crawler larbin_2.6.3 ([email protected])
Luchs.at URL checker luchs.at URL checker
Lycos_Spider Lycos_Spider_(modspider)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_
Spider_Beta2(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide
r_(T-Rex) )
Lycos_Spider_(T-Rex)
MJ12bot Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; [email protected]
uc.edu)
MSN Bot msnbot/0.9 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot)
MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; [email protected])
Mail.Ru Mail.Ru/1.0
Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT)
Mammoth Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
mammoth/1.0 (+http://www.sli-systems.com/)
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
MantraAgent MantraAgent
Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)
Martini Martini
MARTINI
Marvin Marvin v0.3
Masterseek MasterSeek
Maxbot Spider/maxbot.com [email protected]
Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html;
[email protected])
MediaCrawler MediaCrawler-1.0 (Experimental)
MediaSearch MediaSearch/0.1
Mediater Rechercher libwww/5.3.2
MegaSheep MegaSheep v1.0 (www.searchuk.com internet sheep)
Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com)
Melbot WebSpider Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.)
Mercator Mercator-1.x
Mercator-2.0
Mercator-Scrub-1.1
Merl.com larbin_2.1.1 [email protected]
MetaGer_PreChecker MetaGer_PreChecker0.1
Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) ([email protected])
Metadata Labs MLBot (www.metadatalabs.com/mlbot)
Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta
euro.com; crawler at metaeuro dot com)
Metager-Linkchecker MetaGer-LinkChecker
MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de; )
Metaquerier MQbot http://metaquerier.cs.uiuc.edu/crawler
MQbot metaquerier.cs.uiuc.edu/crawler
Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner
.de/)
Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/)
Microsoft Small Business Indexer Microsoft Small Business Indexer
Microsoft URL Control Microsoft URL Control - 6.01.9782
Microsoft URL Control - 6.00.8862
Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/)
miggibot/2006.01.24
Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s
ee getmirar.com)
Missigua Locator 1.9 Missigua Locator 1.9
Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot
.fr)
Miva Miva ([email protected])
MnoGoSearch MnogoSearch/3.2.xx
Mo College 1.9 Mo College 1.9
Moget moget/x.x ([email protected])
Mogimogi mogimogi/1.0
Moi-même no-user-agent-in-crawltrack-for-that-bot Moi
MojeekBot Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)
MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)
Monogol http://www.monogol.de
Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com)
Morris Morris - Mixcat Crawler (+http://mixcat.com)
Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.
com/bot)
MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; [email protected])
Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; [email protected])
MultiText MultiText/0.1
Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.5; [email protected])
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; [email protected])
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; [email protected])
MusicWalker MusicWalker2.0 (+http://www.somusical.com)
Mxbot Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info)
MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx
; [email protected])
Mylinea Mylinea.com Crawler 2.0
NASA Search 1.0 NASA Search 1.0
NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)
NG-SearchBot NG-Search/0.9.8 (http://www.ng-search.com)
NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )
NII larbin_2.6.2 ([email protected])
NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.
89 (like Gecko)
NMG Spider NMG Spider/0.3 (szukanko.com)
NTT Directory nttdirectory_robot/0.9 ([email protected])
NWSpider NWSpider 0.9
Nabot nabot_1.0
NABOT/5.0
Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x)
Nameprotect NP/0.1 (NP; http://www.nameprotect.com; [email protected])
NPBot-1/2.0
NPBot (http://www.nameprotect.com/botinfo.html)
Nationaldirectory NationalDirectory-WebSpider/1.3
NationalDirectoryAddURL/1.0
NatzanBot NatzanBot, http://www.natzan.com
NaverBot dloader(NaverRobot)/1.0
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / [email protected])
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
NaverBot_dloader/1.5
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.
jsp)
NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / [email protected])
NavissoBot NavissoBot
NavissoBot/1.7 (+http://navisso.com/)
Nebulla Nebullabot/2.2 (http://bot.nebulla.info)
Nebulla/V1.0 Spider (http://spider.nebulla.de)
Nebullabot/2.2 (http://bot.nebulla.de)
Nelian Pty Ltd - Spider Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com )
NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience:
http://www.netnose.com)
NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html)
NetSprint -- 2.0 NetSprint -- 2.0
NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com
; [email protected])
NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don'[email protected])
Netprospector Netprospector JavaCrawler
Netscape Robozilla/1.0
NextGenSearchBot NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot
)
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB
ot)
NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe
nSearchBot.aspx)
NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1
NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro
blems contact: [email protected]
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro
blems contact: [email protected]
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob
lems contact: [email protected]
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe
ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
NjuiceBot Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
Noago Spider Noago Spider
NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm)
Norbert Norbert the Spider(Burf.com)
Noxtrumbot noxtrumbot/1.0 ([email protected])
Noyona noyona_0_1
Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com)
NuSearch Spider nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible
; MSIE 4.01)
Nutch NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
Nutch
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis
ts.sourceforge.net)
NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy
[email protected])
NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o
rg/nutch/bot.html; ec2test at lucene.com)
OOZBOT OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Obidos-bot obidos-bot (just looking for books.)
Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3
E
Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup
[email protected])
Ocelli Ocelli/1.3 (http://www.globalspec.com/Ocelli)
Ocelli/1.x (http://www.globalspec.com/Ocelli)
Octora Octora Beta - www.octora.com
Octora Beta Bot Octora Beta Bot - www.octora.com
OmniExplorer OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OnetSzukaj Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)
Onet.pl SA, http://szukaj.onet.pl
OpenISearch OpenISearch/1.x (www.openisearch.com)
OpenPortal4U DigOut4U
OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm)
OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2
OpenWebSpider OpenWebSpider/x
Openfind Openfind Robot/1.1A2
Openbot/3.0+([email protected];+http://www.openfind.com.tw/robot.ht
ml)
Openfind data gatherer, Openbot/3.0+([email protected];+http://www.
openfind.com.tw/robot.html)
OpidooBot OpidooBOT ([email protected])
Oracle Oracle Ultra Search
OrangeBot-Mobile Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19
.123.2.733) OrangeBot-Mobile 2008.0 ([email protected])
OrangeSpider OrangeSpider
Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor
t/crawler.asp)
OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free)
Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat
.de)
PJspider PJspider/3.0 ([email protected]; http://www.portaljuice.com)
PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)
Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/)
Page_verifier page_verifier http://www.securecomputing.com/goto/pv
Pagebull Pagebull http://www.pagebull.com/
Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes ([email protected])
FAST Enterprise Crawler 6 used by Pages Jaunes ([email protected])
FAST Enterprise Crawler 6 used by Pages Jaunes ([email protected])
PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi
on 0.001
Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/
crawl.html)
ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/)
PatchOneSpider Mozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se)
Patwebbot Patwebbot (http://www.herz-power.de/technik.html)
Peerbot PEERbot www.peerbot.com
Petitsage www.petitsage.fr site detector 0.4
PicoSearch PicoSearch/1.0
PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; [email protected])
Piffany Piffany_Web_Spider_v0.x
Piffany_Web_Scraper_v0.x
Pilot Hitlist Marketwave Hit List
Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com)
PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
Pita Pita
PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 [email protected]
PlantyNet_WebRobot_V1.9 [email protected]
PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1
subscribers)
Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin
fo.htm)
Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; [email protected])
Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x)
Polybot polybot 1.0 (http://cis.poly.edu/polybot/)
Pompos Pompos/1.3 http://dir.com/pompos.html
Pompos/1.x [email protected]
Pompos
Pompos/1.2 http://dir.com/pompos.html
Pompos/1.1 http://dir.com/pompos.html
Pompos/1.x http://dir.com/pompos.html
PopJapanSearch Robot/www.pj-search.com
Popdexter Popdexter/1.0
PrivacyFinder/1.1 PrivacyFinder/1.1
ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com)
Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi
a.com)
Psbot psbot/0.1 (+http://www.picsearch.com/bot.html)
Psycheclone psycheclone
Pubblisito [email protected] (http://www.pubblisito.com) il Sud dei Motori di Ricerca
Pumpkin blogsearchbot-pumpkin-3
Purebot Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)
Python-urllib Python-urllib/1.16
QPCreep QPCreep Test Rig ( We are not indexing, just testing )
QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 [email protected]
et)
Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com)
Qube qube/2.0 (+http://qube.qelix.com/v2)
QuepasaCreep QuepasaCreep v0.9.1x
QuepasaCreep ( [email protected] )
QueryN Metasearch QueryN Metasearch
Qweerybot Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)
QweeryBot/3.01 ( http://qweerybot.qweery.nl)
QweeryBot/3.02 ( http://qweerybot.qweery.nl)
R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6
R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6
R6_FeedFetcher(www.radian6.com/crawler)
RAMPyBot RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu
cene.apache.org)
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveram
p.com/bot.html; [email protected])
RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com)
Rambot ramBot xtreme x.x
RankMeter Web Snooper
Rational SiteCheck Rational SiteCheck (Windows NT)
Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper)
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)
RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net)
RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)
RoboPal RoboPal (http://www.findpal.com/)
Rotondo Rotondo/3.1 libwww/5.3.1
RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)
RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html)
S2Bot S2Bot/1.0 (http://search2.net)
SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com)
SBIder SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell
.com/contact-support.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site
sell.com/contact-support.html)
SandCrawler SandCrawler - Compatibility Testing
Savvybot savvybot/0.2
ScanWeb ScanWeb
Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/
/www.schibstedsok.no/bot/)
ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho
laruniverse.com)
Scirus-crawler FAST Enterprise Crawler 6 / Scirus [email protected]; http://www.scirus.com
/srsapp/contactus/
ScollSpider ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots
ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
ScoutJet ScoutJet
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Scrubby Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Scrubby/2.x (http://www.scrubtheweb.com/)
Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache
.org/nutch/bot.html; [email protected])
Search-Engine-Studio Search-Engine-Studio
Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html)
Search.ch search.ch V1.4.2 ([email protected]; http://www.search.ch)
search.ch V1.4
Search17Bot Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
Search4free lwp-trivial/1.34
SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs
a.com)
SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
SearchExpress Spider SearchExpress Spider0.99
SearchMonkey Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/sea
rchmonkey/useragent)
SearchScout DiamondBot
ClariaBot/1.0
Diamond/1.0
SearchSight SearchSight/2.0 (http://SearchSight.com/)
SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi
der.com)
SearchSpider.com/1.1
SearchdayBot SearchdayBot
Searchdnabot searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at sear
chenginedna dot com)
Searchguild SearchGuild_DMOZ_Experiment ([email protected])
Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)
Searchmee! Spider Searchmee! Spider v0.98a
Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Seeker.lookseek Seeker.lookseek.com
Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/)
Semager/1.0 (http://www.semager.de)
Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau)
SeznamBot SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
SeznamBot/1.0 (+http://fulltext.seznam.cz/)
SeznamBot/1.0
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)
SheenBot SheenBot/SheenBot-1.0.4 (Sheen web crawler)
Sherlock sherlock/1.0
Shim-Crawler Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra
[email protected])
Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl
er/; [email protected])
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c
[email protected])
ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
ShopWiki/1.0 ( +http://www.shopwiki.com/)
Shoula Shoula.com Crawler 2.0
SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz)
Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com)
Silk/1.0 silk/1.0 (+http://www.slider.com/silk.htm)/3.7
silk/1.0
Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html)
Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR
SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru)
SiteSpider SiteSpider +(http://www.SiteSpider.com/)
SiteTruth SiteTruth.com site rating system
SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day)
Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en]
Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE
.com)
SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
Slider Slider_Search_v1-de
Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)
Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Slurp Inktomi Mozilla/3.0 (Slurp/cat; [email protected]; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; [email protected]; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/cat; [email protected]; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; [email protected]; http://www.inktomi.com/slurp.html)
Slurp Inktomi (Yahoo) Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071
3 Firefox/2.0.0.5
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch
/slurp)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE
cho/2.0.0.4
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu
rp)
Slurpy Verifier Slurpy Verifier/1.0
SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0
SnapPreview Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
Snapbot Snapbot/1.0
Snoopy Snoopy v1.2
SnykeBot SnykeBot/0.6 (http://www.snyke.com)
SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com)
Sogou web spider Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sohu-search sohu-search
Somewhere Mozilla ([email protected])
Sosospider Sosospider+(+http://help.soso.com/webspider.htm)
Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
Spbot Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Speedfind speedfind ramBot xtreme 8.1
Speedy Spider Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Sphere Scout Sphere Scout&v4.0 - scout at sphere dot com
Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings ([email protected]
)
SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml)
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spinn3r Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.
1); http://spinn3r.com/robot) Gecko/20021130
SplatSearch libwww-perl/5.45
Spock Crawler Spock Crawler (http://www.spock.com/crawler)
Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr
oose.com)
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; [email protected]
om)
StackRambler StackRambler/x.x
StackRambler/2.0 (MSIE incompatible)
StackRambler/2.0
Steeler Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com)
Suchbaer suchbaer.de
suchbaer.de (CrawlerAgent v0.103)
Suchpad suchpad/1.0 (+http://www.suchpad.de)
SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com)
SuperSnooper [email protected]
SurveyBot SurveyBot/2.3 (Whois Source)
Susie !Susie (http://www.sync2it.com/susie)
Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html)
SycaBoT SycaBoT/1.0
SycaBoT-Audio SycaBoT-Audio
SycaBoT-Image SycaBoT-Image
SycaBoT-Programme SycaBoT-Programme
SycaBoT-Video SycaBoT-Video
Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr
[email protected])
Sygol <http://www.sygol.com/> http://www.sygol.com
SygolBot SygolBot http://www.sygol.com
SygolBot http://www.sygol.net
SynooBot SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.
html)
Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)
Synthesio Crawler Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler
Szukacz Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; [email protected])
Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])
TAGword Tagword (http://tagword.com/dmoz_survey.php)
TCDBOT TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot
IE)"
Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/)
TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta
ilrank.com/robot) Gecko/20021130
Talkro Talkro Web-Shot/1.0 (E-mail: [email protected], Home: http://222.122.15.190/w
ebshot)
TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/
TargetSeek.html)
Tasapspider Tasapspider/0.9 (http://www.tasap.com)
Technoratibot Technoratibot/0.7
Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm)
Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w
ww.netseer.com/crawler.html; [email protected])
TelenetDigger TelenetDigger/1.0; htdig/3.1.6 ([email protected])
Teoma teomaagent [email protected]
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
teoma_agent1
Teoma MP
teomaagent1 [[email protected]]
Teradex Mapper Teradex Mapper; [email protected]; http://www.teradex.com
TeragramCrawler TeragramCrawler
TeragramWebcrawler TeragramWebcrawler/1.0
TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)
TheSuBot TheSuBot/0.1 (www.thesubot.de)
Theme Spider Theme Spider (+http://www.themespider.com/spider.html)
Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th
eophrastus.php)
Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)
Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)
TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html
TinEye TinEye/1.1 (http://tineye.com/crawler.html)
TivraSpider tivraSpider/1.0 ([email protected])
Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html)
ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1
Topodia Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia
.com/; [email protected])
Toutatis Toutatis x.x (hoppa.com)
Toutatis x.x-x
Toutatis x-xx.x (hoppa.com)
Traazibot traazibot/testengine (+http://www.traazi.de)
Trampelpfad Trampelpfad-Spider
Trampelpfad-Spider-v0.1
TrendTech http://www.trendtech.dk/spider.asp)
Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/)
Turnitin TurnitinBot
TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)
Turnpike Emporium Turnpike Emporium LinkChecker/0.1
TutorGigBot Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)
TutorGig/1.5 (+http://www.tutorgig.com/crawler)
TweetmemeBot Mozilla/5.0 (compatible; TweetmemeBot/1.0; +http://www.tweetmeme.com/)
Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0
.6 TweetmemeBot
TwengaBot TwengaBot/1.1 (+http://www.twenga.com/bot.html)
TwengaBot/2.0 (http://www.twenga.com/bot.html)
Twiceler Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Twiceler www.cuil.com/twiceler/robot.html
Twiceler-0.9 http://www.cuil.com/twiceler/robot.html
Twiceler www.cuill.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
TygoBot TygoBot
TygoProwler
T�zilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez
illa.de/t_st-odp-entries-agent.html)
UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
Mackster( http://www.ukwizz.com )
UN OCHA FAST Enterprise Crawler 6 used by [email protected] (UN OCHA)
UOLCrawler UOLCrawler ([email protected])
URL-Spider Mozilla/5.0 URL-Spider
URL_Spider_Pro URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp)
URL_Spider_Pro/x.x
URL Spider Pro/x.xx (innerprise.net)
USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
Ultraseek Ultraseek
UnChaosBot <a href="http://www.unchaos.com/"> UnChaos Bot, Hybrid Web Search Engine. </a> (
[email protected])
<a href="http://www.unchaos.com/"> UnChaos </a>, From Chaos To Order, Hybrid Web
Search Engine. ([email protected])
<b> UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc
haos.com </b> ([email protected])
Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt
Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html
Updated updated/0.1beta (updated.com; http://www.updated.com; [email protected])
Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
Uptimebot UptimeBot(www.uptimebot.com)
Uptimebot
Urlchecker1.0 Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)
Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com)
VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; [email protected])
VMBot VMBot/0.9 (VMBot; http://www.verticalmatch.com; [email protected])
VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; [email protected])
VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;[email protected]
c.edu
Vacobot Vacobot; (+http://vaco.ws/bot.html)
Vagabondo Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http
://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w
ebagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt
p://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo/3.0 (webagent at wise-guys dot nl)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://
webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/
)/1.0 Profile
Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; [email protected])
Verizon fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt
[email protected])
FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad
[email protected])
Vermut Mozilla/5.0 (compatible; vermut +http://vermut.aol.com)
Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com)
mozilla/5.0 (compatible; vermut +http://www.aol.com)
Versus versus 0.2 (+http://versus.integis.ch)
Versus Crawler versus crawler [email protected]
VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs
Verzamelgids Verzamelgids/2.2 (http://www.verzamelgids.nl)
verzamelgids.nl - Networking4all Bot/x.x
Vespa Crawler (Yahoo) Vespa Crawler
Visbot VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; [email protected])
Visbot/1.0 (+http://www.visvo.com/bot.html;[email protected])
Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html)
VoilaBot Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor
[email protected])
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/
/www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi
la.com/)
KE_1.0/2.0 libwww/5.2.8
Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t)
Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
Voyager voyager-hc/1.0
voyager/1.0
cfetch/1.0
Vspider vspider/3.x
vspider
W3C-Validator W3C_Validator/1.432.2.10
W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803
W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de
W8net Mozilla/5.0 usww.com-Spider-for-w8.net
WEP Search WEP Search 00
WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler)
WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,[email protected])
WISEbot WISEbot/1.0 ([email protected]; http://wisebot.koreawisenut.com)
WWW.fi crawler www.fi crawler, contact [email protected]
WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de)
Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/)
WasaBot Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)
WasaBot/2.4 (+ http://www.wasalive.com ) Java/1.6.0
WasaBot 0.11 (+http://www.wasalive.com/)
Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; [email protected])
Waypath Waypath development crawler - info at waypath dot com
Waypath Scout v2.x - info at waypath dot com
WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer
elate.org)
WeatherBot WeatherBot v1.4 http://www.ezweather.net
WebAlta Crawler WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT
5.1; ru-RU)
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot
/index.html
WebCorp WebCorp/1.0
WebCorp/1.1
WebFindBot WebFindBot(http://www.web-find.com)
WebGobbler webGobbler/1.x.x
WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S
[email protected])
WebSearchBench WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit�t Dortmund, [email protected], http://websearchbench.cs.uni-dortmund.de
/
WebSearchBench WebCrawler v0.1(Experimental)
WSB, http://websearchbench.cs.uni-dortmund.de
WSB WebCrawler V1.0 (Beta), [email protected]
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit?t Dortmund, [email protected], http://websearchbench.cs.uni-dortmund.de
/
WebStat WebStat/1.0 (Unix; beta; 20040314)
WebVac WebVac ([email protected])
WebarooBot WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
Webbandit webbandit/4.xx.0
Webbot webbot(+http://webbot.com/bot.htm)
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Webclipping Webclipping.com
Webcrawl webcrawl.net
webcrawl.net
Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx)
Webglimpse Webglimpse 2.xx.x (http://webglimpse.net)
Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0
Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de
Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/
WebsiteWorth WebsiteWorth v1.0
Webspinne Webspinne/1.0 [email protected]
Websquash Websquash.com (Add url robot)
Webster Webster v0.3 ( http://webster.healeys.net/ )
Webverzeichnis Webverzeichnis.de - Telefon: 01908 / 26005
Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/]
Wikiwix-bot wikiwix-bot-3.0
Willow Willow Internet Crawler by Twotrees V2.1
WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ?
WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB
ot; [email protected])
WiseGuys Vagabondo/2.0 MT
Vagabondo/1.x MT ([email protected])
Mozilla/3.0 (Vagabondo/2.0 MT; [email protected]; http://webagent.wi
se-guys.nl/)
Mozilla/3.0 (Vagabondo/1.x MT; [email protected]; http://webagent.wise-guys.
nl/)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT ([email protected])
Worio Mozilla/5.0 (compatible; woriobot support [at] worio [dot] com +http://worio.com
))
Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/)
Worio bot Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com)
Worio heritrix bot worio heritrix bot (+http://worio.com/)
WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/)
Wotbox Wotbox/alpha0.x.x ([email protected]; http://www.wotbox.com) Java/1.4.1_02
Wotbox/alpha0.6 ([email protected]; http://www.wotbox.com)
Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)
Wwlib/Linux Wwlib/Linux
Wwwster wwwster/1.4 (Beta, mailto:[email protected])
XP5 Project XP5 [2.03.07-111203]
Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html)
Xirq xirq/0.1-beta (xirq; http://www.xirq.com; [email protected])
Xyleme SA France cosmos/0.9_([email protected])
cosmos/0.8_([email protected])
Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yacy yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/
en)
Yacybot yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/
bot.html
yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net
yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_15; Europe/fr) http://yacy
.net/bot.html
yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net
Yahoo Japan robot Y!J-SRD/1.0
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, [email protected])
YahooYSMcm/2.0.0
Yahoo! Mindset Yahoo! Mindset
Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/
us/ysearch/crawling/crawling-02.html )
Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co
m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])
Yahoo-Test/4.0 Yahoo-Test/4.0
Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com;
http://www.alltheweb.com/
YahooFeedSeeker YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu
blishers.html)
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c
om/rssguide)
YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web
[email protected] ) (KHTML, like Gecko)
YahooSeeker/M1A1-R2D2 SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in
c dot com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile
-search-customer-care AT yahoo-inc dot com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i
nc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http:
//help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo
-inc dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo
.com/help/us/ysearch/crawling/crawling-01.html)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-
01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo.
com/help/us/ysearch/crawling/crawling-01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/
crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust
omer-care AT yahoo-inc dot com)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot
com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch
/crawling/crawling-01.html)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli
ng/crawling-01.html)
Yandex Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
Yandex/1.01.001 (compatible; Win16; I)
YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Yarienavoir yarienavoir.net/0.2
Yellopet spider.yellopet.com - www.yellopet.com
YesupBot Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Yeti Yeti/0.01 (nhn/1noon, [email protected], check robots.txt daily and follows it)
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti
Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m
inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres
den dot de)
YodaoBot Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0
/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/
; )
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide
r/; )
YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu)
Yoogli yoogliFetchAgent/0.1
Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/)
Yoono web-crawler yoono/1.0 web-crawler/1.0
YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine
YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine
Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/)
Zao Crawler Zao-Crawler
Zao-Crawler 0.2b
Zao/0.1 (http://www.kototoi.org/zao/)
ZeBot ZeBot_www.ze.bz ([email protected])
ZeBot_lseek.net ([email protected])
Zealbot Mozilla/4.0 (compatible; Zealbot 1.0)
Zearchit Zearchit
Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/)
Zerxbot zerxbot/Version 0.6 libwww-perl/5.79
Zetbot DeepIndex ( http://www.zetbot.com )
Zeus Zeus ThemeSite Viewer Webster Pro V2.9 Win32
Zeus xxxxx Webster Pro V2.9 Win32
ZBot/1.00 ([email protected])
ZipppBot ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; [email protected])
ZipppBot/0.xx (ZipppBot; http://www.zippp.net; [email protected])
Zippy Zippy v2.0 - Zippyfinder.com
ZoomSpider ZoomSpider - wrensoft.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR
1.1.4322)
Zspider zspider/0.9-dev http://feedback.redkolibri.com/
ZyBorg (LookSmart) ZyBorg
ZyBorg (Wisenut) Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnu
tbot.com)
ZyBorg/1.0 ([email protected]; http://www.WISEnut.com)
aiHitBot Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
eCairn-Grabber eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15




Webmaster tool: free crawlers and spiders tracking script- SEO script - Outil pour webmaster: script gratuit de statistiques des visites des robots    Mentions légales