Skip to the main content

Free and open source since 2005, running on your own server

CrawlTrack Webmaster dashboard

Crawlers and spiders who can visit your website

Here is the list of crawlers used by CrawlTrack. The list is updated regularly. As you will see it is already very long, but there are still many spiders crawling the web that are not listed here.

691 entries are shown below, exactly as the detection database holds them. The list is refreshed from the tools page of your installation, and you can add your own crawler by user agent or by address range.

CrawlerUser agent
192.com192.comAgent
4anything4anything.com LinkChecker v2.0
80legsMozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
A OnlineA-Online Search
ABCdatosABCdatos BotLink/5.xx.xxx#BBL
AOLSqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)
ASAHAASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
AVSearchAVSearch-1.0([email protected])
AbbyMozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Aberja CheckomaAberja Checkoma
Abotabot/0.1 (abot; http://www.abot.com; [email protected])
AboutAbout/0.1libwww-perl/5.47
AboutUsBotMozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
AcoiAcoiRobot
Acoon RobotAcoon Robot v1.50.001
Acoon RobotAcoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com)
AcornAcorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org)
ActivtouristMozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)
AdsBot GoogleAdsBot-Google (+http://www.google.com/adsbot.html)
AesopAESOP_com_SpiderMan
AgentNameAgentName/0.1 libwww-perl/5.48
AibotAIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)
AicrawlerAccoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Aipbotaipbot/1.0 (aipbot; http://www.aipbot.com; [email protected])
AlacraPortalBSpider/2.0 ([email protected])
Aladin.deAladin/3.324
Aleksika DanmarkAleksika Spider/1.0 (+http://www.aleksika.com/)
Alexaia_archiver
Allesklar.deAllesklar/0.1 libwww-perl/5.46
AllratiAllrati/1.1 (+)
AmfibibotAmfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; [email protected])
Amidallalibwww-perl/5.65
AnnomilleAnnoMille spider 0.1 alpha - http://www.annomille.it
AnsearchBotMozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/)
AnswerBusAnswerBus (http://www.answerbus.com/)
AnswerchasePROve AnswerBot 4.0
Antibotantibot-V1.3.3.1/debian-linux-sarge
Any Search InfoMozilla/4.0 (Sleek Spider/1.2)
Anzwers AustraliaAnzwersCrawl/2.0 ([email protected];Engine)
Apexoo SpiderApexoo Spider 1.0
ApocalXExplorerBotApocalXExplorerBot
AportAport
ArabulbotMozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html)
ArabyBotArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)
ArachnoideaArachnoidea ([email protected])
ArchitextSpiderArchitextSpider
Archive.org_botMozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org)
Ariannawww.arianna.it
Arikus_SpiderArikus_Spider
Ask 24x Infoask.24x.info
Askedasked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
Askpeter_botMozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info)
Asteriasasterias/2.0
Astrafind!Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)
AtlocalAtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)
AttentioAttentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; [email protected])
Axadaaxadine/ (Axadine Crawler; http://www.axada.de/; )
AxandraMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)
AxmoAxmoRobot - Crawling your site for better indexing on www.axmo.com search engine.
Ay UpFastBug http://www.ay-up.com
BE Internet Search EngineBlaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)
Ba.beMozilla/4.72 [en] (BACS http://www.ba.be)
BaBoom Web PortalBaboomBot/1.x.x (+http://www.baboom.us)
BabalooSpiderBabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; [email protected])
Backlink CheckBacklink-Check.de (+http://www.backlink-check.de/bot.html)
BanBotBanBots/1.2 ([email protected])
BeamMachineBeamMachine/0.5 (dead link remover of www.beammachine.net)
Beauty (Cosmoty)beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)
BebopBotBebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )
BecomeJPBotMozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html)
BeijingCrawlerBeijingCrawler
BigCliqueBigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; [email protected])
BiglotronBIGLOTRON (Beta 2;GNU/Linux)
BilBasenFAST Enterprise Crawler 6 used by BilBasen ApS ([email protected])
BilgiBetaBotBilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; [email protected])
BilgiBotBilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
BinGetBinGet/1.00.A (http://www.bin-co.com/php/scripts/load/)
BisnisseekCustom Spider www.bisnisseek.com /1.0
Bitlybotbitlybot
Blaiz EnterprisesBlaiz-Bee/1.0 (+http://www.blaiz.net)
BlogRefsBotMozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers)
BlogScopeMozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto)
BlogWatcherblogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)
BlogdexBlogBot/1.x
Blogdimension BlogBotBlogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com)
Bloglines ImagesBloglines-Images/0.1 (http://www.bloglines.com)
BnF botMozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/)
Botbot/1.0 (bot; http://; [email protected])
BotSeerMozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)
BotmobiNokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html [email protected])
Brandwatchmagpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
BrightCrawlerBrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
BruinbotBruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
BtbotBTbot/0.x (+http://www.btbot.com/btbot.html)
BuildCMS crawlerBuildCMS crawler (http://www.buildcms.com/crawler)
BuiltWithMozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html)
BullsEye/IntelliseekBullsEye
BurstFindCrawlerBurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; [email protected])
BuscaplusBuscaplus Robi/1.0 (http://www.buscaplus.com/robi/)
ButterflyMozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
CCBotCCBot/1.0 (+http://www.commoncrawl.org/bot.html)
CEAlarbin_2.6_basileocaml ([email protected])
CMPlibwww-perl/5.41
CUPSPrivacyFinder Cache Bot v1.0
CamcrawlerCamcrawler (+http://www.camdiscover.com/crawler.html)
CamontSpiderCamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
Carlesoncarleson/1.0
Catall SpiderCatall Spider
Catall SpiderCatall-Spider/3.3.3(www.Catall.de)
Ccubeeccubee/x.0
ChangedetectionMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
Chen LiChen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; [email protected])
Cityreview RobotCityreview Robot (+http://www.cityreview.org/crawler/)
CjLogbotMozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot)
Claymont SearchClaymont.com
CloakDetectCloakDetect/0.9 (+http://fulltext.seznam.cz/)
Cnet robotMozilla/4.6 [en] (http://www.cnet.com/)
CoBITSProbeCoBITSProbe
Cometrics botcometrics-bot, http://www.cometrics.de
ComperioFAST Enterprise Crawler 6 used by Comperio AS ([email protected])
Compete.comlarbin_2.2.0 ([email protected])
Computerorgshtdig/3.1.6 (http://computerorgs.com)
ComriteComrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
CoolBotCoolBot
Copyright sheriffcopyright sheriff (+http://www.copyrightsheriff.com/)
CortinaVision Research Lab image spider at vision.ece.ucsb.edu
CougarSearchCougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml)
CrawlerBoyCrawlerBoy Pinpoint.com
Crawling jpegMozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp)
CrawllybotCrawllybot/0.1 (Crawllybot; +http://www.crawlly.com; [email protected])
CroccrawlerCrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
CsCrawlerHi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3
Csci_b659/0.13csci_b659/0.13
CuasarCuasarbot/0.9b http://www.cuasar.com/spider_beta/
CurryGuideCurryGuide SiteScan 1.1
CyberAlertsMozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)
CydralCydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)
DAUM RSS RobotELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)
DNS DiggerMozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
DailyOrbitOrbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
DaviesBotDaviesBot/1.7 (www.wholeweb.net)
Daypopdaypopbot/0.x
DbDigdbDig(http://www.prairielandconsulting.com)
De.comMozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)
DeepIndexerDeepIndexer.ca
Deepak USC/ISIdeepak-USC/ISI
Denmex WebsearchDenmex websearch (http://search.denmex.com)
DepSpidMozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net)
Dev spider2dev-spider2.searchpsider.com/1.3b
DiaGem JapanDiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)
Die Kraehe-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de
DiggitDigger/1.0 JDK/1.3.0rc3
Direct HitMozilla/2.0 (compatible; EZResult -- Internet Search Engine)
DittoDittoSpyder
Dodgebotdodgebot/experimental
DoubanbotDoubanbot/1.0 ([email protected] http://www.douban.com)
DrecombotDrecombot/1.0 (http://career.drecom.jp/bot.html)
DtSearchSpiderdtSearchSpider
DuckDuckBotDuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
E SocietyRobote-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
E StyleISPeStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)
EApolloBoteApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at global-opto dot com)
EMPAS_ROBOTEMPAS_ROBOT
ESISmartSpiderESISmartSpider
Echo.comMozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5
Echo.frEchO!/2.0
EgothorMozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/)
EgotobotEgotoBot/4.8 (+http://www.egoto.com/about.htm)
Elfbotelfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)
EmeraldShield.com Web SpiderEmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)
Enfish TrackerEnfish Tracker
Enoolaenoola (http://www.enoola.com)
EroCrawlerEroCrawler
Eruvo boteruvo-bot 4.8.1 (http://www.eruvo.com)
Euro spiderEuro-Spider Shopping 1.0
EvaalEvaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; [email protected])
EvaalSEEvaalSE - [email protected]
Eventaxeventax/1.3 (eventax; http://www.eventax.de/; [email protected])
Everest VulcanEverest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp)
Evrinid RTMozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid)
ExactSeek_SpiderExactSeek_Spider
ExcaliburExcalibur Internet Spider V6.5.4
ExecrawlExecrawl/1.0 (Execrawl; http://www.execrawl.com/; [email protected])
FDSEMozilla/4.0 (compatible; FDSE robot)
FaXobotFaxobot/1.0
FairShareMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
Fast SearchPycURL
Fastbotfastbot crawler beta 2.0 (+http://www.fastbot.de)
FauBotMozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Favo.eu crawlerfavo.eu crawler/0.6 (http://www.favo.eu)
Feed24Feed24.com
FeedCheckerFeedChecker/0.01
Feedfetcher GoogleFeedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Feedfinderfeedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/
FelixFelix - Mixcat Crawler (+http://mixcat.com)
FilangyFilangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; [email protected])
Findexa CrawlerFindexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
FineBotFineBot
FirstGovFirstGov.gov Search - POC:[email protected]
Firstsbotfirstsbot
FlapbotFlapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com)
FlickBotFlickBot 2.0 RPT-HTTPClient/0.3-3
Fluffy the spiderMozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])
FnooleBotFnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)
Folkd.com SpiderFolkd.com Spider/0.1 beta 1 (www.folkd.com)
ForAll.pl CrawlerForAll.pl-Crawler/1.0
FrancisFrancis/1.0 ([email protected] http://www.neomo.de/)
Freshmeatfreshmeat.net URL validator/1.1
FuchsBotFuchsBot +http://www.fuchsbot.tld
FurlBotMozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; [email protected])
FuseBulbFuseBulb.Com
GAIS RobotGAIS Robot/1.0B2
GEXTEST 00393gsa-crawler (Enterprise; GEXTEST-00393; [email protected],[email protected])
GPU p2p crawlerMozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php)
GSiteCrawlerGSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/)
Gaazgazz/x.x ([email protected])
Gamekitbotgamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/)
GammaSpiderGammaSpider/1.0
GeonaBotGeonaBot 1.x; http://www.geona.com/
Georgia Institute of Technologylarbin_2.6.2 (listonATccDOTgatechDOTedu)
GeourlMozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot)
GirafabotMozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
GlobalQueueLook.com
GnodSpiderGNODSPIDER (www.gnod.net)
Gonzo1gonzo1[P] +http://www.suchen.de/popups/faq.jsp
Goo (Japan)Mozilla/3.0 (Slurp.so/Goo; [email protected]; http://www.inktomi.com/slurp.html)
Google Keyword ToolMozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select/KeywordToolExternal)
Google SitemapsGoogle-Sitemaps/1.0
GreatereraMozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
GrigorBotGrigorBot 0.8 (http://www.grigor.biz/bot.html)
GromitGromit/1.0
Grub clientMozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
Gungho crawlerGungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)
Harvest NGHarvest-NG/1.0.2
Hatena AntennaHatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)
Hbtronix.spiderhbtronix.spider.2 -- http://hbtronix.de/spider.php
HeinrichderMiragoRobotHeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp)
HelixHelix/1.x (+http://www.sitesearch.ca/helix/)
HenriLeRobotMiragoHenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp)
HenryTheMiragoRobotHenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)
Heritrix L3SMozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projects/crawling/)
Heritrix/1.4.0Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net)
Hermits SearchMozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)
HiiglespiderHiiglespider/0.1, Hiigle.com, http://hiigle.com/spider
Hitwise SpiderHitwise Spider v1.0 http://www.hitwise.com
HolmesBotHolmesBot (http://holmes.ge)
HomePageSearchHomePageSearch(hpsearch.uni-trier.de)
HomerbotHomerbot: www.homerweb.com
Honda SearchHonda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])
Htdightdig/3.1.x (root@localhost)
Htdig/3.1.6htdig/3.1.6 ([email protected])
I1searchboti1searchbot/2.0 (i1search web crawler; http://www.i1search.com; [email protected])
ICCrawlerICCrawler - ICjobs (http://www.icjobs.de/bot.htm)
ICRA_Label_spiderICRA_label_spider/x.0
IDBotMozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
IIITBOTIIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in)
INGRIDMozilla/3.0 (INGRID/3.0 MT; [email protected]; http://webmaster.ilse.nl/jsp/webmaster.jsp)
IP2MapBotIP2MapBot/1.1 http://www.ip2map.com
IPiumBotIPiumBot laurion(dot)com
IScraperBotIScraperBot/0.1
IWAgentIWAgent/ 1.0 - www.brandprotect.com
Iaskspider2iaskspider2 ([email protected])
Icsboticsbot-0.1
Ideareideare - SignSite/1.x
IlTrovatoreIlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; [email protected])
ImageWalkerImageWalker/2.0 (www.bdbrandprotect.com)
IncyWincy (Look)IncyWincy(http://www.look.com)
IndexTheWebIndexTheWeb.com Crawler7
Indonesia InteractiveMozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive
Indy LibraryMozilla/3.0 (compatible; Indy Library)
InelaBotInelaBot/0.2 (+http://inelegant.org/bot)
Inet LibraryInet library
InfoFlyInfoFly/1.0 (http://www.versions-project.org/)
InfoLab robotMozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/)
InfoSec Search BotRedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; [email protected])
InfoseekInfoSeek Sidewinder/0.9
Inrialarbin_2.2.1_de_Viennot ([email protected])
Insitor Search robotInsitor.com search and find world wide!
InsitornautInsitornaut
Internet NinjaInternet Ninja x.0
InternetseerInternetSeer.com
IprospectMozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56)
IpselonBotIpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; [email protected])
IseekbotiSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot com)
Ishida Lablarbin_2.2.2 ([email protected])
It botIlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.iltrovatore.it/bot.html; [email protected])
Jayde CrawlerJayde Crawler. http://www.jayde.com
Jeaniejeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51)
JetbotJetbot/1.0
Jobs.de RobotMozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; [email protected]) ( newsexpress e-mail: [email protected] http://www.neofonie.de/loesungen/search/robot.html )
JomjaibotJomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com)
Jongaimpijongaimpi/2.10 (jonga; http://www.jonga.co.za; [email protected])
K2 Spiderk2spider
KAIST AITrc CrawlerKAIST AITrc Crawler
KFSW BotKFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de)
KIT_FireballKIT_Fireball/2.0
KS CrawlerKS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; [email protected])
KSbotKSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; [email protected])
KakleBotKakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; [email protected])
KasparekFirefox_1.0.6 ([email protected])
KeegeebotKeegeebot/2.1 (+http://www.keegee.com/keegee/bot.html)
Kenjin SpiderKenjin Spider
KicktooBotkicktooBotV1.1 [email protected]
Kinja imagebotkinja-imagebot (http://www.kinja.com/)
Kinjabotkinjabot (http://www.kinja.com)
KnowItAllKnowItAll([email protected])
Knowledge.comKnowledge.com/0.x
KrugleKrugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; [email protected])
Kulturarwkulturarw3/0.1
KummKummHttp/1.1 (compatible; KummClient; Linux rulez)
Kyluka crawlMozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; [email protected])
LECodeCheckerLECodeChecker/3.0 libgetdoc/1.0
LNSpiderguyLNSpiderguy
Lawinfo crawlerlawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; [email protected])
Lemur Consultinglarbin_2.6.2 ([email protected])
Liafalarbin_2.2.2_guillaume ([email protected])
LibWeblibWeb/clsHTTP -- [email protected]
LijitSpiderLijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com)
Linguee BotLinguee Bot (http://www.linguee.com/bot; [email protected])
LinkWalkerLinkWalker
Linknzbotlinknzbot
Links2GoMozilla/3.01 (Compatible; Links2Go Similarity Engine)
Links4US CrawlerLinks4US-Crawler, (+http://links4us.com/)
LinksManager.com_botMozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)
LlautLlaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)
Lmspiderlmspider ([email protected])
LocalBotLocalBot/1.0 ( http://www.localbot.co.uk/)
Lockstep SpiderLockstep Spider/1.0
Look.comNetResearchServer(http://www.look.com)
LookdirBotLookdirBot
LovelLovel as 1.0 ( +http://www.everatom.com)
Ltaa_web_crawlerlarbin_2.6.3 ([email protected])
Luchs.at URL checkerluchs.at URL checker
MQBOTMQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; [email protected])
MSNBOT_MobileMSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
MSRBOTMSRBOT (http://research.microsoft.com/research/sv/msrbot)
MaSagoolMaSagool/1.0 (MaSagool; http://sagool.jp/; [email protected])
Mail.RuMail.Ru/1.0
Mainseek_BotMozilla/5.0 (compatible;MAINSEEK_BOT)
MantraAgentMantraAgent
MarinerMariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)
MarvinMarvin v0.3
MasterseekMasterSeek
MaxbotSpider/maxbot.com [email protected]
Maxomobotmaxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; [email protected])
MediaCrawlerMediaCrawler-1.0 (Experimental)
MediaSearchMediaSearch/0.1
Mediater Rechercherlibwww/5.3.2
MegaSheepMegaSheep v1.0 (www.searchuk.com internet sheep)
Megaglobe CrawlerMozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com)
Melbot WebSpiderMelbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.)
Merl.comlarbin_2.1.1 [email protected]
MetaGer_PreCheckerMetaGer_PreChecker0.1
Metadata LabsMLBot (www.metadatalabs.com/mlbot)
Metaeuro Web CrawlerMetaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com)
Metager LinkcheckerMetaGer-LinkChecker
MetagerBotMetagerBot/0.8-dev (MetagerBot; http://metager.de; )
MetaspinnerMetaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)
Metatagsdirmetatagsdir/0.7 (+http://metatagsdir.com/directory/)
Microsoft Small Business IndexerMicrosoft Small Business Indexer
MiggibotA1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24
MirarMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com)
Missigua Locator 1.9Missigua Locator 1.9
MisterbotMisterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; [email protected])
MivaMiva ([email protected])
MnoGoSearchMnogoSearch/3.2.xx
Mo College 1.9Mo College 1.9
Mogetmoget/x.x ([email protected])
Mogimogimogimogi/1.0
Moi mêmeno-user-agent-in-crawltrack-for-that-bot
Monogolhttp://www.monogol.de
MoreoverbotMoreoverbot/5.00 (+http://www.moreover.com)
MorrisMorris - Mixcat Crawler (+http://mixcat.com)
MowserbotMozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.com/bot)
MozDexmozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; [email protected])
MozdexMozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; [email protected])
MultiTextMultiText/0.1
MusicWalkerMusicWalker2.0 (+http://www.somusical.com)
MxbotMozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
My bytebotmy-bytebot/1.0 (+http://spider.my-byte.de/info)
MyFamilyBotMozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx; [email protected])
MylineaMylinea.com Crawler 2.0
NASA Search 1.0NASA Search 1.0
NCSANCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)
NIIlarbin_2.6.2 ([email protected])
NLCrawlerMozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)
NMG SpiderNMG Spider/0.3 (szukanko.com)
NTT Directorynttdirectory_robot/0.9 ([email protected])
NWSpiderNWSpider 0.9
NajdiMozilla/5.0 (compatible; InterseekWeb/3.x)
NatzanBotNatzanBot, http://www.natzan.com
Nelian Pty Ltd SpiderNelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com )
NetNoseMozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)
NetResearchServerNetResearchServer/4.0(loopimprovements.com/robot.html)
NetSprint 2.0NetSprint -- 2.0
NetWhatCrawlerNetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; [email protected])
NetinfoBotNetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
NetluchsNetluchs/0.8-dev ( ; http://www.netluchs.de/; ___don'[email protected])
NetprospectorNetprospector JavaCrawler
NetscapeRobozilla/1.0
NjuiceBotMozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
Noago SpiderNoago Spider
NokodoBotNokodoBot/1.x (+http://nokodo.com/bot.htm)
NorbertNorbert the Spider(Burf.com)
Noxtrumbotnoxtrumbot/1.0 ([email protected])
Noyonanoyona_0_1
Nsyghtnsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com)
NuSearch SpidernuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible; MSIE 4.01)
NutchCVS/0.8 devNutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; [email protected])
NutchEC2TestNutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com)
OOZBOTOOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Obidos botobidos-bot (just looking for books.)
Object Sciences Corp.Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E
Objects SearchObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; [email protected])
OctoraOctora Beta - www.octora.com
Octora Beta BotOctora Beta Bot - www.octora.com
OpenISearchOpenISearch/1.x (www.openisearch.com)
OpenPortal4UDigOut4U
OpenTaggerBotOpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm)
OpenTextSiteCrawlerOpenTextSiteCrawler/2.9.2
OpenWebSpiderOpenWebSpider/x
OpidooBotOpidooBOT ([email protected])
OracleOracle Ultra Search
OrangeBot MobileMozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 ([email protected])
OrangeSpiderOrangeSpider
OrbiterOrbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
Overture WebCrawlerOverture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
OzMonitorMozilla/5.0 (Linux; fr) Firefox (ozMonitor Free)
Ozelotozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; [email protected])
PJspiderPJspider/3.0 ([email protected]; http://www.portaljuice.com)
Page storeMozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
PageBitesHyperBotPageBitesHyperBot/600 (http://www.pagebites.com/)
Page_verifierpage_verifier http://www.securecomputing.com/goto/pv
PagebullPagebull http://www.pagebull.com/
PalmeraBotMozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001
PandoraMozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/crawl.html)
ParaSiteParaSite/1.0b (http://www.ianett.com/parasite/)
PatchOneSpiderMozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se)
PatwebbotPatwebbot (http://www.herz-power.de/technik.html)
PeerbotPEERbot www.peerbot.com
Petitsagewww.petitsage.fr site detector 0.4
PicoSearchPicoSearch/1.0
PictureOfInternetPictureOfInternet/3.0 (http://www.malfunction.org/poi; [email protected])
Pilot HitlistMarketwave Hit List
PingdomPingdom GIGRIB v1.1 (http://www.pingdom.com)
PipeLinerpipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
PitaPita
PluckFeedCrawlerPluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers)
PmozMozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
PodtechMozilla/5.0 (compatible; MSIE 6.0; Podtech Network; [email protected])
PogodakMozilla/5.0 (compatible; pogodak.ba/3.x)
Polybotpolybot 1.0 (http://cis.poly.edu/polybot/)
PopJapanSearchRobot/www.pj-search.com
PopdexterPopdexter/1.0
PrivacyFinder/1.1PrivacyFinder/1.1
ProWebguideProWebGuide Link Checker (http://www.prowebguide.com)
Probe!PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedia.com)
Psbotpsbot/0.1 (+http://www.picsearch.com/bot.html)
Psycheclonepsycheclone
Pubblisito[email protected] (http://www.pubblisito.com) il Sud dei Motori di Ricerca
Pumpkinblogsearchbot-pumpkin-3
PurebotMozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)
Python urllibPython-urllib/1.16
QPCreepQPCreep Test Rig ( We are not indexing, just testing )
QihooBotMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 [email protected])
QuantcastbotMozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com)
Qubeqube/2.0 (+http://qube.qelix.com/v2)
QueryN MetasearchQueryN Metasearch
R6_CommentReaderR6_CommentReader_(www.radian6.com/crawler)
RSS One EngineRSS One Engine/0.72 (+http://www.rss-one.com)
RambotramBot xtreme x.x
RankMeterWeb Snooper
Rational SiteCheckRational SiteCheck (Windows NT)
RedBotRedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
RedCarpet/1.2RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)
RedCellRedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net)
RedKernelRedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)
RoboPalRoboPal (http://www.findpal.com/)
RotondoRotondo/3.1 libwww/5.3.1
RufusBotRufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)
RutterBotRutterBot(+http://www.aktienbetreuer.de/bot.html)
S2BotS2Bot/1.0 (http://search2.net)
SAITsait/Nutch-0.9 (SAIT Research; http://www.samsung.com)
SandCrawlerSandCrawler - Compatibility Testing
Savvybotsavvybot/0.2
ScanWebScanWeb
Schibstedsokbotschibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http://www.schibstedsok.no/bot/)
ScholarUniverseScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; [email protected])
Scirus crawlerFAST Enterprise Crawler 6 / Scirus [email protected]; http://www.scirus.com/srsapp/contactus/
ScorpionBotFooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots
ScoutAntScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
Sdcresearchlabs testbotsdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache.org/nutch/bot.html; [email protected])
Search Engine StudioSearch-Engine-Studio
Search InfoSpider-Sleek/2.0 (+http://search-info.com/linktous.html)
Search17BotMozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
Search4freelwp-trivial/1.34
SearchByUsaSearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; [email protected])
SearchEngineWorlds( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
SearchExpress SpiderSearchExpress Spider0.99
SearchMonkeyMozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/searchmonkey/useragent)
SearchSightSearchSight/2.0 (http://SearchSight.com/)
SearchdayBotSearchdayBot
Searchdnabotsearchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at searchenginedna dot com)
SearchguildSearchGuild_DMOZ_Experiment ([email protected])
Searchit Now RobotSearchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)
Searchmee! SpiderSearchmee! Spider v0.98a
Seeker.lookseekSeeker.lookseek.com
Sensis.com.au Web CrawlerSensis.com.au Web Crawler (search_commentsatsensisdotcomdotau)
SharewarePlazaAgent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)
SheenBotSheenBot/SheenBot-1.0.4 (Sheen web crawler)
Sherlocksherlock/1.0
ShoulaShoula.com Crawler 2.0
SietsCrawlerSietsCrawler/1.1 (+http://www.siets.biz)
Siigle OrumcexSiigle Orumcex v.001 Turkey (http://www.siigle.com)
SirketcebotSirketcebot/v.01 (http://www.sirketce.com/bot.html)
Site Server 3.0 RobotMozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR
SiteBarSiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
SiteBaseBotMozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru)
SiteSpiderSiteSpider +(http://www.SiteSpider.com/)
SiteTruthSiteTruth.com site rating system
SitiDiBot+SitiDi.net/SitiDiBot/1.0 (+Have Good Day)
SkampyMozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en]
SkizzleUser-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)
SkreemRBotMozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
SliderSlider_Search_v1-de
Slurp (Yahoo)Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Slurp China (Yahoo)Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Slurpy VerifierSlurpy Verifier/1.0
SmiffyDCMetaSpiderSmiffyDCMetaSpider/1.0
SnapPreviewMozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
SnapbotSnapbot/1.0
SnoopySnoopy v1.2
SnykeBotSnykeBot/0.6 (http://www.snyke.com)
SoftHypermarketSoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com)
Sohu searchsohu-search
SomewhereMozilla ([email protected])
SosospiderSosospider+(+http://help.soso.com/webspider.htm)
Spam BotMozilla/2.0 (compatible; NEWT ActiveX; Win32)
SpbotMozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Speedfindspeedfind ramBot xtreme 8.1
Sphere ScoutSphere Scout&v4.0 - scout at sphere dot com
SphsearchFAST Enterprise Crawler 6 used by Singapore Press Holdings ([email protected])
Spinn3rMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
SplatSearchlibwww-perl/5.45
Spock CrawlerSpock Crawler (http://www.spock.com/crawler)
Strategic Board BotStrategic Board Bot (+http://www.strategicboard.com)
Suchpadsuchpad/1.0 (+http://www.suchpad.de)
SummizeBotMozilla/5.0 (compatible; SummizeBot +http://www.summize.com)
SuperSnooper[email protected]
SurveyBotSurveyBot/2.3 (Whois Source)
Susie!Susie (http://www.sync2it.com/susie)
SwooglebotSwooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html)
SycaBoTSycaBoT/1.0
SycaBoT AudioSycaBoT-Audio
SycaBoT ImageSycaBoT-Image
SycaBoT ProgrammeSycaBoT-Programme
SycaBoT VideoSycaBoT-Video
SycrawlNutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; [email protected])
Sygol<http://www.sygol.com/> http://www.sygol.com
SyntryxSyntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler
TAGwordTagword (http://tagword.com/dmoz_survey.php)
TCDBOTTCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot IE)"
Tags2dirtags2dir.com/0.8 (+http://tags2dir.com/directory/)
TailRank RobotMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://tailrank.com/robot) Gecko/20021130
TalkroTalkro Web-Shot/1.0 (E-mail: [email protected], Home: http://222.122.15.190/webshot)
TargetSeekMozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html)
TasapspiderTasapspider/0.9 (http://www.tasap.com)
TechnoratibotTechnoratibot/0.7
Tecomi BotTecomi Bot (http://www.tecomi.com/bot.htm)
TeemerTeemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; [email protected])
TelenetDiggerTelenetDigger/1.0; htdig/3.1.6 ([email protected])
Teradex MapperTeradex Mapper; [email protected]; http://www.teradex.com
TeragramCrawlerTeragramCrawler
TeragramWebcrawlerTeragramWebcrawler/1.0
TerrawizBotTerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)
TheSuBotTheSuBot/0.1 (www.thesubot.de)
Theme SpiderTheme Spider (+http://www.themespider.com/spider.html)
TheophrastusMozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php)
Thumbshots de botthumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)
Thunderstones WebinatorMozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)
TimboBottimboBot/0.9 http://www.breakingblogs.com/timbo_bot.html
TinEyeTinEye/1.1 (http://tineye.com/crawler.html)
TivraSpidertivraSpider/1.0 ([email protected])
TkensakuTkensaku/x.x(http://www.tkensaku.com/q.html)
ToileBotMozilla/5.0 (+http://www.toile.com/) ToileBot/0.1
TopodiaTopodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; [email protected])
Traazibottraazibot/testengine (+http://www.traazi.de)
TrendTechhttp://www.trendtech.dk/spider.asp)
TruveoMozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/)
TurnitinTurnitinBot
TurnitinBotTurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)
Turnpike EmporiumTurnpike Emporium LinkChecker/0.1
TazillaMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html)
UN OCHAFAST Enterprise Crawler 6 used by [email protected] (UN OCHA)
UOLCrawlerUOLCrawler ([email protected])
URL SpiderMozilla/5.0 URL-Spider
USyd NLP SpiderUSyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
UltraseekUltraseek
Uni koblenzhttp://www.uni-koblenz.de/~flocke/robot-info.txt
Unido botunido-bot, http://mobicom.cs.uni-dortmund.de/bot.html
Updatedupdated/0.1beta (updated.com; http://www.updated.com; [email protected])
Updated crawlerupdated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])
Urlchecker1.0Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)
Urlfan boturlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
VBSEOMozilla/4.0 (vBSEO; http://www.vbseo.com)
VIProbotVIPr/0.4 (VIProbot; http://www.vipsolutions.hu; [email protected])
VWBOTVWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;[email protected]
VacobotVacobot; (+http://vaco.ws/bot.html)
Vagabondo WAPVagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile
VakesVakes/0.01 (Vakes; http://www.vakes.com/; [email protected])
Versusversus 0.2 (+http://versus.integis.ch)
Versus Crawlerversus crawler [email protected]
VeryGoodSearchVeryGoodSearch.com.DaddyLongLegs
Vespa Crawler (Yahoo)Vespa Crawler
VladMozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html)
Vonna.com b o tMozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t)
Vortex/2.2Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
W3C ValidatorW3C_Validator/1.432.2.10
W3C checklinkW3C-checklink/4.2 [4.20] libwww-perl/5.803
W3SiteSearch CrawlerW3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de
W8netMozilla/5.0 usww.com-Spider-for-w8.net
WEP SearchWEP Search 00
WISEbotWISEbot/1.0 ([email protected]; http://wisebot.koreawisenut.com)
WWW.fi crawlerwww.fi crawler, contact [email protected]
WWWeasel RobotWWWeasel Robot v1.00 (http://wwweasel.de)
Wadaino.jp crawlerwadaino.jp-crawler 0.2 (http://wadaino.jp/)
WavefireWavefire/0.8-dev (Wavefire; http://www.wavefire.com; [email protected])
WeRelateBotWeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; [email protected])
WeatherBotWeatherBot v1.4 http://www.ezweather.net
WebBOTCarnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot/index.html
WebFindBotWebFindBot(http://www.web-find.com)
WebGobblerwebGobbler/1.x.x
WebRankSpiderWebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebSearchWebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; [email protected])
WebStatWebStat/1.0 (Unix; beta; 20040314)
WebVacWebVac ([email protected])
Webbanditwebbandit/4.xx.0
WebclippingWebclipping.com
WebduniabotMozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx)
WebglimpseWebglimpse 2.xx.x (http://webglimpse.net)
Weblog Attitude DiffusionWeblog Attitude Diffusion 1.0
Webmeasurement botwebmeasurement-bot, http://rvs.informatik.uni-leipzig.de
Webmeasurement crawlerwebmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/
WebsiteWorthWebsiteWorth v1.0
WebspinneWebspinne/1.0 [email protected]
WebsquashWebsquash.com (Add url robot)
WebsterWebster v0.3 ( http://webster.healeys.net/ )
WebverzeichnisWebverzeichnis.de - Telefon: 01908 / 26005
WhoiamMozilla/5.0 whoiam [http://www.axxus.de/]
Wikiwix botwikiwix-bot-3.0
WillowWillow Internet Crawler by Twotrees V2.1
WinMEMozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en]
WinkBotWinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; [email protected])
Worio heritrix botworio heritrix bot (+http://worio.com/)
WorldWideWeb X/3.1WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/)
Wume_crawlerwume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)
Wwlib/LinuxWwlib/Linux
Wwwsterwwwster/1.4 (Beta, mailto:[email protected])
XP5Project XP5 [2.03.07-111203]
Xdefineegothor/3.0a (+http://www.xdefine.org/robot.html)
Xirqxirq/0.1-beta (xirq; http://www.xirq.com; [email protected])
Yacyyacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en)
Yahoo Search Japan robotY!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yahoo! MindsetYahoo! Mindset
Yahoo BlogsYahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
Yahoo MMAudVidYahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo Test/4.0Yahoo-Test/4.0
Yahoo VerticalCrawlerYahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/
YahooSeekerYahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;[email protected] ) (KHTML, like Gecko)
YandexBotMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Yarienavoiryarienavoir.net/0.2
Yellopetspider.yellopet.com - www.yellopet.com
YesupBotMozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Yggdrasilyggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de)
YooW!YooW!/1.8.9 RC1 (+http://www.yoow.eu)
YoogliyoogliFetchAgent/0.1
YoonoMozilla/5.0 (compatible; Yoono; http://www.yoono.com/)
Yoono web crawleryoono/1.0 web-crawler/1.0
YottaCarsYottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine
YottaShoppingYottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine
Z Add Link CheckerZ-Add Link Checker (http://w3.z-add.co.uk/linkcheck/)
ZealbotMozilla/4.0 (compatible; Zealbot 1.0)
ZearchitZearchit
Zedzo.digestzedzo.digest/0.1 (http://www.zedzo.com/)
Zerxbotzerxbot/Version 0.6 libwww-perl/5.79
ZetbotDeepIndex ( http://www.zetbot.com )
ZippyZippy v2.0 - Zippyfinder.com
Zspiderzspider/0.9-dev http://feedback.redkolibri.com/
ZyBorg (LookSmart)ZyBorg
aiHitBotMozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
eCairn GrabbereCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15

Looking for the addresses used by the main search engines? The crawler address page lists them. To test one address right now, use the address lookup.

Technical identifiers in this table, including the user agent strings, are reproduced without alteration so that they keep matching real traffic.

Reference shelf

Sources and further reading, all outside this site. They open in a new context and are listed for reference only.