691 entries are shown below, exactly as the detection database holds them. The list is refreshed from the tools page of your installation, and you can add your own crawler by user agent or by address range.
| Crawler | User agent |
|---|---|
| 192.com | 192.comAgent |
| 4anything | 4anything.com LinkChecker v2.0 |
| 80legs | Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 |
| A Online | A-Online Search |
| ABCdatos | ABCdatos BotLink/5.xx.xxx#BBL |
| AOL | Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) |
| ASAHA | ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) |
| AVSearch | AVSearch-1.0([email protected]) |
| Abby | Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html) |
| Aberja Checkoma | Aberja Checkoma |
| Abot | abot/0.1 (abot; http://www.abot.com; [email protected]) |
| About | About/0.1libwww-perl/5.47 |
| AboutUsBot | Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) |
| Acoi | AcoiRobot |
| Acoon Robot | Acoon Robot v1.50.001 |
| Acoon Robot | Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) |
| Acorn | Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) |
| Activtourist | Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) |
| AdsBot Google | AdsBot-Google (+http://www.google.com/adsbot.html) |
| Aesop | AESOP_com_SpiderMan |
| AgentName | AgentName/0.1 libwww-perl/5.48 |
| Aibot | AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China) |
| Aicrawler | Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) |
| Aipbot | aipbot/1.0 (aipbot; http://www.aipbot.com; [email protected]) |
| Alacra | PortalBSpider/2.0 ([email protected]) |
| Aladin.de | Aladin/3.324 |
| Aleksika Danmark | Aleksika Spider/1.0 (+http://www.aleksika.com/) |
| Alexa | ia_archiver |
| Allesklar.de | Allesklar/0.1 libwww-perl/5.46 |
| Allrati | Allrati/1.1 (+) |
| Amfibibot | Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; [email protected]) |
| Amidalla | libwww-perl/5.65 |
| Annomille | AnnoMille spider 0.1 alpha - http://www.annomille.it |
| AnsearchBot | Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) |
| AnswerBus | AnswerBus (http://www.answerbus.com/) |
| Answerchase | PROve AnswerBot 4.0 |
| Antibot | antibot-V1.3.3.1/debian-linux-sarge |
| Any Search Info | Mozilla/4.0 (Sleek Spider/1.2) |
| Anzwers Australia | AnzwersCrawl/2.0 ([email protected];Engine) |
| Apexoo Spider | Apexoo Spider 1.0 |
| ApocalXExplorerBot | ApocalXExplorerBot |
| Aport | Aport |
| Arabulbot | Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) |
| ArabyBot | ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) |
| Arachnoidea | Arachnoidea ([email protected]) |
| ArchitextSpider | ArchitextSpider |
| Archive.org_bot | Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org) |
| Arianna | www.arianna.it |
| Arikus_Spider | Arikus_Spider |
| Ask 24x Info | ask.24x.info |
| Asked | asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) |
| Askpeter_bot | Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) |
| Asterias | asterias/2.0 |
| Astrafind! | Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) |
| Atlocal | AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) |
| Attentio | Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; [email protected]) |
| Axada | axadine/ (Axadine Crawler; http://www.axada.de/; ) |
| Axandra | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) |
| Axmo | AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine. |
| Ay Up | FastBug http://www.ay-up.com |
| BE Internet Search Engine | Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) |
| Ba.be | Mozilla/4.72 [en] (BACS http://www.ba.be) |
| BaBoom Web Portal | BaboomBot/1.x.x (+http://www.baboom.us) |
| BabalooSpider | BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; [email protected]) |
| Backlink Check | Backlink-Check.de (+http://www.backlink-check.de/bot.html) |
| BanBot | BanBots/1.2 ([email protected]) |
| BeamMachine | BeamMachine/0.5 (dead link remover of www.beammachine.net) |
| Beauty (Cosmoty) | beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) |
| BebopBot | BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) |
| BecomeJPBot | Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html) |
| BeijingCrawler | BeijingCrawler |
| BigClique | BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; [email protected]) |
| Biglotron | BIGLOTRON (Beta 2;GNU/Linux) |
| BilBasen | FAST Enterprise Crawler 6 used by BilBasen ApS ([email protected]) |
| BilgiBetaBot | BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; [email protected]) |
| BilgiBot | BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) |
| BinGet | BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/) |
| Bisnisseek | Custom Spider www.bisnisseek.com /1.0 |
| Bitlybot | bitlybot |
| Blaiz Enterprises | Blaiz-Bee/1.0 (+http://www.blaiz.net) |
| BlogRefsBot | Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers) |
| BlogScope | Mozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto) |
| BlogWatcher | blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) |
| Blogdex | BlogBot/1.x |
| Blogdimension BlogBot | Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com) |
| Bloglines Images | Bloglines-Images/0.1 (http://www.bloglines.com) |
| BnF bot | Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/) |
| Bot | bot/1.0 (bot; http://; [email protected]) |
| BotSeer | Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) |
| Botmobi | Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html [email protected]) |
| Brandwatch | magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) |
| BrightCrawler | BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) |
| Bruinbot | BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) |
| Btbot | BTbot/0.x (+http://www.btbot.com/btbot.html) |
| BuildCMS crawler | BuildCMS crawler (http://www.buildcms.com/crawler) |
| BuiltWith | Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) |
| BullsEye/Intelliseek | BullsEye |
| BurstFindCrawler | BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; [email protected]) |
| Buscaplus | Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) |
| Butterfly | Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8 |
| CCBot | CCBot/1.0 (+http://www.commoncrawl.org/bot.html) |
| CEA | larbin_2.6_basileocaml ([email protected]) |
| CMP | libwww-perl/5.41 |
| CUPS | PrivacyFinder Cache Bot v1.0 |
| Camcrawler | Camcrawler (+http://www.camdiscover.com/crawler.html) |
| CamontSpider | CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html |
| Carleson | carleson/1.0 |
| Catall Spider | Catall Spider |
| Catall Spider | Catall-Spider/3.3.3(www.Catall.de) |
| Ccubee | ccubee/x.0 |
| Changedetection | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html ) |
| Chen Li | Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; [email protected]) |
| Cityreview Robot | Cityreview Robot (+http://www.cityreview.org/crawler/) |
| CjLogbot | Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) |
| Claymont Search | Claymont.com |
| CloakDetect | CloakDetect/0.9 (+http://fulltext.seznam.cz/) |
| Cnet robot | Mozilla/4.6 [en] (http://www.cnet.com/) |
| CoBITSProbe | CoBITSProbe |
| Cometrics bot | cometrics-bot, http://www.cometrics.de |
| Comperio | FAST Enterprise Crawler 6 used by Comperio AS ([email protected]) |
| Compete.com | larbin_2.2.0 ([email protected]) |
| Computerorgs | htdig/3.1.6 (http://computerorgs.com) |
| Comrite | Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected]) |
| CoolBot | CoolBot |
| Copyright sheriff | copyright sheriff (+http://www.copyrightsheriff.com/) |
| Cortina | Vision Research Lab image spider at vision.ece.ucsb.edu |
| CougarSearch | CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) |
| CrawlerBoy | CrawlerBoy Pinpoint.com |
| Crawling jpeg | Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) |
| Crawllybot | Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; [email protected]) |
| Croccrawler | CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) |
| CsCrawler | Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3 |
| Csci_b659/0.13 | csci_b659/0.13 |
| Cuasar | Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ |
| CurryGuide | CurryGuide SiteScan 1.1 |
| CyberAlerts | Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) |
| Cydral | CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) |
| DAUM RSS Robot | ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html) |
| DNS Digger | Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) |
| DailyOrbit | Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) |
| DaviesBot | DaviesBot/1.7 (www.wholeweb.net) |
| Daypop | daypopbot/0.x |
| DbDig | dbDig(http://www.prairielandconsulting.com) |
| De.com | Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) |
| DeepIndexer | DeepIndexer.ca |
| Deepak USC/ISI | deepak-USC/ISI |
| Denmex Websearch | Denmex websearch (http://search.denmex.com) |
| DepSpid | Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) |
| Dev spider2 | dev-spider2.searchpsider.com/1.3b |
| DiaGem Japan | DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) |
| Die Kraehe | -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de |
| Diggit | Digger/1.0 JDK/1.3.0rc3 |
| Direct Hit | Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) |
| Ditto | DittoSpyder |
| Dodgebot | dodgebot/experimental |
| Doubanbot | Doubanbot/1.0 ([email protected] http://www.douban.com) |
| Drecombot | Drecombot/1.0 (http://career.drecom.jp/bot.html) |
| DtSearchSpider | dtSearchSpider |
| DuckDuckBot | DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) |
| E SocietyRobot | e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) |
| E StyleISP | eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) |
| EApolloBot | eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at global-opto dot com) |
| EMPAS_ROBOT | EMPAS_ROBOT |
| ESISmartSpider | ESISmartSpider |
| Echo.com | Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 |
| Echo.fr | EchO!/2.0 |
| Egothor | Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) |
| Egotobot | EgotoBot/4.8 (+http://www.egoto.com/about.htm) |
| Elfbot | elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) |
| EmeraldShield.com Web Spider | EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) |
| Enfish Tracker | Enfish Tracker |
| Enoola | enoola (http://www.enoola.com) |
| EroCrawler | EroCrawler |
| Eruvo bot | eruvo-bot 4.8.1 (http://www.eruvo.com) |
| Euro spider | Euro-Spider Shopping 1.0 |
| Evaal | Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; [email protected]) |
| EvaalSE | EvaalSE - [email protected] |
| Eventax | eventax/1.3 (eventax; http://www.eventax.de/; [email protected]) |
| Everest Vulcan | Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp) |
| Evrinid RT | Mozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid) |
| ExactSeek_Spider | ExactSeek_Spider |
| Excalibur | Excalibur Internet Spider V6.5.4 |
| Execrawl | Execrawl/1.0 (Execrawl; http://www.execrawl.com/; [email protected]) |
| FDSE | Mozilla/4.0 (compatible; FDSE robot) |
| FaXobot | Faxobot/1.0 |
| FairShare | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc) |
| Fast Search | PycURL |
| Fastbot | fastbot crawler beta 2.0 (+http://www.fastbot.de) |
| FauBot | Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/) |
| Favo.eu crawler | favo.eu crawler/0.6 (http://www.favo.eu) |
| Feed24 | Feed24.com |
| FeedChecker | FeedChecker/0.01 |
| Feedfetcher Google | Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) |
| Feedfinder | feedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/ |
| Felix | Felix - Mixcat Crawler (+http://mixcat.com) |
| Filangy | Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; [email protected]) |
| Findexa Crawler | Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) |
| FineBot | FineBot |
| FirstGov | FirstGov.gov Search - POC:[email protected] |
| Firstsbot | firstsbot |
| Flapbot | Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com) |
| FlickBot | FlickBot 2.0 RPT-HTTPClient/0.3-3 |
| Fluffy the spider | Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected]) |
| FnooleBot | FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) |
| Folkd.com Spider | Folkd.com Spider/0.1 beta 1 (www.folkd.com) |
| ForAll.pl Crawler | ForAll.pl-Crawler/1.0 |
| Francis | Francis/1.0 ([email protected] http://www.neomo.de/) |
| Freshmeat | freshmeat.net URL validator/1.1 |
| FuchsBot | FuchsBot +http://www.fuchsbot.tld |
| FurlBot | Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; [email protected]) |
| FuseBulb | FuseBulb.Com |
| GAIS Robot | GAIS Robot/1.0B2 |
| GEXTEST 00393 | gsa-crawler (Enterprise; GEXTEST-00393; [email protected],[email protected]) |
| GPU p2p crawler | Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php) |
| GSiteCrawler | GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) |
| Gaaz | gazz/x.x ([email protected]) |
| Gamekitbot | gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) |
| GammaSpider | GammaSpider/1.0 |
| GeonaBot | GeonaBot 1.x; http://www.geona.com/ |
| Georgia Institute of Technology | larbin_2.6.2 (listonATccDOTgatechDOTedu) |
| Geourl | Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot) |
| Girafabot | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com) |
| GlobalQueue | Look.com |
| GnodSpider | GNODSPIDER (www.gnod.net) |
| Gonzo1 | gonzo1[P] +http://www.suchen.de/popups/faq.jsp |
| Goo (Japan) | Mozilla/3.0 (Slurp.so/Goo; [email protected]; http://www.inktomi.com/slurp.html) |
| Google Keyword Tool | Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select/KeywordToolExternal) |
| Google Sitemaps | Google-Sitemaps/1.0 |
| Greaterera | Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) |
| GrigorBot | GrigorBot 0.8 (http://www.grigor.biz/bot.html) |
| Gromit | Gromit/1.0 |
| Grub client | Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) |
| Gungho crawler | Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) |
| Harvest NG | Harvest-NG/1.0.2 |
| Hatena Antenna | Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) |
| Hbtronix.spider | hbtronix.spider.2 -- http://hbtronix.de/spider.php |
| HeinrichderMiragoRobot | HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) |
| Helix | Helix/1.x (+http://www.sitesearch.ca/helix/) |
| HenriLeRobotMirago | HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) |
| HenryTheMiragoRobot | HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) |
| Heritrix L3S | Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projects/crawling/) |
| Heritrix/1.4.0 | Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) |
| Hermits Search | Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) |
| Hiiglespider | Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider |
| Hitwise Spider | Hitwise Spider v1.0 http://www.hitwise.com |
| HolmesBot | HolmesBot (http://holmes.ge) |
| HomePageSearch | HomePageSearch(hpsearch.uni-trier.de) |
| Homerbot | Homerbot: www.homerweb.com |
| Honda Search | Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected]) |
| Htdig | htdig/3.1.x (root@localhost) |
| Htdig/3.1.6 | htdig/3.1.6 ([email protected]) |
| I1searchbot | i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; [email protected]) |
| ICCrawler | ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) |
| ICRA_Label_spider | ICRA_label_spider/x.0 |
| IDBot | Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) |
| IIITBOT | IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) |
| INGRID | Mozilla/3.0 (INGRID/3.0 MT; [email protected]; http://webmaster.ilse.nl/jsp/webmaster.jsp) |
| IP2MapBot | IP2MapBot/1.1 http://www.ip2map.com |
| IPiumBot | IPiumBot laurion(dot)com |
| IScraperBot | IScraperBot/0.1 |
| IWAgent | IWAgent/ 1.0 - www.brandprotect.com |
| Iaskspider2 | iaskspider2 ([email protected]) |
| Icsbot | icsbot-0.1 |
| Ideare | ideare - SignSite/1.x |
| IlTrovatore | IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; [email protected]) |
| ImageWalker | ImageWalker/2.0 (www.bdbrandprotect.com) |
| IncyWincy (Look) | IncyWincy(http://www.look.com) |
| IndexTheWeb | IndexTheWeb.com Crawler7 |
| Indonesia Interactive | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive |
| Indy Library | Mozilla/3.0 (compatible; Indy Library) |
| InelaBot | InelaBot/0.2 (+http://inelegant.org/bot) |
| Inet Library | Inet library |
| InfoFly | InfoFly/1.0 (http://www.versions-project.org/) |
| InfoLab robot | Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) |
| InfoSec Search Bot | RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; [email protected]) |
| Infoseek | InfoSeek Sidewinder/0.9 |
| Inria | larbin_2.2.1_de_Viennot ([email protected]) |
| Insitor Search robot | Insitor.com search and find world wide! |
| Insitornaut | Insitornaut |
| Internet Ninja | Internet Ninja x.0 |
| Internetseer | InternetSeer.com |
| Iprospect | Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) |
| IpselonBot | IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; [email protected]) |
| Iseekbot | iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot com) |
| Ishida Lab | larbin_2.2.2 ([email protected]) |
| It bot | IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.iltrovatore.it/bot.html; [email protected]) |
| Jayde Crawler | Jayde Crawler. http://www.jayde.com |
| Jeanie | jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) |
| Jetbot | Jetbot/1.0 |
| Jobs.de Robot | Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; [email protected]) ( newsexpress e-mail: [email protected] http://www.neofonie.de/loesungen/search/robot.html ) |
| Jomjaibot | Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com) |
| Jongaimpi | jongaimpi/2.10 (jonga; http://www.jonga.co.za; [email protected]) |
| K2 Spider | k2spider |
| KAIST AITrc Crawler | KAIST AITrc Crawler |
| KFSW Bot | KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de) |
| KIT_Fireball | KIT_Fireball/2.0 |
| KS Crawler | KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; [email protected]) |
| KSbot | KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; [email protected]) |
| KakleBot | KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; [email protected]) |
| Kasparek | Firefox_1.0.6 ([email protected]) |
| Keegeebot | Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) |
| Kenjin Spider | Kenjin Spider |
| KicktooBot | kicktooBotV1.1 [email protected] |
| Kinja imagebot | kinja-imagebot (http://www.kinja.com/) |
| Kinjabot | kinjabot (http://www.kinja.com) |
| KnowItAll | KnowItAll([email protected]) |
| Knowledge.com | Knowledge.com/0.x |
| Krugle | Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; [email protected]) |
| Kulturarw | kulturarw3/0.1 |
| Kumm | KummHttp/1.1 (compatible; KummClient; Linux rulez) |
| Kyluka crawl | Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; [email protected]) |
| LECodeChecker | LECodeChecker/3.0 libgetdoc/1.0 |
| LNSpiderguy | LNSpiderguy |
| Lawinfo crawler | lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; [email protected]) |
| Lemur Consulting | larbin_2.6.2 ([email protected]) |
| Liafa | larbin_2.2.2_guillaume ([email protected]) |
| LibWeb | libWeb/clsHTTP -- [email protected] |
| LijitSpider | LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) |
| Linguee Bot | Linguee Bot (http://www.linguee.com/bot; [email protected]) |
| LinkWalker | LinkWalker |
| Linknzbot | linknzbot |
| Links2Go | Mozilla/3.01 (Compatible; Links2Go Similarity Engine) |
| Links4US Crawler | Links4US-Crawler, (+http://links4us.com/) |
| LinksManager.com_bot | Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html) |
| Llaut | Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) |
| Lmspider | lmspider ([email protected]) |
| LocalBot | LocalBot/1.0 ( http://www.localbot.co.uk/) |
| Lockstep Spider | Lockstep Spider/1.0 |
| Look.com | NetResearchServer(http://www.look.com) |
| LookdirBot | LookdirBot |
| Lovel | Lovel as 1.0 ( +http://www.everatom.com) |
| Ltaa_web_crawler | larbin_2.6.3 ([email protected]) |
| Luchs.at URL checker | luchs.at URL checker |
| MQBOT | MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; [email protected]) |
| MSNBOT_Mobile | MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) |
| MSRBOT | MSRBOT (http://research.microsoft.com/research/sv/msrbot) |
| MaSagool | MaSagool/1.0 (MaSagool; http://sagool.jp/; [email protected]) |
| Mail.Ru | Mail.Ru/1.0 |
| Mainseek_Bot | Mozilla/5.0 (compatible;MAINSEEK_BOT) |
| MantraAgent | MantraAgent |
| Mariner | Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) |
| Marvin | Marvin v0.3 |
| Masterseek | MasterSeek |
| Maxbot | Spider/maxbot.com [email protected] |
| Maxomobot | maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; [email protected]) |
| MediaCrawler | MediaCrawler-1.0 (Experimental) |
| MediaSearch | MediaSearch/0.1 |
| Mediater Rechercher | libwww/5.3.2 |
| MegaSheep | MegaSheep v1.0 (www.searchuk.com internet sheep) |
| Megaglobe Crawler | Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) |
| Melbot WebSpider | Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.) |
| Merl.com | larbin_2.1.1 [email protected] |
| MetaGer_PreChecker | MetaGer_PreChecker0.1 |
| Metadata Labs | MLBot (www.metadatalabs.com/mlbot) |
| Metaeuro Web Crawler | Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com) |
| Metager Linkchecker | MetaGer-LinkChecker |
| MetagerBot | MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) |
| Metaspinner | Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/) |
| Metatagsdir | metatagsdir/0.7 (+http://metatagsdir.com/directory/) |
| Microsoft Small Business Indexer | Microsoft Small Business Indexer |
| Miggibot | A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 |
| Mirar | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com) |
| Missigua Locator 1.9 | Missigua Locator 1.9 |
| Misterbot | Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; [email protected]) |
| Miva | Miva ([email protected]) |
| MnoGoSearch | MnogoSearch/3.2.xx |
| Mo College 1.9 | Mo College 1.9 |
| Moget | moget/x.x ([email protected]) |
| Mogimogi | mogimogi/1.0 |
| Moi même | no-user-agent-in-crawltrack-for-that-bot |
| Monogol | http://www.monogol.de |
| Moreoverbot | Moreoverbot/5.00 (+http://www.moreover.com) |
| Morris | Morris - Mixcat Crawler (+http://mixcat.com) |
| Mowserbot | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.com/bot) |
| MozDex | mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; [email protected]) |
| Mozdex | Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; [email protected]) |
| MultiText | MultiText/0.1 |
| MusicWalker | MusicWalker2.0 (+http://www.somusical.com) |
| Mxbot | Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html) |
| My bytebot | my-bytebot/1.0 (+http://spider.my-byte.de/info) |
| MyFamilyBot | Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx; [email protected]) |
| Mylinea | Mylinea.com Crawler 2.0 |
| NASA Search 1.0 | NASA Search 1.0 |
| NCSA | NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) |
| NII | larbin_2.6.2 ([email protected]) |
| NLCrawler | Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko) |
| NMG Spider | NMG Spider/0.3 (szukanko.com) |
| NTT Directory | nttdirectory_robot/0.9 ([email protected]) |
| NWSpider | NWSpider 0.9 |
| Najdi | Mozilla/5.0 (compatible; InterseekWeb/3.x) |
| NatzanBot | NatzanBot, http://www.natzan.com |
| Nelian Pty Ltd Spider | Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com ) |
| NetNose | Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) |
| NetResearchServer | NetResearchServer/4.0(loopimprovements.com/robot.html) |
| NetSprint 2.0 | NetSprint -- 2.0 |
| NetWhatCrawler | NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; [email protected]) |
| NetinfoBot | NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) |
| Netluchs | Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don'[email protected]) |
| Netprospector | Netprospector JavaCrawler |
| Netscape | Robozilla/1.0 |
| NjuiceBot | Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot |
| Noago Spider | Noago Spider |
| NokodoBot | NokodoBot/1.x (+http://nokodo.com/bot.htm) |
| Norbert | Norbert the Spider(Burf.com) |
| Noxtrumbot | noxtrumbot/1.0 ([email protected]) |
| Noyona | noyona_0_1 |
| Nsyght | nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) |
| NuSearch Spider | nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible; MSIE 4.01) |
| NutchCVS/0.8 dev | NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; [email protected]) |
| NutchEC2Test | NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) |
| OOZBOT | OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) |
| Obidos bot | obidos-bot (just looking for books.) |
| Object Sciences Corp. | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E |
| Objects Search | ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; [email protected]) |
| Octora | Octora Beta - www.octora.com |
| Octora Beta Bot | Octora Beta Bot - www.octora.com |
| OpenISearch | OpenISearch/1.x (www.openisearch.com) |
| OpenPortal4U | DigOut4U |
| OpenTaggerBot | OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) |
| OpenTextSiteCrawler | OpenTextSiteCrawler/2.9.2 |
| OpenWebSpider | OpenWebSpider/x |
| OpidooBot | OpidooBOT ([email protected]) |
| Oracle | Oracle Ultra Search |
| OrangeBot Mobile | Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 ([email protected]) |
| OrangeSpider | OrangeSpider |
| Orbiter | Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) |
| Overture WebCrawler | Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) |
| OzMonitor | Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) |
| Ozelot | ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; [email protected]) |
| PJspider | PJspider/3.0 ([email protected]; http://www.portaljuice.com) |
| Page store | Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) |
| PageBitesHyperBot | PageBitesHyperBot/600 (http://www.pagebites.com/) |
| Page_verifier | page_verifier http://www.securecomputing.com/goto/pv |
| Pagebull | Pagebull http://www.pagebull.com/ |
| PalmeraBot | Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001 |
| Pandora | Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/crawl.html) |
| ParaSite | ParaSite/1.0b (http://www.ianett.com/parasite/) |
| PatchOneSpider | Mozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se) |
| Patwebbot | Patwebbot (http://www.herz-power.de/technik.html) |
| Peerbot | PEERbot www.peerbot.com |
| Petitsage | www.petitsage.fr site detector 0.4 |
| PicoSearch | PicoSearch/1.0 |
| PictureOfInternet | PictureOfInternet/3.0 (http://www.malfunction.org/poi; [email protected]) |
| Pilot Hitlist | Marketwave Hit List |
| Pingdom | Pingdom GIGRIB v1.1 (http://www.pingdom.com) |
| PipeLiner | pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) |
| Pita | Pita |
| PluckFeedCrawler | PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) |
| Pmoz | Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) |
| Podtech | Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; [email protected]) |
| Pogodak | Mozilla/5.0 (compatible; pogodak.ba/3.x) |
| Polybot | polybot 1.0 (http://cis.poly.edu/polybot/) |
| PopJapanSearch | Robot/www.pj-search.com |
| Popdexter | Popdexter/1.0 |
| PrivacyFinder/1.1 | PrivacyFinder/1.1 |
| ProWebguide | ProWebGuide Link Checker (http://www.prowebguide.com) |
| Probe! | PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedia.com) |
| Psbot | psbot/0.1 (+http://www.picsearch.com/bot.html) |
| Psycheclone | psycheclone |
| Pubblisito | [email protected] (http://www.pubblisito.com) il Sud dei Motori di Ricerca |
| Pumpkin | blogsearchbot-pumpkin-3 |
| Purebot | Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/) |
| Python urllib | Python-urllib/1.16 |
| QPCreep | QPCreep Test Rig ( We are not indexing, just testing ) |
| QihooBot | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 [email protected]) |
| Quantcastbot | Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) |
| Qube | qube/2.0 (+http://qube.qelix.com/v2) |
| QueryN Metasearch | QueryN Metasearch |
| R6_CommentReader | R6_CommentReader_(www.radian6.com/crawler) |
| RSS One Engine | RSS One Engine/0.72 (+http://www.rss-one.com) |
| Rambot | ramBot xtreme x.x |
| RankMeter | Web Snooper |
| Rational SiteCheck | Rational SiteCheck (Windows NT) |
| RedBot | RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) |
| RedCarpet/1.2 | RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) |
| RedCell | RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) |
| RedKernel | RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) |
| RoboPal | RoboPal (http://www.findpal.com/) |
| Rotondo | Rotondo/3.1 libwww/5.3.1 |
| RufusBot | RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) |
| RutterBot | RutterBot(+http://www.aktienbetreuer.de/bot.html) |
| S2Bot | S2Bot/1.0 (http://search2.net) |
| SAIT | sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) |
| SandCrawler | SandCrawler - Compatibility Testing |
| Savvybot | savvybot/0.2 |
| ScanWeb | ScanWeb |
| Schibstedsokbot | schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http://www.schibstedsok.no/bot/) |
| ScholarUniverse | ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; [email protected]) |
| Scirus crawler | FAST Enterprise Crawler 6 / Scirus [email protected]; http://www.scirus.com/srsapp/contactus/ |
| ScorpionBot | Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots |
| ScoutAnt | ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ |
| Sdcresearchlabs testbot | sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache.org/nutch/bot.html; [email protected]) |
| Search Engine Studio | Search-Engine-Studio |
| Search Info | Spider-Sleek/2.0 (+http://search-info.com/linktous.html) |
| Search17Bot | Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php) |
| Search4free | lwp-trivial/1.34 |
| SearchByUsa | SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; [email protected]) |
| SearchEngineWorlds | ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) |
| SearchExpress Spider | SearchExpress Spider0.99 |
| SearchMonkey | Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/searchmonkey/useragent) |
| SearchSight | SearchSight/2.0 (http://SearchSight.com/) |
| SearchdayBot | SearchdayBot |
| Searchdnabot | searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at searchenginedna dot com) |
| Searchguild | SearchGuild_DMOZ_Experiment ([email protected]) |
| Searchit Now Robot | Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) |
| Searchmee! Spider | Searchmee! Spider v0.98a |
| Seeker.lookseek | Seeker.lookseek.com |
| Sensis.com.au Web Crawler | Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) |
| SharewarePlaza | Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) |
| SheenBot | SheenBot/SheenBot-1.0.4 (Sheen web crawler) |
| Sherlock | sherlock/1.0 |
| Shoula | Shoula.com Crawler 2.0 |
| SietsCrawler | SietsCrawler/1.1 (+http://www.siets.biz) |
| Siigle Orumcex | Siigle Orumcex v.001 Turkey (http://www.siigle.com) |
| Sirketcebot | Sirketcebot/v.01 (http://www.sirketce.com/bot.html) |
| Site Server 3.0 Robot | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR |
| SiteBar | SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) |
| SiteBaseBot | Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru) |
| SiteSpider | SiteSpider +(http://www.SiteSpider.com/) |
| SiteTruth | SiteTruth.com site rating system |
| SitiDiBot | +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) |
| Skampy | Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] |
| Skizzle | User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) |
| SkreemRBot | Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) |
| Slider | Slider_Search_v1-de |
| Slurp (Yahoo) | Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
| Slurp China (Yahoo) | Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
| Slurpy Verifier | Slurpy Verifier/1.0 |
| SmiffyDCMetaSpider | SmiffyDCMetaSpider/1.0 |
| SnapPreview | Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 |
| Snapbot | Snapbot/1.0 |
| Snoopy | Snoopy v1.2 |
| SnykeBot | SnykeBot/0.6 (http://www.snyke.com) |
| SoftHypermarket | SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) |
| Sohu search | sohu-search |
| Somewhere | Mozilla ([email protected]) |
| Sosospider | Sosospider+(+http://help.soso.com/webspider.htm) |
| Spam Bot | Mozilla/2.0 (compatible; NEWT ActiveX; Win32) |
| Spbot | Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ ) |
| Speedfind | speedfind ramBot xtreme 8.1 |
| Sphere Scout | Sphere Scout&v4.0 - scout at sphere dot com |
| Sphsearch | FAST Enterprise Crawler 6 used by Singapore Press Holdings ([email protected]) |
| Spinn3r | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130 |
| SplatSearch | libwww-perl/5.45 |
| Spock Crawler | Spock Crawler (http://www.spock.com/crawler) |
| Strategic Board Bot | Strategic Board Bot (+http://www.strategicboard.com) |
| Suchpad | suchpad/1.0 (+http://www.suchpad.de) |
| SummizeBot | Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) |
| SuperSnooper | [email protected] |
| SurveyBot | SurveyBot/2.3 (Whois Source) |
| Susie | !Susie (http://www.sync2it.com/susie) |
| Swooglebot | Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) |
| SycaBoT | SycaBoT/1.0 |
| SycaBoT Audio | SycaBoT-Audio |
| SycaBoT Image | SycaBoT-Image |
| SycaBoT Programme | SycaBoT-Programme |
| SycaBoT Video | SycaBoT-Video |
| Sycrawl | NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; [email protected]) |
| Sygol | <http://www.sygol.com/> http://www.sygol.com |
| Syntryx | Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler |
| TAGword | Tagword (http://tagword.com/dmoz_survey.php) |
| TCDBOT | TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot IE)" |
| Tags2dir | tags2dir.com/0.8 (+http://tags2dir.com/directory/) |
| TailRank Robot | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://tailrank.com/robot) Gecko/20021130 |
| Talkro | Talkro Web-Shot/1.0 (E-mail: [email protected], Home: http://222.122.15.190/webshot) |
| TargetSeek | Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html) |
| Tasapspider | Tasapspider/0.9 (http://www.tasap.com) |
| Technoratibot | Technoratibot/0.7 |
| Tecomi Bot | Tecomi Bot (http://www.tecomi.com/bot.htm) |
| Teemer | Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; [email protected]) |
| TelenetDigger | TelenetDigger/1.0; htdig/3.1.6 ([email protected]) |
| Teradex Mapper | Teradex Mapper; [email protected]; http://www.teradex.com |
| TeragramCrawler | TeragramCrawler |
| TeragramWebcrawler | TeragramWebcrawler/1.0 |
| TerrawizBot | TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) |
| TheSuBot | TheSuBot/0.1 (www.thesubot.de) |
| Theme Spider | Theme Spider (+http://www.themespider.com/spider.html) |
| Theophrastus | Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php) |
| Thumbshots de bot | thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) |
| Thunderstones Webinator | Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) |
| TimboBot | timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html |
| TinEye | TinEye/1.1 (http://tineye.com/crawler.html) |
| TivraSpider | tivraSpider/1.0 ([email protected]) |
| Tkensaku | Tkensaku/x.x(http://www.tkensaku.com/q.html) |
| ToileBot | Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 |
| Topodia | Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; [email protected]) |
| Traazibot | traazibot/testengine (+http://www.traazi.de) |
| TrendTech | http://www.trendtech.dk/spider.asp) |
| Truveo | Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) |
| Turnitin | TurnitinBot |
| TurnitinBot | TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) |
| Turnpike Emporium | Turnpike Emporium LinkChecker/0.1 |
| Tazilla | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html) |
| UN OCHA | FAST Enterprise Crawler 6 used by [email protected] (UN OCHA) |
| UOLCrawler | UOLCrawler ([email protected]) |
| URL Spider | Mozilla/5.0 URL-Spider |
| USyd NLP Spider | USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) |
| Ultraseek | Ultraseek |
| Uni koblenz | http://www.uni-koblenz.de/~flocke/robot-info.txt |
| Unido bot | unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html |
| Updated | updated/0.1beta (updated.com; http://www.updated.com; [email protected]) |
| Updated crawler | updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected]) |
| Urlchecker1.0 | Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) |
| Urlfan bot | urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html |
| VBSEO | Mozilla/4.0 (vBSEO; http://www.vbseo.com) |
| VIProbot | VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; [email protected]) |
| VWBOT | VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;[email protected] |
| Vacobot | Vacobot; (+http://vaco.ws/bot.html) |
| Vagabondo WAP | Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile |
| Vakes | Vakes/0.01 (Vakes; http://www.vakes.com/; [email protected]) |
| Versus | versus 0.2 (+http://versus.integis.ch) |
| Versus Crawler | versus crawler [email protected] |
| VeryGoodSearch | VeryGoodSearch.com.DaddyLongLegs |
| Vespa Crawler (Yahoo) | Vespa Crawler |
| Vlad | Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) |
| Vonna.com b o t | Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) |
| Vortex/2.2 | Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) |
| W3C Validator | W3C_Validator/1.432.2.10 |
| W3C checklink | W3C-checklink/4.2 [4.20] libwww-perl/5.803 |
| W3SiteSearch Crawler | W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de |
| W8net | Mozilla/5.0 usww.com-Spider-for-w8.net |
| WEP Search | WEP Search 00 |
| WISEbot | WISEbot/1.0 ([email protected]; http://wisebot.koreawisenut.com) |
| WWW.fi crawler | www.fi crawler, contact [email protected] |
| WWWeasel Robot | WWWeasel Robot v1.00 (http://wwweasel.de) |
| Wadaino.jp crawler | wadaino.jp-crawler 0.2 (http://wadaino.jp/) |
| Wavefire | Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; [email protected]) |
| WeRelateBot | WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; [email protected]) |
| WeatherBot | WeatherBot v1.4 http://www.ezweather.net |
| WebBOT | Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot/index.html |
| WebFindBot | WebFindBot(http://www.web-find.com) |
| WebGobbler | webGobbler/1.x.x |
| WebRankSpider | WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) |
| WebSearch | WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; [email protected]) |
| WebStat | WebStat/1.0 (Unix; beta; 20040314) |
| WebVac | WebVac ([email protected]) |
| Webbandit | webbandit/4.xx.0 |
| Webclipping | Webclipping.com |
| Webduniabot | Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) |
| Webglimpse | Webglimpse 2.xx.x (http://webglimpse.net) |
| Weblog Attitude Diffusion | Weblog Attitude Diffusion 1.0 |
| Webmeasurement bot | webmeasurement-bot, http://rvs.informatik.uni-leipzig.de |
| Webmeasurement crawler | webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ |
| WebsiteWorth | WebsiteWorth v1.0 |
| Webspinne | Webspinne/1.0 [email protected] |
| Websquash | Websquash.com (Add url robot) |
| Webster | Webster v0.3 ( http://webster.healeys.net/ ) |
| Webverzeichnis | Webverzeichnis.de - Telefon: 01908 / 26005 |
| Whoiam | Mozilla/5.0 whoiam [http://www.axxus.de/] |
| Wikiwix bot | wikiwix-bot-3.0 |
| Willow | Willow Internet Crawler by Twotrees V2.1 |
| WinME | Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] |
| WinkBot | WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; [email protected]) |
| Worio heritrix bot | worio heritrix bot (+http://worio.com/) |
| WorldWideWeb X/3.1 | WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) |
| Wume_crawler | wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) |
| Wwlib/Linux | Wwlib/Linux |
| Wwwster | wwwster/1.4 (Beta, mailto:[email protected]) |
| XP5 | Project XP5 [2.03.07-111203] |
| Xdefine | egothor/3.0a (+http://www.xdefine.org/robot.html) |
| Xirq | xirq/0.1-beta (xirq; http://www.xirq.com; [email protected]) |
| Yacy | yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en) |
| Yahoo Search Japan robot | Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) |
| Yahoo! Mindset | Yahoo! Mindset |
| Yahoo Blogs | Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) |
| Yahoo MMAudVid | Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) |
| Yahoo Test/4.0 | Yahoo-Test/4.0 |
| Yahoo VerticalCrawler | Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/ |
| YahooSeeker | YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;[email protected] ) (KHTML, like Gecko) |
| YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
| Yarienavoir | yarienavoir.net/0.2 |
| Yellopet | spider.yellopet.com - www.yellopet.com |
| YesupBot | Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) |
| Yggdrasil | yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de) |
| YooW! | YooW!/1.8.9 RC1 (+http://www.yoow.eu) |
| Yoogli | yoogliFetchAgent/0.1 |
| Yoono | Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) |
| Yoono web crawler | yoono/1.0 web-crawler/1.0 |
| YottaCars | YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine |
| YottaShopping | YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine |
| Z Add Link Checker | Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) |
| Zealbot | Mozilla/4.0 (compatible; Zealbot 1.0) |
| Zearchit | Zearchit |
| Zedzo.digest | zedzo.digest/0.1 (http://www.zedzo.com/) |
| Zerxbot | zerxbot/Version 0.6 libwww-perl/5.79 |
| Zetbot | DeepIndex ( http://www.zetbot.com ) |
| Zippy | Zippy v2.0 - Zippyfinder.com |
| Zspider | zspider/0.9-dev http://feedback.redkolibri.com/ |
| ZyBorg (LookSmart) | ZyBorg |
| aiHitBot | Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) |
| eCairn Grabber | eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15 |
Looking for the addresses used by the main search engines? The crawler address page lists them. To test one address right now, use the address lookup.
Technical identifiers in this table, including the user agent strings, are reproduced without alteration so that they keep matching real traffic.
Reference shelf
Sources and further reading, all outside this site. They open in a new context and are listed for reference only.
- Web crawler what these agents do
- Googlebot the best known entry
- Robots exclusion standard the rules crawlers follow
- User agent the string being matched
- SourceForge project submit a missing crawler
- CrawlTrack support forum share a new crawler