Skip to the main content

Free and open source since 2005, running on your own server

CrawlTrack Webmaster dashboard

Identified search engine crawlers

Google, Bing, Yahoo and Baidu constantly search the web for new pages to add to their index. CrawlTrack tells you when crawlers went to your site. This page gives the user agents and address ranges they use.

All visitors on your site, human or crawler, send a query with two useful parameters: the address from where the query is sent, and the user agent. In the human case the browser supplies a user agent such as Firefox or Internet Explorer. These two pieces of information help to identify the famous crawlers.

Google

Google uses different crawlers with a specific mission for each of them in theory. In practice they often help each other.

Googlebot

The best known, its mission is to collect data for the Google index.

User agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Addresses used: 66.249.65.xx 66.249.68.xx 66.249.71.xx

Google Adsense

Used to know the content of pages that carry AdSense advertisements, so the ads match the page content.

User agent: Mediapartners-Google Addresses used: 66.249.65.xx 66.249.68.xx 66.249.71.xx

Googlebot Image

As its name suggests, it finds images that will enhance the Google image index.

User agent: Googlebot-Image/1.0 Addresses used: 66.249.71.xx

Bing

Bing, the Microsoft search engine, has changed its name several times. It was formerly known as MSN Search, then for a short time Live Search, before taking the present name of Bing in 2009. The crawler used did not change its name for a long time; it was always MSN Bot. Since October 2010 a new bot name, Bingbot, progressively replaces it.

Bingbot

User agent: Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm) Addresses used: 65.52.108.xx 207.46.16.xx 207.46.195.xx 207.46.199.xx 207.46.204.xx

MSN Bot

The good old MSN Bot continues to serve the Bing index.

User agents: msnbot/1.0 (+http://search.msn.com/msnbot.htm) msnbot/1.1 (+http://search.msn.com/msnbot.htm) Addresses used: 65.55.104.xx 65.55.208.xx 65.55.209.xx

Yahoo

Slurp Inktomi

The main crawler used by Yahoo.

User agents: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) Addresses used: 67.195.37.xx 67.195.110.xx 67.195.111.xx 67.195.112.xx 67.195.113.xx 67.195.114.xx 67.195.115.xx 72.30.65.xx 72.30.78.xx 72.30.79.xx 72.30.81.xx 72.30.142.xx 72.30.161.xx 74.6.8.xx 74.6.17.xx 74.6.18.xx 74.6.22.xx

Baidu

Baidu is the Chinese search engine. If you are looking for Chinese visitors, it is the place to be.

User agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) Addresses used: 123.125.66.xx 220.181.7.xx 61.135.162.xx

Address ranges change. Use the address lookup to test a live address, and keep your crawler list updated from the tools page.

Official crawler documentation

Sources and further reading, all outside this site. They open in a new context and are listed for reference only.