French
protection
CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.





diable

Custom Search






Identified search engines crawlers

Google, Bing, Yahoo or Baidu are constantly enthusiasm to search the web for new pages to add to their index. You can used CrawlTrack to find out when the crawlers went on your site, but if you want more information on IP addresses and user-agents used by the crawlers you will find them here. All visitors on your site (human or crawler) send a query to your site wincluding two different parameterd. It's the IP address from where is send the query and the User-agent (in human case, it's the browser Internet Explorer or Firefox for example who supply that User-agent). These two informations will help you to idenfied these famous crawlers.


Google

Google use different crawler with a specific mission for each of them in theory. In practical, they often help each other.

GoogleBot
The better known, his mission is to collect datas for the Google index.

User-agent:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

IP addresses used:
66.249.65.xx
66.249.68.xx
66.249.71.xx


Google-Adsences
He is used to know the content of the pages that carry advertisements adsence to allow the display of advertisements tailored to page content.

User-agent:
Mediapartners-Google

IP addresses used:
66.249.65.xx
66.249.68.xx
66.249.71.xx


Googlebot-Image
As its name suggests it is used to find images that will enhance the image indexes of Google.

User-agent:
Googlebot-Image/1.0

IP addresses used:
66.249.71.xx

Bing

Bing the Microsoft search engine has changed its name several times, it was formerly known as MSN search. It was for a short time Live search before taking the present name of Bing in 2009. The crawler used has not changed his name for a long time, it was always MSN Bot. But since October 2010, a new bot name Bingbot will progressively replace it.

Bingbot
User-agent:
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

Adresses IP utilisées:
65.52.108.xx
207.46.16.xx
207.46.195.xx
207.46.199.xx
207.46.204.xx


MSN Bot
The good old MSN Bot has continued to serve to enrich the index of Bing

User-agent:
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)


IP addresses used:
65.55.104.xx
65.55.208.xx
65.55.209.xx


Yahoo

Slurp Inktomi (Yahoo)
It's the main crawler used by Yahoo.

User-agent:
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)


IP addresses used:
67.195.37.xx
67.195.110.xx
67.195.111.xx
67.195.112.xx
67.195.113.xx
67.195.114.xx
67.195.115.xx
72.30.65.xx
72.30.78.xx
72.30.79.xx
72.30.81.xx
72.30.142.xx
72.30.161.xx
74.6.8.xx
74.6.17.xx
74.6.18.xx
74.6.22.xx


Baidu

Baidu, it's the chinese search engine. If you are looking for chinese visitors, it's the place to be.

Baiduspider
It's the crawler used by Baidu to crawl the web.

User-agent:
Baiduspider+(+http://www.baidu.com/search/spider.htm)

IP addresses used:
123.125.66.xx
220.181.7.xx
61.135.162.xx