CrawlTrack blog

As a webmaster you will find on this page informations that will help you. You will be able to better manage your site, protect it and improve his ranking and then of course the number of visitors on it. I will use also that blog to give informatuion about CrawlTrack and it's development. And as there is not only internet and script in life, sometimes I will speak of different subject sutch as travel for example.

CrawlProtect 3.0.0 is available
Author: Jean-Denis Brun   May 06th 2012

After several months of work, I'm please to release the new version of CrawlProtect. That version is a complete review in order to add new features but also to have an easy to maintain code (base on the same framework than CrawlTrack). That new version is using an MySQL database to avoid any compatibilities issues with hosting equipped with Suhosin (or equivalent script), blocking any file creation by a script. The main new features are: fully manageable htaccess, spam filtering on forum and/or blog, picture hotlinking blocage and the last but not the least, the possibility to follow all your sites from the same interface. New languages are also available (Italian and russian). The best to see all these improvment is to download it.

Travel in japan    December 11th 2010

In April 2008, I have spent two weeks in Japan with my 15 years old daughter. We spent one week in Tokyo, then we comute to Kyoto in Shinkansen to spend the second week there.

CrawlTrack is on SourceForge    December 04th 2010

It's a new start for CrawlTrack development, to allow a team work, Crawltrack is now on SourceForge in order to use Subversion and tracker. Philippe, known as Kissifrot, who has already done a lot of work on the lasts releases is the first developper to join me. If you would like to join us, don't hesitate to contact me.

Bingbot change his user-agent    October 23rd 2010

After only one month of running, Microsoft has change BingBot user-agent. It's a tiny change, just a ; more. The new user-agent is now:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

CrawlTrack users have to update their CrawlerList to continue to detect BingBot, the updated list is the CrawlerList 96.

Interview   October 10th, 2010

It should be the start of glory, I have been recently contacted two times to answer to interviews.

The first one has been publish on in romanian, there is a translation tool at the bottom of the page if you don't speak romanian.

The second one is publish (in french) on .

CrawlProtect   October 9th, 2010

In order to concentrate more my activities, from today CrawlProtect is now host on www.crawltrack.com

You will find here all the content of the old www.crawlprotect.com, and if you have a look on the documentation page you will be able to find new function which will appear in the next CrawlProtect release.

Bingbot    October 2nd, 2010

Bing, Microsoft searchengine, had annonced it a few months ago, the old MSNBot will be relayed by a new bot, Bingbot.

The annoncement were speaking of a start-up in October 2010, in fact Bingbot was a little bit in advance as I have detected it's first visit September the 29th around 1 am.
So, if it's not already done, don't forget to upgrade your CrawlTrack to be able to detect it (Crawltrack detect Bingbot since release 3.2.5 and crawlerlist 95).

Bingbot use the user agent:
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm).

Today, MSNBot is still working, but we can think that it will progresssively disapear.

Identified search engines crawlers

Google, Bing, Yahoo or Baidu are constantly enthusiasm to search the web for new pages to add to their index. You can used CrawlTrack to find out when the crawlers went on your site, but if you want more information on IP addresses and user-agents used by the crawlers this article will interest you.

Crawlers known by CrawlTrack

CrawlTrack can detect and identified hundreds of different crawlers. The complete list is available by clicking on the link below.

Tool to identified the origin of an IP

The tool propose here can, with the help of Maxmind database, identified the country of origin of an IP. This combined with the database of the CrawlTrack used on that site, give the posibility to know if that IP is known as used by a crawler or a hacker. Don't hesitate to use it.

Avoid duplicate content

As experimented webmaster you know that duplicate content is very bad for your ranking. You are very cautious to avoid to have the same content duplicate on differents pages. But are you sure that your pages are really unique? If you didn't pay attention, you can have all your pages with two different url: www.example.com/page.htm and example.com/page.htm. For your home page it can be up to four different url !!! You will find here solution to avoid that.

How to protect your site from hacking attempts?

The web is not a peacefull world, you have to know that soon your site will be a hacker target. On www.crawltrack.net, CrawlTrack detect between 500 and 10 000 attacks per day. You cannot ignore that, or one day, you will loose the control on your site!!!