Skip to the main content

Free and open source since 2005, running on your own server

CrawlTrack Webmaster dashboard

CrawlTrack blog

As a webmaster you will find information here that helps you manage your site, protect it and improve its ranking, and therefore the number of visitors. The blog is also where CrawlTrack development is announced.

CrawlProtect 3.0.0 is available

Jean Denis Brun, 6th May 2012

After several months of work, I am pleased to release the new version of CrawlProtect. That version is a complete review, in order to add new features but also to have an easy to maintain code, based on the same framework as CrawlTrack. The new version uses a MySQL database to avoid compatibility issues with hosting equipped with Suhosin or an equivalent script that blocks file creation by a script.

The main new features are a fully manageable htaccess file, spam filtering on forums and blogs, picture hotlink blocking, and the possibility to follow all your sites from the same interface. New languages are also available, Italian and Russian. The best way to see all these improvements is to download it.

Two weeks in Japan

11th December 2010

In April 2008 I spent two weeks in Japan with my 15 year old daughter. We spent one week in Tokyo, then moved to Kyoto by Shinkansen to spend the second week there. The photographs and the day by day route are in the travel journal.

CrawlTrack is on SourceForge

4th December 2010

It is a new start for CrawlTrack development. To allow team work, CrawlTrack is now on SourceForge in order to use Subversion and the tracker. Philippe, known as Kissifrot, who has already done a lot of work on the latest releases, is the first developer to join me. If you would like to join us, do not hesitate to contact me.

Bingbot changes its user agent

23rd October 2010

After only one month of running, Microsoft changed the Bingbot user agent. It is a tiny change, just one more semicolon. The new user agent is now: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm).

CrawlTrack users have to update their crawler list to continue to detect Bingbot. The updated list is crawler list 96.

Interviews

10th October 2010

It should be the start of glory: I have recently been contacted twice to answer interviews. The first one was published in Romanian, with a translation tool at the bottom of the page. The second one was published in French.

CrawlProtect moves to CrawlTrack

9th October 2010

In order to concentrate my activities, from today CrawlProtect is hosted on www.crawltrack.net. You will find here all the content of the old CrawlProtect site, and on the documentation page you will be able to find the new functions that appear in the next CrawlProtect release.

Bingbot arrives

2nd October 2010

Bing, the Microsoft search engine, announced it a few months ago: the old MSNBot will be relayed by a new bot, Bingbot. The announcement spoke of a start in October 2010. In fact Bingbot was a little in advance, as I detected its first visit on 29th September around 1 am. If it is not already done, do not forget to upgrade your CrawlTrack to be able to detect it. CrawlTrack detects Bingbot since release 3.2.5 and crawler list 95.

Bingbot uses the user agent: Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm). MSNBot is still working, but it will progressively disappear.

Resources for webmasters

  • Identified search engine crawlers

    Google, Bing, Yahoo and Baidu constantly search the web for new pages. This page lists the addresses and user agents their crawlers use.

  • Crawlers known by CrawlTrack

    CrawlTrack detects and identifies hundreds of different crawlers. The complete list is available on this page.

  • Tool to identify the origin of an address

    The tool identifies the country of origin of an address. Combined with the CrawlTrack database of this site, it tells you whether that address is known as a crawler or a hacker.

  • Avoid duplicate content

    Duplicate content is bad for your ranking. Two urls for every page, and up to four for the home page, is a risk you can remove with one rule.

  • How to protect your site from hacking attempts

    The web is not a peaceful world. Soon your site will be a hacker target, and on this site CrawlTrack detects between 500 and 10,000 attacks per day.

Sources and tools mentioned

Sources and further reading, all outside this site. They open in a new context and are listed for reference only.