CrawlProtect 3.0.0 is available
Jean Denis Brun, 6th May 2012
After several months of work, I am pleased to release the new version of CrawlProtect. That version is a complete review, in order to add new features but also to have an easy to maintain code, based on the same framework as CrawlTrack. The new version uses a MySQL database to avoid compatibility issues with hosting equipped with Suhosin or an equivalent script that blocks file creation by a script.
The main new features are a fully manageable htaccess file, spam filtering on forums and blogs, picture hotlink blocking, and the possibility to follow all your sites from the same interface. New languages are also available, Italian and Russian. The best way to see all these improvements is to download it.
Two weeks in Japan
11th December 2010
In April 2008 I spent two weeks in Japan with my 15 year old daughter. We spent one week in Tokyo, then moved to Kyoto by Shinkansen to spend the second week there. The photographs and the day by day route are in the travel journal.
CrawlTrack is on SourceForge
4th December 2010
It is a new start for CrawlTrack development. To allow team work, CrawlTrack is now on SourceForge in order to use Subversion and the tracker. Philippe, known as Kissifrot, who has already done a lot of work on the latest releases, is the first developer to join me. If you would like to join us, do not hesitate to contact me.
Bingbot changes its user agent
23rd October 2010
After only one month of running, Microsoft changed the Bingbot user agent. It is a tiny change, just one more semicolon. The new user agent is now: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm).
CrawlTrack users have to update their crawler list to continue to detect Bingbot. The updated list is crawler list 96.
Interviews
10th October 2010
It should be the start of glory: I have recently been contacted twice to answer interviews. The first one was published in Romanian, with a translation tool at the bottom of the page. The second one was published in French.
CrawlProtect moves to CrawlTrack
9th October 2010
In order to concentrate my activities, from today CrawlProtect is hosted on www.crawltrack.net. You will find here all the content of the old CrawlProtect site, and on the documentation page you will be able to find the new functions that appear in the next CrawlProtect release.
Bingbot arrives
2nd October 2010
Bing, the Microsoft search engine, announced it a few months ago: the old MSNBot will be relayed by a new bot, Bingbot. The announcement spoke of a start in October 2010. In fact Bingbot was a little in advance, as I detected its first visit on 29th September around 1 am. If it is not already done, do not forget to upgrade your CrawlTrack to be able to detect it. CrawlTrack detects Bingbot since release 3.2.5 and crawler list 95.
Bingbot uses the user agent: Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm). MSNBot is still working, but it will progressively disappear.
Resources for webmasters
Identified search engine crawlers
Google, Bing, Yahoo and Baidu constantly search the web for new pages. This page lists the addresses and user agents their crawlers use.
Crawlers known by CrawlTrack
CrawlTrack detects and identifies hundreds of different crawlers. The complete list is available on this page.
Tool to identify the origin of an address
The tool identifies the country of origin of an address. Combined with the CrawlTrack database of this site, it tells you whether that address is known as a crawler or a hacker.
Avoid duplicate content
Duplicate content is bad for your ranking. Two urls for every page, and up to four for the home page, is a risk you can remove with one rule.
How to protect your site from hacking attempts
The web is not a peaceful world. Soon your site will be a hacker target, and on this site CrawlTrack detects between 500 and 10,000 attacks per day.
Sources and tools mentioned
Sources and further reading, all outside this site. They open in a new context and are listed for reference only.
- SourceForge project development home of CrawlTrack
- Bing Webmaster guidelines crawler behaviour from Microsoft
- Google Search Central crawling and indexing documentation
- Yandex Webmaster crawler information from Yandex
- Baidu Search Resource Platform crawler information from Baidu
- Maxmind GeoLite the geolocation data used by the tools
- Zenphoto the gallery software behind the travel journal
- Japan National Tourism Organization travel information for Japan
- Wikipedia, Tokyo background on the capital
- Matomo open source analytics comparison
- Google sitemaps helping crawlers find pages
- Bing sitemaps the Microsoft equivalent
- Robots exclusion standard the protocol behind robots.txt
- HTTP caching why fresh pages matter
- Internet Archive older copies of this blog
- Request Indexing refreshing a listing