All visitors on your site, human or crawler, send a query with two useful parameters: the address from where the query is sent, and the user agent. In the human case the browser supplies a user agent such as Firefox or Internet Explorer. These two pieces of information help to identify the famous crawlers.
Google uses different crawlers with a specific mission for each of them in theory. In practice they often help each other.
Googlebot
The best known, its mission is to collect data for the Google index.
Google Adsense
Used to know the content of pages that carry AdSense advertisements, so the ads match the page content.
Googlebot Image
As its name suggests, it finds images that will enhance the Google image index.
Bing
Bing, the Microsoft search engine, has changed its name several times. It was formerly known as MSN Search, then for a short time Live Search, before taking the present name of Bing in 2009. The crawler used did not change its name for a long time; it was always MSN Bot. Since October 2010 a new bot name, Bingbot, progressively replaces it.
Bingbot
MSN Bot
The good old MSN Bot continues to serve the Bing index.
Yahoo
Slurp Inktomi
The main crawler used by Yahoo.
Baidu
Baidu is the Chinese search engine. If you are looking for Chinese visitors, it is the place to be.
Address ranges change. Use the address lookup to test a live address, and keep your crawler list updated from the tools page.
Official crawler documentation
Sources and further reading, all outside this site. They open in a new context and are listed for reference only.
- Google crawler verification how Google publishes its ranges
- Bing crawler information Bingbot and partners
- Yahoo Slurp the Yahoo crawler help page
- Baidu spider documentation Baidu crawler guidance
- CrawlTrack crawler list the live list on the production site
- Googlebot help the Google crawler family
- Yandex robot documentation Yandex crawler behaviour
- Robots exclusion standard crawler etiquette
- Robots.txt specification the standard itself
- Internet Archive crawler the archiving crawler