Is MSN Search getting ahead of Google?

I just looked at our blog statistics to find out how often search engine spiders, such as the MSNBot or Googlebot, visit this site and how many pages they index. As it turns out, the Googlebot either doesn’t like our site that much or it has serious capacity problems. Compared to other search engine spiders, the Googlebot visits our site less frequently and indexes less pages. Unfortunately, this results in having bad rankings on Google, which is of course the number one search engine.

The strange thing is that although our rankings are much better in Yahoo and MSN, most of the hits via search engines come from Google. It seems that nearly no one is actually using these two engines – at least not tech-savvy people who normally visit our blog.

Let’s have a look at the web statistics of our blog from yesterday concerning the search engine spiders. The statistic shows all spider visits of this month. The number after the + is the amount of successful hits for the robots.txt file and is therefore a good indicator for the visit frequency:

As you can see, the MSN spider visits our site quite often and indexes a lot of pages whereas the Googlebot seems to be rather uninterested. I’m not sure, and maybe Google doesn’t like new sites, but I suspect that Google has serious capacity problems and MSN is really catching up. I know that search result quality is more important than the amount of indexed pages, but how can a search engine return good results without knowing most sites?

The Googlebot just visits once in a while, looks at our blog and moves on. It might be a protection against young spam sites or link farms, since this blog is fairly new. Whatever the reasons are, as of this posting the Googlebot neither has indexed our articles nor the postings completely. Hopefully this will change in the future with more content and more sites linking to us.