Yahoo and Microsoft's Poor Indexing of Blogs is Hurting Them in Search

I just got done reading an interview with Larry Cornett, Director of User Experience Design at Yahoo and Kathryn Kelly, Director of PR for Yahoo! Search over at search engine land. The interview was pretty dry and talked a lot about higher click through rates for ads and how search positioned on portals affected things and some of the new stuff with Yahoo Answers that I'm not a big fan of, etc.

Gord Hotchkiss did the interview and one of the things that he noticed was that Yahoo seemed to talk more about their advertisers and balancing search with advertising than other search companies:

"One difference you'll notice from Marissa's interview last week is the continual reference to Yahoo's advertisers as key stakeholders in the experience. At Yahoo, whenever user experience is mentioned, it's always balanced with the need for monetization."

Which may have something to do with why Yahoo's search results feel worse than Google, but I think I figured out a bigger problem for Yahoo and Microsoft that they need to address if they hope to regain market share in search and that is that they are horible at indexing blogs.

Now some people might say, so what, big deal, blogs are mostly full of hot air, why should they be given higher relevancy than other sites etc..., but hear me out on this one.

I just checked my stats package for search traffic on thomashawk.com (out of the last 1,000) and the got the numbers below. I left off any site with less than 1%

708 google.com
66 images.google.com
55 search.yahoo.com
44 google.co.uk
33 google.ca
30 blogsearch.google.com
19 search.msn.com
19 search.blogger.com
11 images.google.ca
10 images.google.co.uk

Now lets think about this. 5.5% of my search traffic came from Yahoo and another 1.9% of my search traffic came from Microsoft. And most of the rest essentially came from Google. These numbers are far, far, higher for Google than what all the measurement tracking services etc. give as market share numbers etc. But what these numbers tell me is that Google is indexing my blog far, far, better than Yahoo or Microsoft.

Now, let's just assume for a second that it's not just my blog. That in general Google ranks blog results much higher than Yahoo or Microsoft. Well, as a blogger I can tell you that a great deal of my search results are for stuff on blogs, with my own blog being probably the most commonly searched. So many times when I write a blog post I think for a second and say, yeah, I know I blogged about something like that before, I just can't remember when. Maybe it was an obit for Dr. Gene Scott or Hunter S. Thompson. Maybe it was my photo essay on Disneyland. Maybe it was an article I wrote on how to find the best photos on Flickr, or whatever. The point is that a blogger frequently needs to refer back to previous stuff that they've written.

But not just stuff that I've written. So many times I'll see something and then say to myself, yes, didn't Robert Scoble write something about that three weeks ago. Or, where is that post again where Jeremy Zawodny or Shelley Powers came after me. See bloggers blog, but they also are voracious readers of other blogs.

So if every time I look at my stats package I'm reminded of how superior Google is at indexing blogs, who's going to get set as the default search engine in that little box up on the upper right hand side of my Firefox screen? That little box which dictates where 99% of my search is done? Yep. Google.

Now, how many bloggers are there? I have no clue, but per Dave Sifry's last Technorati update there are 57 million blogs and counting.

Now the thing about bloggers is that they probably know more than the average person about technology and the internet. I'm saying generally speaking, on average. And so many of these people tend to be early adopters and are looked up to by all of their friends who are not as savvy about technology. So when they go home to their parent's house for Christmas what do they do? (Speaking for myself here). Well they take over their parent's PCs and do things like download Firefox and set it as the default browser with Google as the default search engine. They do a virus scan and put shortcuts where they should go and clean things up. They reset the homepage of the browser from MSN to something else, etc. etc.

And who wins here amongst the search engines? Google of course.

Now lest you think I just blindly believe my stats package that Google indexes my blog better than Yahoo or Microsoft here's some real world examples.

Let's do a Search for Canon 40D. Google, Yahoo, Microsoft

On Google a blog post I wrote on the new rumored 40D is the 8th selection on the first page. On Yahoo the same post is the 18th on the second page. On live.com17th.

Now lets take another term, Michael Devlin (who I recently wrote a rant about that he should get the death penalty if he's convicted).

Google, Yahoo, Microsoft

Ok here I'm the 10th result for Michal Devlin on Google right there on the front page. Nowhere to be found at all on Yahoo (it doesn't count if it's not on one of the first three pages). And the 76th result on live.com on the second page.

I could go on, but I think you get the point. Google indexes blogs far, far better than Yahoo or Microsoft. Bloggers know this from looking at their stats packages. Bloggers then feel better about Google (even if subconsciously) and they also get more relevant results when looking up stuff from their blog or their blogger friends on Google vs. the other engines. And bloggers tell all their friends that Google is the best search engine.

Now whether or not it's right that Google ranks blogs so high is a whole other debate. There is a strange sense of power when you know that you can permanently affect the Google result of any person or company with a mildly unique name simply by putting their name in a headline on your blog. And of course having your blog post stick around with someone's name for years can have pretty significant impact which is part of the reason Jill Greenberg was so pissed at me when I called her a child abuser. I'm now the number three search result for her name and that's probably going to stick for a long long time.

My advice to Microsoft and Yahoo? Increase the relevancy that you give blogs. Bloggers will see the increased traffic in their referral logs and start to check that out. They will feel better about you and if you really send them a lot of traffic maybe even start using your service or blogging about the increase, etc. etc.

All of this is just my own anecdotal evidence of course and certainly nothing that's been given much depth of research as far as I know, but as a jumping off point there just might be something to this. Now, I need to go back and change that little Y in my search bar up above back to G. It somehow doesn't look right as a Y up there.

Now let me ask you. Are you a blogger? If so what is the breakdown of the search traffic driven to your blog? Is it mostly Google like mine?