Everyone knows that Google is the best search engine, right? I think we’re in need of objective measures of search relevance to help answer this question. Last year Danny Sullivan called for the search engines to report on relevancy. One thing lacking from this plea was an indication of what these relevancy measures might be. He had mentioned some options a few years ago but they were mainly surveys of peoples opinion which are cumbersome, expensive and subjective.

SLI has several measures of search relevancy that we make available to our site search customers. We make these available so our customers can see them improving as the search learns. These could easily be applied to web search.

The first of these is the average rank of the clicks on search results. The perfect search engine would have an average rank of 1 - that is everyone would click on the first result which would wholly satisfy them. A person clicking on a result doesn’t mean the result was good - but it does indicate that they thought from the title and description that it would be a good result. If a result doesn’t satisfy the user then common behavior is to return to the search results page and click on another - resulting in a higher average rank. This graph shows how the average rank can improve over time when learning from user click throughs.

Another simple relevance metric we offer is click through rate, i.e. the number of clicks on search results divided by the number of searches. This is somewhat crude because bad search results may encourage people to click on more than one result. Ideally this number should be 1 - i.e. everyone who searches clicks on one result. If it is less than one then it shows that a portion of people are not clicking on any search results.

Both of these could be refined more - but I haven’t seen any examples of people asking for such objective measures for web search.

We have some interesting data from one of our partners on the average rank of Google and Yahoo results. I’ll share that with you in a subsequent post. Any guesses from those of you that don’t know?

This entry was posted by Shaun Ryan on Friday, January 13th, 2006 at 2:25 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS feed. You can leave a response, or trackback from your own site.