We’ve put an interesting press release out today - in which we use the site search behavior on DeepDiscountDVD to predict the outcome of the Oscars. This is a bit of fun and came from the wisdom of crowds theory. This suggests that groups are often smarter than any of the people in them. To me this is counter-intuitive - but I’m sure if I asked the rest of the people in the office then, as a group, we wouldn’t think so :-). There is a book on this subject.

The idea was to compare the site search traffic on DeepDiscountDVD for Oscar nominees in a few of the main categories. We weren’t sure that this would work - most of the recent movies have not been released on DVD. However we found that doesn’t stop people searching for them and DeepDiscountDVD does sell posters of Brokeback mountain.

We found that rather than looking at the total number of searches for each nominee, it was necessary to look at how the search traffic had increased. This stopped George Clooney from automatically winning the best director - there are more searches for him than any of the other directors. This approach had it’s own flaws - but I don’t think it really matters. I’m looking forward to seeing how our predictions pan out.

Next week I’m going to the Search Engine Strategies conference in New York. I’m on the Advanced Search Term Research Tools panel and will be speaking on using site search for keyword research. The basic idea is that when you are sitting around trying to work out which search terms to use for your search engine marketing, you should be looking at what people are typing into your own search box. This lets you see the language of your existing users. You can see the relative popularity of the search terms and all the variations that people use. Of course, as part of our site search service we provide extensive reporting to make it easy to do this research.

It’s been interesting to see this conference evolve over the years. I went to the first SES in San Francisco in 1999 (I think). At the time I was working for Snap.com and we had just released the Live Directory. The conference was much smaller. Sergey Brin was one of several people on the “meet the crawlers” panel. In 1999 a random person you sat next to would most likely be running an adult site (they’re always ahead of the pack when it comes to new media). Now the conference is huge, the Google guys are famous and you’re more likely to be sitting next to someone from a Fortune 500 company. There’s even a list of SES parties.

Is Google setting the standard for enterprise search or is it the other way round?

Gordon Hotchkiss recently posted an article discussing their eye tracking studies and how Google is setting the standard for search results presentation. This talks about the information ’scent’ in search results pages, with the scent strongest at the top and bolding adding cues that help with scanning. The footnote asserts that enterprise search needs to match the standard set by Google.

I agree with that but think that enterprise search is able to add a lot more scent to the search results page because it has access to structured meta-data. In particular images. As far as information scent goes an image reeks a thousand bolded words. Other meta data such as category, brand, and price can be included in the search results and add additional information scents for the person scanning the results. Compare the results of Edwin Watts site search to a Google site restricted search for the same term. Enterprise search is also able to provide more comprehensive indexing and less duplicates.

Will the images and other meta data that you see in enterprise search eventually influence the look and feel of Google and other search engines? Over the years various search engines have tried incorporating thumbnails to include this visual scent - but it has really never taken off. Google desktop search includes thumbnails but the problem with them is they are too small to see the relevant images on the page. Ask.com gets around this by showing larger images when you hover over the binoculars icon. However that doesn’t help the scanner.

Google does have a lot of these images - both for their image search and from the data they get from merchants listed in Froogle. Will we eventually see some of these images in the main Google results? Claria have teamed with Ditto to add images and logos to Yahoo sponsored results on their searchscout property. I think the result is an enhanced search experience.

I’m just back from a week’s holiday in sunny Hawkes Bay, New Zealand. I great place to visit if you like wine.

While I was away, here at SLI we announced our partner program along with a bunch of new partners and plenty more in the pipeline. For me this is a great example of what happens when you employ someone who knows what they’re doing.

Jordan Zweigoron started working for us last year and bought some great experience from WebEx. He’s doing a wonderful job with our partners. It was something we needed to do and in the past we (mainly me) had not given it the attention it deserved and consequently nothing happened on the partner front.

This has happened time and time again. When we employed a graphic designer they did a much better looking web site than the engineer who first tried (the engineer is a great engineer - he’s just not a graphic designer). When we employed a sales person he did a much better job than the bumbling idiot who was doing it (although it did give me a good appreciation of the job).

None of this is surprising - in a small, fast growing company everyone starts out wearing many hats. As you grow and employ specialists you shed the hats. It’s great to see that the hat fits with Jordan - I’m looking forward to growing with our new partners.

It was recently made public that the US government has asked MSN, Yahoo, AOL and Google for some of their search logs. They’ve all handed over the search logs - except Google who is fighting the government. This has raised a lot of concerns about privacy - Danny Sullivan has covered this fairly comprehensively.

The only contribution I’d like to make to this is to give you an idea of what the data might look like that the search engines gave the feds. Here is a link to a file containing almost half a million search terms (zip 2.2MB), from consumer search activity. The file is 7.1MB when it’s unzipped - you can open it in wordpad. It’s really interesting looking at what people search for.

The file is from a day’s activity back in 2001. We’ve removed phone numbers and email addresses. It is ordered alphabetically and each search term is only shown once. Some processing has been done on these keywords - the terms are all lower case, have had white space and punctuation removed and some other processing performed.

This isn’t exactly the format that the search engines would have given the data to the feds and although it does make interesting reading, it does underscore how difficult it would be to get any useful information out of it. Privacy issues aside this seems like a silly request from the Department of Justice. Can anyone see how they could analyze this file to support the child protection law?

FYI - This file is not censored and does contain adult search terms - viewer discretion is advised.