1000 Flowers Bloom

Explicit vs. Implicit Data and the Wisdom of Crowds

Harrison Hoffman has an interesting article up on CNet about the Wisdom of Crowds failing. The phenomena that he's really observing though is a gaming of the system at IMBD. People are voting the Dark Knight movie up and the Godfather down in order to influence the rankings.

Explicit information where people take an action to rate or vote can be wonderful and create fantastic resources. However, it doesn't matter if you're at IMBD or Digg or Yelp it can also be gamed.

At Aggregate Knowledge we have found that Implicit information where systems look at what people are naturally doing tends to be much more reliable.

It's not what you say - it's what you do!

Fresh ID gets it wrong

Kristi Colvin at Fresh ID has a rant about recommendation engines that I think really gets it wrong.

Kristi believes that recommendations don't serve users but in fact only serve the retailers or the publishers that use them. How can that be? You can't force someone to buy something just because you show it to them in a recommendation window. The only reason someone buys something is because they ultimately want it.

Kristi - where's your data that users don't like recommendation engines besides your individual subjective experience?

The true measure of how well a recommendation engine works is by seeing how many people find things they want through it, not the subjective measure of whether you think it looks good or not.

What comes after Search?

It's clear that Search has been a dominant force on the Internet and that Google is has won. According to Compete, Google had a 71.5% share in search in May 2008.

Interestingly though - time spent searching is going down. Data from the Online Publishers Association says that time spent searching dropped 15% from June 2006 to June 2007. Time spent was 4.5% in June 2007.

We need to help people the other 95% of the time that they're online!

Recommendation Systems in Social Media

Muhammad Saleem has done a nice write up of how consumer social media sites are using Collaborative Filtering over on Read/Write Web.

The only thing I disagree with is equating Collaborative Filtering with Online Recommendations. Collaborative Filtering is only one technique that one could use to provide recommendations.CF is a good tool but you have to be careful to apply the right tool to the right situation.

At Aggregate Knowledge we actually use a variety of different algorithms types to determine what someone might be interested in. For example taking into account seasonality, recency/frequency and lexical matches to show you things you're most likely to want to see.

Digg Launches Recommendation Engine Beta

Digg just announced the launch of their recommendation engine. There's a really interesting video with their lead scientist Anton Kast talking about how the technology works on the Digg blog. More coverage here at VentureBeat and TechCrunch.

Digg has been one of the real pioneers in the Discovery space with their use of explicit user data to help surface interesting content.Some of their new approaches are similar to techniques that we have worked on at Aggregate Knowledge and I'm curious to see how well they work. My guess is that they will see a decent sized bump user engagement if you look at pages per user or click through rate on their recommendations.

Anton really nails it when he talks about how some of the biggest problems are doing this in real time and at scale. It shows a great understanding of the real problems in teh space.

If any Digg folks read this I'd love to get an account to check it out and give feedback!