Hacking Memeorandum: More Proof That Algorithms Don’t Work Pete Cashmore and Tara Hunt blog on a recent experiment that they and a few others pursued to try and get something listed on Memorandum’s first page. The point that they seem to be making is not to be critical of Memeorandum’s algorithm but that algorithms in general can at times be exploited and that there’s nothing quite like a set of human eyes to validate an algorithm. I do agree with that and feel that as the best algorithms continue to be developed and build real search and filter value that it makes more and more sense to look for ways to have humans participate as components of the algorithms.
When Flickr first introduced their interestingness feature, which through an algorithm promised superior photography, immediately people began to figure it out and hack it. At one point a bad camera phone shot of a case of toilet paper which was submitted by Fubuki (one of Flickr’s more popular photographers) appeared as the most interesting photo of the day on Flickr. The reason why Fubuki’s joke photo shot to the number one spot was because he submitted it to the deleteme group on Flickr which is a game based on tagging and comments. As human tags and comments are deemed (among other secret sauce components) to add to interestingness, this photo quickly advanced.
The photo didn’t last long at the top spot and at some point was removed from the interestingness photostream presumably by a human editor of some sort.
Another way that Flickr uses human influence on their algorithm is by creating a “may offend” button. By pushing this button it effectively kills the promotion of a photo in the interesting algorithm as it is publicly shown. This allows flickr to use their own members to effectively police their most public photos to keep the rather distasteful porn element of Flickr as much out of the spotlight as possible.
In my own opinion Flickr and Memorandum both work remarkably well. Still, I would expect that Gabe (and his team?) at memeorandum do in fact, like Flickr, at least on a limited basis review the stories on their first page and do have the capability to block stories that might not be appropriate for their front page if they are hacked in.
The bigger question in my mind though is how do you continue to use human input to create smarter and better algorithms in the future. There are two elements that I think are critical in using human input to improve algorithms.
1. Build large strong communities to rank, rate, tag, filter, etc. whatever you are working with.
and
2. Develop competent editorial staff to review the work of your algorithm and allow them to make adjustments where needed manually.
At present this is largely behind the success of Flickr’s interestingness and Digg’s news.
The downside to humans is that they are expensive. And while Flickr seems to have figured out the perfect formula and combination to get the benefits of a community filter AND at the same time have their community pay them, this is a very rare thing. Once you have great community for your algorithm though I think you need to finish the job by putting the icing on the cake with human editorial input. Although this will in fact cost money it will in fact also finely tune algorithms and make them things of perfection that purr like a finely tuned kitten.
In Flickr’s case, for example, their image search technology is vastly superior to Google and Yahoo!’s image search which lacks more of the human or community touch. Still though things can be improved. Do a search for San Francisco on Flickr and then rank it for interestingness. By and large these are pretty interesting photos (even if I’m biased because some are mine). But one photo that sticks out at me is the one from merlinmann. While it may be interesting due to its technological and historical significance, it is not interesting from a strictly “san francisco” perspective. But as a “san francisco” tag was assigned and it was found interesting for non “san francisco” reasons it has been bumped up.
Similarly my photo on the last line of chairs that I took at SBC Park probably is not interesting in a “san francisco” sense per se. Flickr should consider hiring editors to at a minimum go through the first five pages of interestingness results for their top searches in order to neutralize the search placement where it could conflict with the search term at hand. This would be very little work and could even be done by one person. It could ruffle a few feathers as it might impact the popularity of certain photos but it would be good to improve search results for the greater community at large.