Duke Listens!: On the stairway to the Celestial Jukebox - a blog by Paul Lamere

A classic problem in traditional collaborative filtering recommendation is the 'cold start' problem. It is hard to generate recommendations for new items because there isn't enough taste data about the new items to make reliable correlations with other items. That's where content analysis comes in. The cold start problem can be alleviated by basing recommendations on similarity of content as well as the wisdom of the crowds. New items can be analyzed and enrolled into a recommender, making these items available and recommendable.

However, using content-based techniques doesn't guarantee the elimination of the cold-start problem. Pandora, everyone's favorite Internet Radio, uses content-analysis to drive their customized radio. However, since Pandora performs all of their analysis by hand, there may be some lag before your favorite artist makes it into the Pandora catalog.

There's another content-based recommender - BookLamp.org. The BookLamp F.A.Q describe BookLamp as a

"book recommendation system that uses the full text of a book to match it to other books based on scene-by-scene measurements of elements such as pacing, density, action, dialog, description, perspective, and genre, among others. In other words, BookLamp.org is a Pandora.com for books, based on an author's writing style. If you match against multiple books, the self-learning system adjusts your formulas to make the match specific to your tastes. As the system moves out of beta, it will also incorporate human feedback into the recommendation systems, blending the strengths of social networks with the strengths of computer analysis. Ultimately, we want users to be able to create and share their own formulas, creating a community of book lovers that have tools to discover and share books in a way never before possible. Because the system matches books through objective data from the text itself instead of relying solely on social networks to generate recommendations, the recommendations are impervious to outside influences such as advertising or author marketing. It also allows you to match to a far greater detail than alternative systems. With BookLamp, you can request a book similar to Stephen King's The Stand, but half the length, first person, literary mainstream fiction, with slightly more dialog, less description, and a rising action level across the first 10 scenes. If that's what you're looking for".

It is a neat idea, and sounds very similar to the types of things we are doing with Search Inside the Music and Project Aura. Using content analysis gives you better ways to help people discover new items. However, BookLamp has its own cold start problem. Again, from the BookLamp FAQ:

Does BookLamp Work? Can I use it right now to find a book to read?

The simple answer to this question is that while BookLamp works, it doesn't have enough books in the database to work well. While the technology behind the system is capable of finding you books to read right now, BookLamp will remain a technology demonstration until we have a large enough database of books to give the system enough data to make realistic recommendations. Without more books, not only will most users have a hard time finding a book to match against, but the system will have a limited number of books that are capable of being matches. In other words, if we don't have a book in the database that matches, we won't be able to recommend a book for you. Additionally, with so few books in the database, we're not able to match against all the metrics that we would like. In order to be the most effective, BookLamp needs to match against 7 to 8 metrics; with less than 300 books in the database, we're having to make recommendations after matching against only 3 or 4 metrics. To get any matches at all, we've had to turn down the sensitivity of the measures (see the next question) a bit already.We estimate that it will take a database of at least 10,000 books to make BookLamp a usable system. The more, the better.

So BookLamp has a bit of a problem, with only 300 books in its database, it is not going to be the best book recommender. And unlike music, it is not so easy to enroll a new book - scanners and page turners are involved. So BookLamp is trying to figure out its next step. If I were them, I'd build a recommender for the Gutenberg project with its over 25,000 titles. Of course there are no NY Times best sellers in the bunch, but it would be a great way to fine tune the content-analysis while providing a service to a worthy project.

As a way to demonstrate some of the capabilities of Project Aura, we've created Aardvark, a blog recommender. Of course, the world probably doesn't need another blog recommender - but by building a blog recommender we get some experience dealing with a really large dataset that has lots of interesting characteristics. For one thing, we are using content analysis, classification and autotagging to help identify relevant content. We use incoming links and attention to determine how much authority a particular entry has on a topic. Since we use content as well as social data to identify relevant content, we are less susceptible to the cold start problem, and we are able to create recommendations based upon topics not just the usual "people who read X also read Y".

We didn't want to build yet another blog reader, everyone already has their favorite way to read blogs - and we didn't want to spend a lot of time crafting an attractive web interface like the Findory. Instead, we wanted to focus primarily on the back-end, on the quality of the recommendations. So instead of spending our time writing a fancy blog reader that you wouldn't want to use anyway, we just made it work with our favorite blog reader: Google Reader.

Here's how it works. Read your blogs in Google Reader like you always do. When you read an item that is particularly interesting add a star (using the Google Reader 'add star' link). Over time you should build up a pretty good collection of items that represents your interest. For Aardvark to generate recommendations for you, we need to learn what your preferences are. Luckily, the Google Reader allows you to export your starred items as a feed.

In Google Reader, navigate to your settings/tags section, where you can 'view public page' for your starred items. On this page you'll find a link to your starred item feed. Copy that so you can paste it into Aardvark. Here are my starred items.

Now you are ready to get recommendations from Aardvark. First, login to Aardvark (note that this isn't publicly available yet).

You can use your OpenID to register. When you register you paste in the URL to your starred item feed. Now Aardvark can notice the items that you are starring and generate recommendations for you based upon these items.

Once you register, Aardvark will give you a few feeds that you can add to Google Reader. This is where Aardvark will send your recommendations:

After you add the recommendation feed to Google Reader, Aardvark will keep the feed filled with items that you might find interesting:

Well, there you have it, a quick tour of how you can use Aardvark to get blog recommendations. We aren't quite ready to make it available to the general public - but if you think you might be interested, you can at least start to star items in Google Reader, so that when we are ready, you'll be ready.

BTW, Jeff built the web interface and integrated the OpenId support. The site really looks great.

Aardvark, our blog recommender that relies on Project Aura, our web-scale recommender has a crawler that builds a link map so we can derive the authority of a blog based upon the link structure of the blogosphere. For fun, I built a link map of the top 60 or so most linked blogs at blogs.sun.com. It is interesting to see all of the connections, and of course Jonathan's is the most linked to blog of all. Here's an excerpt, the full version is pretty large. There's a totally hardcore PDF as well that shows the link structure for the top 200 blogs.

(60 node PDF version)
(200 node PDF version)

Update: Henry Story points out some obvious, missing links - turns out we haven't crawled all of blogs.sun.com yet, so if your well connected blog is not in the map, it will be soon.

Over the past few months, the Project Aura team has been building a web-scale recommender. As an example of what one can do with Project Aura, we've implemented a blog recommender called Aardvark that uses Project Aura. Aardvark crawls the webs looking for blogs and recommends stories to you based upon your interests. We've found that blog recommender is an excellent example for recommendation - there's lots of things to recommend (we crawl millions of new stories in a week), there's high churn on the content (a post that is 5 days old is ancient history) and people seem to be genuinely frustrated with blog reading - either there are too many posts to slog through in a day, or it is hard to find where to start reading about a particular topic.

To demonstrate some of the under-the-hood technologies of the recommender (document similarity, classification and autotagging), and to add a bit of sizzle to what can be a bit of a boring demo ("now let me show you 10 blog posts that you might (yawn) find interesting"), we've built a dashboard for Aardvark. The dashboard is a 3D app that shows the status of the live running system. Blog headlines scroll through the space as they are crawled from the web.

Clicking on any headline brings up the story. We can look at the manual tags and autotags for the story, find similar stories, get recommendations and explore the topic space of the blogosphere through this interface. Of course, this isn't the type of interface that anyone would use to read blogs on a daily basis, but it is a great way to show off what is going on behind the scenes in the recommender.

If you are interested in learning more about Project Aura, and Aardvark, and if you'd like to see the dashboard in action, be sure to come to our session at JavaOne. The session is TS-5841: Project Aura: Recommendation for the Rest of Us (Next Generation Web / Cool Stuff), and is at 12:10 PM on Tuesday, May 6.

In my extreme busy-ness of last month, I missed this item. In the spirit of the NetFlix prize, MyStrands has issued a 'call for recommender startups'. They are looking for teams with an original, creative, viable and relevant recommender idea. Five finalists will travel to Lausanne Switzerland for RecSys08 where the final selection will be announced. The winner will receive an offer of a $100,000 investment in their idea/company. Unlike the NetFlix prize, the MyStrands prize is loan/investment whose terms are 'determined by Strands in its sole discretion.'

I'm not in any position to participate since I don't meet the eligibility requirements (individuals or sole proprietors and privately held businesses), but if I did, I'd be a little cautious about responding to this call, since my cool recommender idea would be published in a public forum - where anyone could see it and copy it. On the other hand, if I had a good idea, that was hard to duplicate or had some secret sauce - this might be just the thing to give me a year or two to turn it into the next Google.

Via MyStrands Blog » Looking for the best early-stage recommender start-ups

Next month, a couple of students will arrive at Sun Labs to intern over the summer working on aspects of Project Aura and Search Inside the Music.  The interns always bring energy, new ideas, and new music to the labs.  It is a fun time for us old stodgy researchers and hopefully a fun time for the interns as well.  One of our summer interns, Will, has already been thinking quite a bit about issues around recommendation and has been writing about them in his blog.  Some good posts:

• Blog Recommenders 
• Using exemplars in a music recommender
• The Role of Recommendations
  
• Why I like MusicBrainz

Back in February, when we were interviewing intern candidates (and there were many excellent candidates to chose from), my colleague Doug Eck (the brains behind the machine learning on Search Inside the Music) recommended one of his students for the internship.  During the hiring process, I mentioned to  Doug that the competition was pretty stiff - I pointed him to a couple of Will's posts as an example of the competition.  Well, ever the diligent professor, working hard to get his students the best opportunities, 20 minutes later, Doug sent me a link to his student's "blog".  Here's a screen capture (with bits obscured to protect the innocent).

Clearly a highly relevant and insightful blog, covering many fascinating topics.  A must read - immediately added to my set of RSS subscriptions.
 

 

Steve and I our giving a talk about Project Aura at JavaOne.   The talk is at 12:10 PM on Tuesday, May 6.   It is a good timeslot (not too late and on the first day), but there are lots of excellent talks that occur at the same time.  If you come to the talk, be sure to say hi. 

Here's the talk abstract: 

Project Aura is an open-source recommendation engine, written in the Java™ programming language, being developed by researchers at Sun Labs.

Recommendation technology is a key enabler for the next-generation web. Recommenders will be essential to helping us wade through the huge volume of content such as news, music, video, blogs, and podcasts. However, current recommendation technology that relies on the wisdom of crowds has several drawbacks that can ultimately lead to poor recommendations.

Project Aura takes a novel approach to recommendation, avoiding many of the problems inherent in traditional recommender systems. This session presents the technology behind Project Aura, along with an analysis of how well Project Aura performs. It shows how you can use the Project Aura API to generate high-quality recommendations in your Java technology-based applications. Finally, it shows the capabilities of Project Aura by demonstrating Aardvark, a blog recommender built with Project Aura.

I've been a very infrequent blogger lately - I've been spending every free moment coding - getting ready for the Sun Labs open house - our annual labs event where we highlight the work we've done in the last year.  This week, Steve, Jeff and I get to show what we've been working on for the last 6 months or so - Project Aura.

 Project Aura is a web-scale, open, hybrid recommendation system that uses social data (the wisdom of the crowds) combined with the 'aura' of information extracted directly from content or mined from the web to make recommendations.  By combining content-based methods with social methods Project Aura can avoid much of the 'cold start' problems that plague traditional collaborative filtering recommenders, while providing a way to offer explainable recommendations. 

 Project Aura is built on top of Project Caroline - a platform for development and delivery of dynamically scalable Internet-based services.

Here's a photo of our demo setup ... you can see a bit of our visualization of the recommender, and a poster showing our distributed datastore (and steve too!)

 

Steve and I are giving a talk at JavaOne about our recommendation project.  We submitted our slides a couple of weeks ago and we've received some feedback.  According to the J1 organizers,  our Slide 18 must go!  It is too controversial!   

Our slide 18  shows how social recommenders can be vulnerable to hacking and manipulation. It shows a screenshot of the amazon recommender linking Pat Robertson's "Six Steps to Spiritual Revival" to a gay sex manual.   The story behind the recommendation prank is here.   Apparently that's too racy for JavaOne.  

 

  We offered this alternative, which shows the Last.fm similar artists for Hillary Clinton - which have apparently been manipulated Last.fm listeners to tie her to politicians with a less than stellar reputation. Once again, this was a  No go with the JavaOne crew - politics can be a bit of a touchy subject too.  

So that leaves us looking for some less controversial examples of recommendation hacking - no sex and no politics allowed.  Our current best candidate is the boring  Vote For the Worst  which is a site that encourages people to vote for the worst American Idol Candidate - surely there's something better than that. Any suggestions?

Last week I went over to Bentley college to talk about recommender systems for a class on user-centered design.  Not because I know anything about user-centered design - quite the opposite.  I'm hoping that some of the students in the class will apply their user-centered design skills to help us improve the user experience of our recommender.   It was fun talking to the class,  Managing a User-Centered Design Team.   The students  asked great questions, and  seemed  ready to  dive in.

I've promised to leave some pointers to some background reading for usability and recommender systems, so I'll be posting links in this blog post as I come across them.  

The first article is Interaction Design for Recommender Systems by Kirsten Swearingen, and Rashmi Sinha.  This article presents a study of user interaction with 11 online recommender systems.  Of particular interest is the observations about the role of transparency in  recommendations.  (Note also that author Rashmi Sinha has written some excellent articles about tagging as well: A cognitive analysis of tagging, and a Social Analysis of Tagging).

Update: Andre Vellino suggests Nava Tintarev's "A Survey of Explanations in Recommender Systems"

There's an ever growing number of videos on the web.  YouTube seems to have an endless supply of everything from professional documentaries to phonecam captures of the latest campus tasing.  Despite this video glut, finding interesting videos is not so easy.  My current approach is to go to YouTube or Digg and just watch the most popular videos of the day.  It's not too satisfying.  A video of a 30 second car crash  has a much better chance of making these charts than a thoughtful, well-crafted film. Clearly tools for helping us explore this long tail of video will be increasingly important.

ffwd is  new service (still in private beta), that is hoping to help you find discover web video.   I've taken it for a quick spin.  They have a pretty nifty enrollment process - where you click on shows that you like so they can get an idea of your taste.   Once you've selected your shows they assign you a video personality based upon your selections (I'm a 'comedy writer', apparently I like comedies).   Once you are enrolled you can start to discover videos.

The video discovery is labeled as 'alpha' and they aren't kidding. The recommendations don't seem to be related to what I like at all.  They offered reality shows (The Apprentice, Extreme Makeover Home Edition), weight loss success stories from San Antonio,  a 'sexy webcam girl', nothing that I was interested in. The videos are coming from video sharing sites like iFilm and YouTube.   Since they are in alpha, we can forgive them their recommendations, especially since they  do seem to be looking for the right type of people to build their recommender.  They have a job posting for an AI expert with skills like:

• computational linguistics
• collaborative filtering
• behavioral profiling
• relationship mapping
• semantic clustering
• symbolic systems
• Bayesian statistics
• feedback analysis
• personalized search
  

 That's a pretty good sign.    I think ffwd is targeting a real void - video discovery on the web - it will be interesting to see if they can do better than what we see now on Digg or YouTube.  Oh yeah, The ffwd blog is pretty interesting too.

In a new article in the Guardian, Jemima Kiss suggests that "If web 2.0 could be summarized as interaction, web 3.0 must be about recommendation and personalization".  Jemima hits the nail on the head when she says that privacy, credibility and trust are more important than ever for the next generation of tools we'll use to find content on the web.  She also wonders what the implications are for the music critic.  Will the machines replace the music review?  Personally, I don't think that will happen, but the dynamics of power certainly our changing.  The listening habits of millions of people can be strong indication of what is 'good music' - perhaps more so than the opinion of the few critics in the ivory towers.

I went to Amazon during lunch to buy a book recommended by Tim Spaulding over at the LibraryThing Thingology blog:

 

While I was there I noticed Amazon recommending this book:

 

 which I couldn't resist. When I added it to my cart,  I was also offered this recommenation:

 

which again, I couldn't resist. Amazon then suggested this book:

I really like some of the things that Elias has done with Processing - so I added that one to my cart too.

At that point, I quickly hit the 'Check Out' button before I went broke.  Amazon's recommender turned my $27 order into a $121 order.  Multiply that by a million and it is easy to see why companies like Amazon and Netflix are investing so much in recommender systems. 

tagurself is a widget that displays your interests as a tag cloud.  You give tagurself a URL to an APML, RSS, blog or web page and it will give you a web 2.0 tag cloud.  It works especially well with APML - for example here's the tagurself cloud based upon my last.fm music interests (using the tastebroker to generate my listening APML).

 

If you go to one of the many social music sites out there and get 'similar artists' recommendations for Jimi Hendrix.  You are likely to get a list such as the one you get from Last.fm:

 

There's no arguing that this is a good list - but it is also a rather diverse list.  Eric Clapton's blues guitar is quite different from the psychedelic acid rock of the Doors. I'd really like to know a bit more about the recommendations - in particular I'd like to know why a particular artist was recommended. This can help me gain trust in the recommender as well as help direct me to artists based on criteria that  are most relevant to me.  Unfortunately, most recommenders are based strictly on consumption habits, so the only recommendation explanation available is the Amazonian "People who listened to Jimi Hendrix also listened to Eric Clapton and the Doors"  - which is not too helpful for me.

We want to make recommendations transparent - so that you can ask 'why did you recommend this' and get a useful answer beyond the typical 'people who bought X also bought Y'.  In order to generate transparent  recommendations you need to have some understanding of the content - there has to be some way of knowing that Hendix tracks typically contain distorted guitar, for example.  Companies like Pandora, with their music genome project have spent years analyzing to hundreds of thousands of tracks, assigning 400+ attributes to each track. This lets Pandora give those great explanation that makes them so popular with their users:

Pandora can generate these excellent explanations because they've taken the time (and spent lots of money) to listen to and extensively label all of their music.  Most companies won't have the time, money or patience to do this - and even Pandora that is committed to this approach, can't keep up with the volume of new music that is generated every year.  Luckily, there are other sources of content description that we can use to generate recommendations.

One such source are social tags.  Social tags have been all of the rage in the web 2.0 world.  Sites like del.icio.us, Flickr and Last.fm demonstrate how the such tags can be extremely useful for searching and organizing content, especially non-text content.  Social tags can also be used to give us good transparent recommendations.  

Here are the most frequent social tags applied to Jimi Hendrix at Last.fm:

 

Now tags like 'rock' and 'blues' occur rather frequently for many artists, so are less descriptive than tags like 'guitar god' which occur infrequently across artists.  So we can take this into account and generate a list of the most distinctive tags for Jimi Hendrix.  These are tags that occur rarely across the entire set of artists, but occur frequently for Hendrix.  (This is the classic TF-IDF term weighting technique).  Here are the distinctive tags for Hendrix:

The less descriptive tags such as 'rock' have fallen off the list while extremely targeted tags like 'jimi hendix' and 'acid rock' have risen to the top.  This list is a much better description of Hendrix than the 'Frequent Tags' list and can serve as the basis for our transparent recommendations.

With these distinctive tags, we can generate recommendations based upon the cosine distance of these distinctive tags to the distinctive tag sets of other artists.  This type of recommendation, based upon the similarity of distinctive tags, gives us surprisingly good results.  My colleague, and resident neologist, Steve, coined the term 'tagomendations' for recommendations based on the social tags.  Here are the tagomendations for Jimi Hendrix (ordered by artist popularity):

Interestingly (and surprisingly) our tagomendations compared favorably to recommendations generated with the more traditional collaborative filtering techniques when we evaluated them in a survey.  And of course, with these tagomendations based upoon distinctive tags we can now explain why a recommendation was made, based upon how the distinctive tags for two artists overlap.  

For instance, for the Clapton recommedation, we can look at how the distinctive tags for Clapton and Hendrix overlap:

Clearly if you like Hendrix because of his blues guitar playing, you might want to give Clapton a listen. Compare this to the overlapping tags of The Door and Hendrix:

This is quite a different vibe - with the focus on psychedelia of the 60s.

These tag clouds showing how the distinctive tags for recommended artists overlap with the seed artist give me the opportunity to explore the recommendations based on my taste.  If I like Hendrix because I am a fan of face-melting guitar, I will quickly find artists, like Joe Satriani, Gary Moore, or Steve Vai - but if the reason I like Hendrix is because of the 60s, psychedelic vibe, I may find Jefferson Airplane or Steppenwolf more to my taste.

That's it in a nutshell - how we are using social tags to generate transparent,  explainable  recommendations.   And by the way, to do the heavy lifting for our Tagomendations  we are using the text search engine called Minion, developed by the Advanced Search Technology group here at Sun Labs.  Minion is a high quality, highly configurable search engine that is perfect for doing these types of experiments.  Look for Minion, It's coming to an open source repository near you very soon.

Likewise, we hope to release our web-based music explorer that can generate transparent tagomendations soon.  Here's a little bit of what it looks like: