Monday May 12, 2008

There's a fascinating and informative slide deck from Nielsen SoundScan called the State of the Industry that was presented at NARM last week. Some highlights:

Music is really long tail - in 2007, 450,344 of the 570,000 albums sold were purchased less than 100 times. 1,000 albums accounted for 50% of all album sales.

The music industry had its biggest sales week since they started keeping records, with 58 million units sold in the last week of 2007. The previous record was 47 million during the last week of 2006.

13% of all album sales come from American Idol and the Disney franchises.

CD sales are down 31% since 2004, but digital music sales are up 490%.

Surprisingly, Vinyl sales are coming back - they grew 15% in 2007 and are up 70% in the first 3 months of this year. Mostly in indie vinyl.

1 out of 4 albums are purchased in a non-traditional retail store (i.e. internet, or at a concert).

80,000 albums were released in 2007

844 million digital tracks sold in 2007, 1% of all digital tracks accounted for 80% of all track sales.
There's lots more data in there. Definitely worth a good look. - Thanks Oscar

Sunday May 11, 2008

The Echo Nest has released their new recommendation API. The API will give you recommendations based upon a single artist or a set of recommended artists. Echo Nest is one of the few commercial recommenders that doesn't rely on collaborative filtering for recommendations. Instead, the Echo Nest combines information from "online cultural data, text analysis, audio analysis and user activity". The API is very easy to use. You make queries like this to lookup artist IDs:
http://api.echonest.com/lookup?api_key=MY_KEY&names=the+beatles
Once you get an artist ID you can get recommendations with this request:
http://api.echonest.com/recommendations?api_key=MY_KEY&ids=AR6XZ861187FB4CECD
Some recommendations using the new Echo Nest Recommendation API Seed Artist of Jimi Hendrix:
  • Cream,
  • HendriX
  • Stevie Ray Vaughan
  • Jeff Beck
  • The Jimi Hendrix Experience
  • Eric Clapton
  • Funkadelic
  • Buddy Miles
  • Michael Hill
  • Doyle Bramhall II
Seed artist of Led Zeppelin:
  • LED
  • Deep Purple
  • Bonham
  • Page and Plant
  • Black Sabbath
  • Aerosmith
  • The Who
  • Bad Company
  • AC/DC
  • Cream
Seed artist of Miles Davis:
  • Miles David and John Coltrane
  • Miles Davis Sextet
  • Miles Davis Quintet
  • John Coltrane
  • Charlie Parker
  • Kenny Dorham
  • Wayne Shorter
  • Wynton Marsalis
  • Bill Evans
  • Thelonious Monk
Seed artist of the Beatles
  • Badfinger
  • George Harrison
  • The Hollies
  • The Animals
  • John Lennon and Paul McCartney
  • Paul McCartney
  • The Beau Brummels
  • The fourmost
  • Peter and Gordon
  • George Martin
Seed artist of Emerson Lake and Palmer
  • Rick Wakeman
  • Emerson, Lake & Palmer
  • Patrick Moraz
  • Yes
  • Gentle Giant
  • Steve Hackett
  • King Crimson
  • Colosseum
  • Tony Banks
  • Graham Bond

The recommendations, for the most part look pretty good. There's less popularity bias than you'd typically see with a collaborative filtering algorithm. Still, there are some funny bits. If you like Miles Davis, there are 3 other types of Miles Davis that you might like (but I bet you could have figured that out without a fancy recommender). Also, cover and tribute bands seem to feature prominently.

There were a few little glitches with the web services. The returned XML was not always well formed ('&' characters were not properly turned into entities), some artist searches failed for no apparent reason (I couldn't find 'the beatles' or 'weezer' with the 'suggest' call).

I'm not sure if these recommendations would pass the recommender Turing test. Not too many people would offer a recommendation of Lennon and McCartney for Beatles fans - it is too obvious. Same with Jimi Hendrix and the Jimi Hendrix Experience. However, the Graham Bond recommendation for Emerson Lake and Palmer is brilliant and one that I've never seen made by any other recommender (human or machine).

The Echo Nest gives these recommendations for free (for up to a certain amount per day). If you want to make many requests, you can buy a license from the nest. Also, the Echo Nest is offering free unlimited recommendations to the first 100 small music sites that register for the API. You can't beat free, so sign up quick!

I'm really pleased to see the Echo Nest releasing these services. This is just the type of service that the next generation web companies that are trying new ways to deliver music really need.
The first thing I'll do (well, 2nd after filing the expense report), is to download and install Hudson. Hudson is tool for building and testing software projects continuously. It makes it easier for developers to integrate, test and obtain the freshest, tested build. Hudson manages distributed building and testing, integerates with all of the major source code control systems, works with JUnit, Findbugs, makes RSS feeds for build status. I was really impressed with the demo that I saw at JavaOne. Installing it was dead simple: Just Download the jar and type
java -jar hudson.war

If Hudson works as well as it seemed to in the JavaOne talk, then I think we'll be using this as our build and integration tool for Project Aura.

Thursday May 08, 2008

Notes from the Tech Talk at the SanFran music tech summit. This panel was a discussion about the technology behind some of the most popular music sites. Moderator is Colin Brumelle.

photo.jpg

photo-1.jpg

Panelists:

  • Colin Brumelle - Moderator
  • Tom Conrad - CTO Pandora
  • Marc Urbaitel - CTO In-Ticketing
  • Shaun Haber - Warner Bros. Records - Director of operations, using Drupal to build an artist platform.
  • Jeremy Riney - Project Playlist, CTO, Founder
  • Jack Moffit - Xiph, Chesspark - IM, gaming
Why did you chose a particular type of technology:

Marc - uses php - quicker turnaround time, lets them be much more nimble. Open source is good.

Shaun - open source CMS - chose Drupal: a big reason is active developer community.

Jeremy - also uses Drupal - Paylist is the largest Drupal user with 25 million active users.

Tom - were on Java, Oracle, Jetty servlets due to legacy reasons. Oracle was a disaster, so they ported it all to postgres. Re-implemented Oracle procesdures at the Java language. Some core routines in C, - huge memcachd - 200 servers, 2000 interactions per second. 64 bits linux, intel CPUs, the shiny frontend is flash. They didn't have anyone who knows about flash. Used openLazlo to build the application using javascript and their framework and compile it down to flash. Tom says Lazlo is a great piece of software.

Jack - Perl, then Python, with webware frameworks, mysql, postgres, now Jango (rails-like python), they run everything on Amazon EC2 and S3. Wrote lots of Javascript - use all scriptaculous, prototype and others.

Colin: Is EC2 the future?

Jack - Went through CO-LO hell. Was hard to provision new hardware. On Amazon, they can type one command and get 10 more machines. Jack is very happy with EC2, S3. Jeremy was concerned with complexity but Jack says it was not too hard.

Tom: If they were starting today, they would be considering cloud computing like EC2. The hardest part to scale horizontally is the database. Risk become predicting the future. How do you provision just the right amount of servers. This would become a guessing game.

Marc uses cloud computing to do scaling testing (buying lots of tickets at once).

Tom - also the cloud is useful for data recovery - use the cloud to serve as the failover. Pandora decided to do their own CDN. The save much money they by doing it themselves.

Tom says don't by Foundry load balancers

Questions from Derick of CDBaby - Tells the story about how he rewrote CD Baby for PHP and Ruby On Rails. After 2 years of frustration, he threw it all away. Nothing to do with Rails - but keeping the two systems (PHP and Rails) alive was hard. Derick also lauds EC2. Tom does say that you are still paying a margin to Amazon for this so it could cost you more than doing it yourself.

Tom talks about "test driven development". They can rip their system apart and put it back together and be confident that it will work because of their tests


Digital Thought Leaders Panel
Originally uploaded by PaulLamere.
The digital thought leaders panel moderated by Brian Zisk, with Tim Westergren of Pandora, Aza Raskin of Songza, Michael Pertricone of the consumer electronics association and Ty Roberts of Gracenote/Sony.

First topic is the well-worn topic of how we navigate the intellectual property minefield of music. How can companies make money while still compensating artists. I wish the panel would focus a bit more about technology and less on rights and IP. There's a separate legal track. As K7lim says "this is a kindergarten discussion of IP policy."

Tim Westegren calls for better, simpler design is necessary to engage with listeners, especially new listeners. "Simplicity brings people in" says Aza.

Brian asks: "What is the Future?" - Ty Roberts talks about music product. He's interested in 'music packaging' - augmenting the simple MP3 with all of the ancillary metadata (album art, reviews and bios). Aza suggests 'continuity of experience'. Eliminate the facebook.iTunes silos - get rid of having to worry about where your music is coming from. Michael says it is 'Simplicity'

Tim points out that radio has always been popular. He says that people don't want to spend alot of time administrating their listening experience. He suggest that music will be everywhere, supporting by advertising. Once this is in place, there will be lots of ways that people can use and interact with the music.

Aza points to the Kindle as a good example of where things should go with music. "Feels like free" is key - whether it is ad supported or some other model.

Discussion about the "metadata problem".

Tim offers advice to artists - add a new member of the band - a non-musician - to be the marketing person to get the band exposure on the 'nets.


Brian opens the summit
Originally uploaded by PaulLamere.
Brian Zisk has just launched the SanFran Musc Tech Summit. There are hundreds of music tech folks gathered in Japan town. It looks to be a fun event.

Monday May 05, 2008


CommunityOne
Originally uploaded by PaulLamere.
Javaone week has started. We are waiting for the communityone for for the keynote to start. Lots of fun.

Saturday May 03, 2008

Olinda is a prototype digital radio that has your social network built in, showing you the stations your friends are listening to. It’s customisable with modular hardware, and aims to provoke discussion on the future and design of radios for the home.

Wednesday Apr 30, 2008

Here's an unusual recommendation from MyStrands, but in a good way, not a bad way. While I was listening to some Aphex Twin I checked the MyStrands application to see what they recommended. MyStrands evidently didn't have enough data to give a good recommendation, and so they told me that - they didn't blindly give me a bad recommendation, they recommended something popular, but they also told me that they were punting on a specific recommendation.

strands.png

Friday Apr 25, 2008

There's a nifty contrarian minds article about my colleague Steve. There's a nice bit in the article about the genesis of Project Aura - how we took the ideas from Search Inside the Music combined with the tech from the advanced search group to make something new. It's a good read.

Monday Apr 21, 2008

This is big. One of the secret weapons underlying the Search Inside the Music project and Project Aura is a high quality search engine called Minion. Minion handles everything that has to do with Text for these projects. In addition to traditional search, we use Minion for document similarity (the core technique used for Tagomendations), item clustering, sense disambiguation, classification and autotagging. Minion is a research-oriented search engine - meaning that it is designed to allow for all sorts of variations. It is ultra-configurable and has a simple API. The big news is that the process to open source the Minion engine is underway. Steve Green (aka the search guy) has created a Minion project on Java.net - and soon, the Minion search engine will be available for all.

Steve promises to start posting regularly about the engine, so check out the Search Guy blog: The Search Guy

Friday Apr 18, 2008

  1. Take a poll to find out what music people like the least.
  2. Put all this horrible music into one song.
  3. ???
  4. Profit

There's no way you will enjoy this song.

The most unwanted music is over 25 minutes long, veers wildly between loud, quiet, fast and slow. The orchestra features accordion, bagpipe and an operatic soprano singing atonal music, advertising jingles, political slogans, and "elevator" music, while a children's choir sings jingles, holiday and cowboy songs. The song is a masterpiece in modern horror. Read more about this project.

Via Mike Love’s blog

If you go to last.fm right now, you will probably receive an 'unable to connect' error from your browser. Rumor has it that Miley Cyrus didn't like the way she had been tagged at last.fm so she invoked her secret Hannah Montana powers to take the site down. The official story is just a cover.
There's an interesting story in the Washington Post about CBS's plans for Last.fm.

CBS Interactive's Quincy Smith On Re-Org: 'More CBS Corp. Firepower Focused On Interactive' - washingtonpost.com: "Last.fm: It's almost a year since CBS plunked down $280 million for the UK music recommendation start-up and Smith knows he needs to deliver on all that promise. Last.fm has been growing rapidly on its own and has been given start-up like breathing room but it's time to bring it more into the fold. As we've written here before, Last.fm also needs more U.S. traction. Yes, Smith said, they bought Last.fm in part because of its strong international appeal. 'But it's our job, frankly, as a corporation, to make them more valid in the United States.' To that end, CBS will ramp up on-air promos with 'a lot more call outs' starting in May and, very soon, Last.fm will get its own CMO 'charged with plugging them more into the CBS mainstream.' In late May, CBS will start sending viewers of certain shows to the show's music on Last.fm. 'The downside of Last.fm right now, from my perspective, has nothing to do with the service itself ... the question is how do you blow it up and make it a major media brand in the United State?' Last.fm is working with CBS Interactive's Entertainment unit. As Smith put it, 'CBS has done much less of a job with Last.fm than Last.fm has done with Last.fm.'"

No doubt going mainstream with Last.fm is necessary for CBS to justify their $280 million dollar investment - but for a site like Last.fm, going mainstream is really tricky. Last.fm listeners tend to be highly engaged music fans - whose taste runs to less mainstream music - you are more apt to find artists like Radiohead, Arcade Fire and the Postal Service in the top ten at Last.fm compared to the mainstream pop artists like Mariah Carey and Miley Cyrus that populate the Billboard Pop 100. As Last.fm goes mainstream, there is going to be a culture clash as the mainstream listeners collide with the old-school last.fm listeners. Teenyboppers (and their angry parents) will wonder why Hannah Montana is tagged with 'hardcore death metal' and 'lesbian'. Old-school last.fm listeners will leave for alternative sites as last.fm loses its focus as the best site to discover new indie and alternative music. It's Digg all over again. As Digg went mainstream, it lost its focus on technology - and all the hard-core tech readers went back to slashdot. The problem for Last.fm is that as it goes mainstream, it risks losing much of its core audience - the highly engaged music fan - and this user base of hard core music fans is one of Last.fm's biggest asset.

Thursday Apr 17, 2008

In the TED Talk Releasing the music in your head, Tod Machover talks about new ways for people to interact with music. Tod argues that music is not just for listening. Everyone can create music, given the right tools. Tod builds systems that allow people to interact with music, experiment with music and even to perform music, even without formal music training.

Tod introduces Dan Ellsey -a composer with cerebral palsy who uses Hyperscore to write and perform his music. It's an interesting talk.

This blog copyright 2008 by plamere