Going Bollywood

I just got back from a series of customer meetings with technologists from the telecommunications, media and entertainment industry (they are, after all, converging on the same market, monetizing consumers). I was joined by Greg, and a number of folks from Sun, notably Jeff Bonwick and Bill Moore, along with Matt Ahrens, the co-inventors of the ZFS file system.

All three industries are largely underserved by innovation - their requirements and opportunities are vastly outstripping the rate at which the industry is innovating. (That's code for, "their technology budgets are growing.")

There were a number of choice anecdotes from the sessions, my two favorites being the following.

The first was relayed by the CTO of a major movie studio, who explained the value of very, very long term deep storage archives.

His company had recently pulled a more than fifty year old movie out of a salt mine - where it was stored on 35mm film color separations. Before you ask, vinyl film outlasts the standards that the industry produces (a point on which I've written previously), and salt mines are environmentally more stable than datacenters). They pulled the separations to remaster and reissue the film on DVD.

In so doing, the resulting movie actually increased in resolution for the viewer - the devices used to display the movie today (likely a laptop or high definition TV) offer better resolution than what was available to the original viewers (likely a 50's era TV or movie theater). The movie improved with time - the modern viewer saw the movie in greater detail than the original viewers (suggesting one should always store higher resolution data than the display you see in front of you).

After release of the digitally remastered version, the DVD rose to number 8 on the Amazon best seller list.

Cost of production? Near $0. The effort was nearly pure profit.

And they have a library of approximately 30,000 films.

That's a ton of value in a salt mine (if all the titles are as compelling, which is unlikely, but an interesting thought exercise, nonetheless).

The second anecdote relates to the advent of very high resolution digital cameras - the highest, and most desirable are currently known as "4K," offering 4096 x 3112 pixels (!) per frame - yielding cameras that spew 100's of megabytes of imagery per second).

The director of one feature length film wanted to keep all the footage from a soon to be introduced new movie. He wanted to preserve outtakes and all, for the eventual "Behind the Making of..." or "Director's Cut..." versions of the movie. The digital master for an average 4k film is roughly 9 Terabytes - that's for the version you and I would see in a theater.

But the total archive including out takes and secondary/tertiary angles (bits are a lot cheaper than film, so why not set up three or four cameras for every shot?) was roughly (drum roll please)... a PETABYTE (or a thousand terabytes, or roughly 500,000 iPods). Equivalent to about a million feet of 35mm film.

That's a lot of data. To be archived for... well, to the first anecdote, likely forever (like health records or airport surveillance). And now you know one of the (many) principle motivations behind a file system we built at Sun, ZFS. The focus of the ZFS team is both the scale, simplicity and quality of storage (on Mac OS X, BSD, Solaris and Linux).

This is a great overview of ZFS by Jeff and Bill - of why ZFS matters to the media and entertainment industry, and likely anyone concerned with high quality, high scale and high productivity, storage.

As I was told by Jeff (parroting a storage executive), there are only two types of disk drives in the industry.

Drives that have failed, and drives that are about to fail. And now you know what inspired ZFS, and what's inspiring interest in it.

Posted on 08:50PM Oct 07, 2007 | Comments[47]

Share this post  | | | | | |