Sunday September 09, 2007

A Word (or Two) on Redshift

I've received a very interesting array of comments from the Information Week redshift story  but nothing quite rivals a slashdot spanking. I keep seeing a set of misconceptions --- which I'll take as a failure to communicate :) --- so let me take a shot re-summarizing the basic points.

Redshift is an observation about the growth of computing demand of applications. If your application's computing needs are growing faster than Moore's Law, then color it red. If they are growing slower, or about the same, color it blue.  

Redshift applications are under-served by Moore's Law. The simple and obvious consequence is that the infrastructure required to support redshift apps needs to scale up. That is, the absolute number of processors, storage and networking units will grow over time. Conversely, infrastructure required by blue-shift apps will shrink as you get to consolidate them onto fewer and fewer systems.

Refining just a little, redshift apps appear to fall into three basic categories:

1. Sum-of-BW. Basically, these are all of the consumer-facing apps (think YouTube, Yahoo!, Facebook, Twitter) that are serving the bits behind an Internet whose aggregate BW is growing faster than Moore's Law.
   

2. *-Prise. These are software-as-a-service style consolidation of apps that, at most enterprises, are blue. But there is a huge market over which to consolidate, so growth rates can become quite large (think eBay, SuccessFactors, Salesforce.com)
   

3. HPC. Technical high performance computing was the pioneer of horizontal scale. For a good reason: halve the price of a unit of computing or storage, and most HPC users will buy twice as much. These apps are expanding gasses in the Nathan Myhrvold sense (think financial risk simulation, weather simulation, reservoir management, drug design).

Why it's a big deal now is my assertion (okay, SWAG) that we are nearing an inflection point where the majority, volume-wise, of computing infrastructure is in support of redshift applications. One the other side of this point is a kind of phase change where the great mass of computing is delivered through redshift-purposed infrastructure.

And if you believe this and you are in the business of building computing infrastructure, then you might want to think really hard about what all this means in terms of what is import and and where you invest your R&D dollars. Read: it's much more about how hardware and software conspire to become scalable systems, than it is about individual boxes.

Oh, and I guess I have to explain my abuse of a very well-understood physical phenomenon. The spectrum emitted from an object moving away from you looks like it has shifted in its entirety to a lower frequency (and thus "towards red" for the visible spectrum). When measuring the spectra of many galaxies, Hubble observed a correlation between the distance and spectrum: the further away a galaxy is from us, the greater the average redshift. A reasonable explanation for this is that space itself is expanding.

And thus (blame me, my lame marketing), the demand for scalable infrastructure is an expanding opportunity. Fact was, I didn't want to change my slides. Apologies to cosmologists everywhere.