Sunday September 09, 2007 
Sunday September 09, 2007 I've received a very interesting array of comments from the Information Week redshift story but nothing quite rivals a slashdot spanking. I keep seeing a set of misconceptions --- which I'll take as a failure to communicate :) --- so let me take a shot re-summarizing the basic points.
Redshift is an observation about the growth of computing demand of applications. If your application's computing needs are growing faster than Moore's Law, then color it red. If they are growing slower, or about the same, color it blue.
Redshift applications are under-served by Moore's Law. The simple and obvious consequence is that the infrastructure required to support redshift apps needs to scale up. That is, the absolute number of processors, storage and networking units will grow over time. Conversely, infrastructure required by blue-shift apps will shrink as you get to consolidate them onto fewer and fewer systems.
Refining just a little, redshift apps appear to fall into three basic categories:
Sum-of-BW. Basically, these are all of the consumer-facing apps (think YouTube, Yahoo!, Facebook, Twitter) that are serving the bits behind an Internet whose aggregate BW is growing faster than Moore's Law.
*-Prise. These are software-as-a-service style consolidation of apps that, at most enterprises, are blue. But
there is a huge market over which to consolidate, so growth rates can become quite large (think eBay, SuccessFactors, Salesforce.com)HPC. Technical high performance computing was the pioneer of horizontal scale. For a good reason: halve the price of a unit of computing or storage, and most HPC users will buy twice as much. These apps are expanding gasses in the Nathan Myhrvold sense (think financial risk simulation, weather simulation, reservoir management, drug design).
Why it's a big deal now is my assertion (okay, SWAG) that we are nearing an inflection point where the majority, volume-wise, of computing infrastructure is in support of redshift applications. One the other side of this point is a kind of phase change where the great mass of computing is delivered through redshift-purposed infrastructure.
And if you believe this and you are in the business of building computing infrastructure, then you might want to think really hard about what all this means in terms of what is import and and where you invest your R&D dollars. Read: it's much more about how hardware and software conspire to become scalable systems, than it is about individual boxes.
Oh, and I guess I have to explain my abuse of a very well-understood physical phenomenon. The spectrum emitted from an object moving away from you looks like it has shifted in its entirety to a lower frequency (and thus "towards red" for the visible spectrum). When measuring the spectra of many galaxies, Hubble observed a correlation between the distance and spectrum: the further away a galaxy is from us, the greater the average redshift. A reasonable explanation for this is that space itself is expanding.
And thus (blame me, my lame marketing), the demand for scalable infrastructure is an expanding opportunity. Fact was, I didn't want to change my slides. Apologies to cosmologists everywhere.
Post a Comment:
Comments are closed for this entry.
Calendar
RSS Feeds
Search
Links
Greg - we grasped the Redshift concept right away, perhaps because this is a phenomena that has been happening in IT Storage for a while.
See our Redshift Storage post at http://blogs.sun.com/TA/entry/redshift_storage
Posted by Taylor Allis on September 10, 2007 at 12:38 PM PDT #
Greg,
I think you are spot on on this one. If anyone is confused, they need to listen to your presentation at the Sun analyst conference.
The only question I would have would be is redshift computing a niche of a handful of very large compute farms, kind of like HPC today, or will it be a major part of the market? If the former, redshift chasing companies become companies like SGI and Cray. If a little larger, there are places for companies like Rackable, a little more general purpose than HPC-only. If still larger, does it become a specialized portion of several computer companies, like the high-end RISC/UNIX server divisions of IBM, HP, and Sun? Is Blackbox the new Starfire?
There are a million other questions, some have not even been formed yet. Core computing infrastructure could move to *-Prise companies, but then *-Prise customers might install racks of servers to mash up the feeds from multiple *-Prise API vendors.
However, if an enterprise consolidates an application onto a virtual machine on a local server, there is no reason that VM has to be hosted in the enterprise's datacenter. So even the organizations overserved by Moore's law might migrate there apps to virtualized compute farms at *-Prise mega data centers.
At the same time, there are three huge changes in IT architecture occurring at the same time. One is programming APIs have moved up from the operating system to middleware, on both the server (Java and web scripting languages) and the client (the browser is now the interface). The second change is occuring now as basic hardware abstraction (drivers) move into the hypervisor, relieving the OS from having to support drivers for every disparate hardware configuration. The third big change which will happen over the next few years is the filesystem will sediment out of the server and into the storage network, and will most likely be based on pNFS. You have to have a parallel network filesystem for redshift applications.
So yes, there are many questions, but one real answer emerges. And that is "Yes". And that answer is to Nick Carr's question.
Posted by Mark on September 10, 2007 at 04:21 PM PDT #
Actually, my major in college was cosmology, for a short time, before I changed it. I usually realte it to einstein and relativity, though. And, quite frankly, I'm confused by your labeling. As a business, (government, military?)user(especially if I was interested in HPC), I would want things to do the opposite, to blueshift, not redshift. Blue means I'm moving faster(and thus performing better), relative to what I'm chasing, or my "goal". Red, would mean, what I'm chasing, or my "goal" was moving furthur away from me. While cost is one factor in making requisitions for computing resources, it's not the sole factor, especially among market leaders. Market leaders want performance, no matter what the cost, sometimes within reason, sometimes beyond reason. ;)Give me the fastest car on the block, and then we'll talk, basically. Market leaders are usually your commodity computing people, those who just throw another LINUX box on the stack. Linux boxers are usually DIY(Do-It-Yourselfers), which is okay in the short term, and effective as a temporary solution, but when you get the time(and the money), you want it done right, i.e. something that was engineered for scale and performance from the start, not build-as-you-go. So, give me the fastest car on the block, already. :)
*
p.s. I'm not an engineer, so I don't know if this is doable, but I had a dream of a three dimensional processor, in college. Basically, like having a whole stack of servers, in one processor. Do you grok what I'm talking about? If it's possible, it would probably require a revolution in software to run it, but it seems it would still be scalable, although, cooling might present a problem.
Posted by Rod Sandcones on September 12, 2007 at 10:17 AM PDT #