Hal Stern's thoughts on the economy, software, services, technology, and snowmen. Hal Stern: The Morning Snowman

Short form: we have just posted a 2-part video featuring Jonathan Schwartz and Greg Papadopoulos on "Why the OS Matters" (part 1 part 2) on YouTube. It's our first video version of Innovating@Sun, our audio-mostly, mostly-fun podcast about fun things happening mostly at Sun.

Long form: I love it when things don't turn out exactly as scripted. We have been working on a video edition of our podcast for a few weeks, and actually recorded the whole show in about a take and a third (the third was me suffering through lip flap, dub flub and assorted other recording maladies caused by too much cold medicine and not enough coffee). The three of us just kept on talking, with tape rolling, and ended up in a discussion of operating systems and why they matter. Still. Or more than ever, depending upon your point of view.

Clearly, this is my own point of view, as I've been quoted before saying that as soon as you try to remove the operating system, you start adding in things like networking and resource management and filesystems and soon enough, you have an operating system. The trick is to make the operating system strike the right balance of hard-core performance and soft-spoken general purpose uses. Make it broadly appealing, and the hard-core developers will find ways to make it scream for a variety of applications.

Recording this -- essentially unexpurgated, unedited, and unprompted -- reminded me of the similarly direct-to-tape track It's About Time by Pierre Moerlen and Gong. The band is in a tight groove, you hear a door open as someone delivers a needed piece of equipment, the rejoinder "It's about time" from the band, and then direct to groove again.

A quick Google check confirms this as the first public linkage of Greg, Jonathan and "groove" as a feeling, not a developer environment.

During a customer visit today, someone asked me what Sun thinks about liquid cooling of computer systems. Seems that they are doing a serious evaluation of the next flavor of cooling equipment to install in their data centers. My answer was somewhat oblique -- the best way to cool a system is to start by generating less heat. It's basic conservation, something unchanged since Oakland funk boys Tower of Power wrote Only So Much Oil in the Ground.

Just as I shudder thinking about some of the cars my family owned in the 1970s, requiring an inter-service station map to plot our gas-guzzling course, I'm sure our children will bristle at the notion of rooms full of chillers that use as much power as the equipment they're cooling. The solution is to generate less heat -- cooler CPUs, better system packaging, and even improved inter-chip communications like the Proximity I/O work in Sun Labs that reduces energy consumption (and therefore heat) at a systems level. Whenever we think of some resources -- oil or cooling capacity -- as infinite, we design badly. When they're constraints, we drive responsibility. I still worry that Tannenbaum's station wagon full of tapes only gets 4 miles to the gallon.

Just finished reading Rick Wakeman: The Caped Crusader, a semi-biography of the Yes keyboards player that covers his life from childhood through the "main sequence" of Yes albums ending with Going For The One. Found it on amazon.com through a used bookseller for about $30, which was significantly better than the occasional copy that shows up on eBay for closer to $100.

Of all of the anecdotes and quips in the book, though, the one that stuck with me was that Wakeman played the Mellotron on David Bowie's "Space Oddity," marking one of the first recorded uses of that keyboard (pun intended). He was paid nine pounds sterling for the gig -- session fees for session work, never mind the fact that Bowie was breaking a lot of rock and roll glass. I guess you never know when you're on the way up until you can look back from the next hill.

We deal with nesting in many different contexts: VPNs (IP networks inside of other IP networks), filesystems (directories of directories, or mount points on top of other mount points), containers (instances of the operating system inside of other instances) and the more common functional recursion that's taught in 101-level coding classes. The common thread is that nesting (or recursion, in a more generalized form) lets you build complex structures more simply (in some cases), provided you do the necessary design work to allow the structures to nest in the first place: ensure there's a basic order of operations so the system can recover from failure, avoid mangling global state when there's more than one instance, and identify (and avoid) infinite nesting. And if you do a good job, you can treat everything below you as a black box.

Today, we have a hierarchy of nesting structures that give data center architects a wide variety of choices and control points. The announcement of Sun's Project Blackbox is just the highest-level abstraction in this stack:

CPU-level virtualization. Define the instruction set and system interfaces, and run multiple operating systems on slices of a CPU. Sun is previewing Logical Domains for our UltraSPARC T1 processor; you can also use VMware or Xen on your favorite x64 processor at the lowest level of virtualization. We've also announced Solaris support for Xen, making Solaris both a guest and a host operating system.

OS-level virtualization. Containers, or the ability to run an instance of the OS inside itself. Useful for isolating application code that you don't know or trust, or for providing a complete system image view for applications that are somewhat anti-social and assume they're in the one-app-per-box world of a few years ago. If you agree on the OS interfaces, and can isolate and aggregate at that level, then you're good to go.

Stack-level virtualization. Define the interfaces to which you deploy code, and ensure that you don't slide outside that bounding box, and you can virtualize up the stack. It's not a foreign concept; it's basically how salesforce.com is built, as well as any number of web site hosting companies that give you an httpd server, a PHP engine, and a set of packages like WordPress that you can drop on top of their stack. Want to use something not on the package list? Find someone who provides (and manages) at the OS level. Being able to compress your interface requirements into a statement of stacks, though, gives you incredible flexibility and power in terms of how you deploy. I had first-hand experience with this last month -- one of my volunteer projects was running the mail and web server for a local religious organization. The entire system could be described as IMAP4, Apache, and majordomo. After seven years of nearly flawless operation, the system died an ugly death -- so we simply shipped the user data to another hosting provider that gave us those three interfaces and we were back on the air the next day. The long pole in the time tent wasn't system configuration but rather moving our DNS entries.

Data center level virtualization. How do you manage it? Capture events? Do physical security? Identify the basic unit of work, deployment, and cost for power, cooling, space, and compute/storage density? These are all questions that get asked as part of virtualization at a lower level of abstraction (anyone who wants to implement CPU-level virtualization but hasn't built a model for allocating applications to CPU resources is making work, not progress). Answer these questions about your data center and you can build a box -- literally -- around it.

That's Project Blackbox: stimulating conversation about data center abstraction. Generating some interest in literally boxing it up and moving it to where the space, power, cooling, or environmentals are more friendly.

With all due respect and apologies to Jon Anderson, the word is live: Innovating@Sun is now available.

Tuesday, October 17, marks the first of our Innovating@Sun podcasts, a kind of funky, edgy, view of what's going on at Sun and why it's relevant -- to employees, customers, passers-by and anyone else who can spare 10 minutes of airtime. My simple goal was a rough stylistic mash-up of Stephen Colbert, James Governor, and Mark Cuban, without straying from technical accuracy or overloading the volume compander on our recording console.

The last time I had this much fun with a microphone was when I was the voice of Sunday Morning Jazz on WPRB 103.3 FM, in the mid-80s, and I had a limited but loyal crew of listeners (including my parents who were sometimes able to pick up the signal with appropriate hand-holding of the FM antenna). Some of the faithful were incarcerated in the Trenton state correctional facility, and it seems their choice of Sunday morning listening was made for them by someone who liked my show and had popular (or declarative) control over the dial at that time.

Here's hoping you like the thoughts, whether or not you're forced to listen to them.

During our Customer Engineering Conference (CEC) this past week, I described advertising as repeated messages that make you buy things you didn't know you needed. I don't know what you call buying things you don't need, and if I were to make a joke about it involving Yankees baseball management George Steinbrenner would fire me from my blog.

It all comes down to something Mark Cuban wrote in his blog immediately after his Mavericks lost in the NBA Finals: you have to want to do the work. The Yankees didn't do the work. The Tigers and three other teams did and their post season continues. It's frequently not glamorous -- it's about practice, and mental positioning, and being prepared, and learning as much as you can. Many lessons in there for technical sales as well, because that's another team effort that requires everyone to do the work (a CEC attendee suggested to me that we print up t-shirts with Cuban's "Do The Work" ethic on them, so I'm not alone in this thought).

My baseball highlight of the weekend was watching Chris Young throw a fantastic game for the Padres, giving him the same number of wins in the post season as the entire Yankees starting rotation for about a tenth of the cost. ESPN magazine called Young "The Bigger Unit" in a cover-titled story almost two years ago, when he had just been called up to the Rangers (and beat the Yankees in his first start). He's big, he's strong, he's Young, and he's even hockey-related (his wife is a member of the Patrick family, as in hockey's former Patrick Division).

Oh yeah, he's also a Princeton graduate, which is where our family first intersected with his career, watching him play the pivot on the Tiger hoops team, until the Pittsburgh Pirates drafted him and said "Hardball, not hardwood." How about that for a story line for the New York papers -- locally educated, multi-sport, bigger than (most of us) life player, and you can make Rangers jokes with his wife? He did the work for four years at Princeton -- finishing in Tiger town before going to a series of very small baseball towns, doing the work for his job before doing a job that worked on the Cardinals in Game 3.

It's not the highest-paid athlete, or the household name, or the flashiest person picked up by the press. It's the players, inordinately big or small, who come prepared to work hard that make a difference.

The NHL inches toward better fan interactivity, this time with a limited blogging section on nhl.com. You can create "pages", which are equivalent to blog entries, but there is almost no HTML allowed, making this more of a walled (non-Madison Square) garden than a true blogging environment. It's better than the eBay blogs, but not by much -- blog content doesn't appear to be visible to Google or other crawlers, and the tagging environment is largely self-referential.

I guess I'm not "getting" the half-way houses of content. Linda Cohn (ESPN on-air personality and former hockey player) laments the lack of a hockey audience. Commissioner Bettman has repeatedly said that hockey isn't an ideal TV sport, and he's been looking to electronic properties to grow the popularity of the sport both in North America and in the rest of the world that provides about a third of the NHL's talent. What better way to do this than making making the web site a true medium of the fan? Maybe we'll be critical, or loud, or flat-out wrong, but we'll be passionate, productive, and visible as well.

I'm already signed up -- and my first challenge was to find a way to get my NHL "Community Connect" zone accessible from the outside world. It's beta, it's got some rough edges, but it's still safer for armchair (left) wings than skating into the corner.