Darrin Johnson's Weblog: Need for speed...

I have been an avid gamer all the way back to playing the original Oregon Trail from MECC via a "dumb" terminal connected to a central mainframe using an acoustic coupled modem. Fortunately gaming has come a long way especially playing on my Xbox 360 consoles (Standard and Elite thanks to my wife!). Here are my favorite Xbox 360 games (in order):

1. The Elder Scrolls IV: Oblivion - The ultimate console RPG with hundreds of hours of game play.
2. Ghost Recon Advanced Warfighter (GRAW) 2 - This game turned me into a huge fan of tactical shooters.
3. Halo 3 - An outstanding first person shooter and likely the first to get me into online play.
4. Bioshock - An immersive first person shooter like no other. Who's your Big Daddy?
5. Command & Conquer 3 Tiberium Wars - Best darn RTS available on a console with easy to use controls.
6. Tom Clancy's Rainbow Six: Vegas - Another worthy tactical shooter.

How did these games get on my favorites list? Mostly because they are easy to learn (sorry I am not of the "combos" generation), immersive (it helps in HD), and allows with lots of interruptions with out penalizing the player.

There are others (e.g. Call of Duty 3, GRAW, Gears of War, FEAR, Crackdown, Lost Planet, etc.) which I own but haven't gotten around to playing so I can't rank them as of yet. However there are a number of cool games on the horizon that might keep me from getting around to the older games:

• Call of Duty 4: Modern Warfare
• Blacksite: Area 51
• Tom Clancy's EndWar
• Grand Theft Auto IV

So many cool games and so little time to play.

Tivo, and DVRs in general, have been both a boon and bane to society. On one hand they do a great job recording the shows you want to see and other hand they record ALL the shows you want to see. The consequence is that you end up watching more television than ever before. Google Reader (www.google.com/reader) is also both a boon and bane. Gone are the days of trying to memorize websites and RSS feeds, having bookmark lists the length of your arm, custom links on your desktop, etc. Google Reader creates a fire hose of information with a remarkably easy to use interface and even provides easy to add bundles on a variety of topics including news, sports, technology, video, etc. What you don't find in the bundles can easily be added manually or by clicking on any RSS feed icon. Once you have your feeds in place it is trivial to scan them to either ignore them (and the counter automatically decrements) or clicking on them for more information. Google Reader even gives you trends to show how addicted you are:

From your 81 subscriptions, over the last 30 days you read 15,280 items, starred 1 items, shared 0 items, and emailed 0 items.

Ironically one of my feeds is Zen Habits which recommends to simplify your life and shy away from such foolish tools. What can I say I am addicted! Those of you who still rely on memorization, bookmarks, custom links, etc. should give Google Reader (or similar tool) a try.

My wife and I decided to get away for the weekend and head over to Half Moon Bay to have some good seafood, see the sites, pick up some pumpkins and simply relax. We decided to splurge and stay at the Ritz-Carlton in Half Moon Bay in part because of the tremendous brand reputation of customer service. Ironically, customer service certainly wasn't their strength with a comedy of customer service errors (13 to be exact) that individually could be written off but as a whole was a stunning disappointment. The manager and assistant manager quickly acknowledge the problems as being fault of the staff and were apologetic. The both repeatedly asked what they could do (beyond trying to resolve each individual problem as they arose). This brought to mind the question..."What is good customer service worth?" or put another way "What kind of compensation would mitigate the bad service?". Other questions also factor in..."Should cost of services/goods factor into the equation?", "Should brand reputation be a factor?", "Does the severity or frequency issues factor into it?", etc. These questions however lead to a natural desire to negotiate for the maximum compensation possible.

I however take a different approach. Instead of negotiating, I prefer to let the person know that it isn't up to me to ask for compensation for the trouble but instead up to them to decide what value I am to them as a customer. They need to understand the value of good customer service with the consequence of their decision having a direct impact on whether or not I remain a customer (and tell others). It is my belief that ultimately the payout is far greater in that the company is more likely to make the changes needed to eliminate the service issues in the future. In the case of Ritz-Carlton in Half Moon Bay, I genuinely believe that the message was understood and that I will gladly be a customer again. Additionally, although I had insisted that I am not looking for compensation for the issues, they took care of the room charges which was completely unexpected.

Sun has launched OpenEco.org which is an online community to provide free tools and resources for calculating, tracking, and comparing greenhouse gas (GHG) emissions. Dave Douglas, VP of Eco-responsibility at Sun, says, "This project came about because we were trying to calculate our own carbon footprint ... and we found that the tools out there were proprietary or expensive, or else companies were building spreadsheets (to calculate their footprint) on an ad hoc basis."

http://openeco.org/
http://weblog.infoworld.com/sustainableit/archives/2007/09/sun_delivers_op.html

In popular media and technical journals we frequently see reports of the latest security weaknesses, hackers breaking into systems, services and data being compromised, etc. Professor Arun Sood, at George Mason University, has taken a novel approach to confronting these security threats. The fundamental approach of Self Cleansing Intrusion Tolerance (SCIT) is to create a known good version of a service (e.g. identity, dns, webserving, etc.) which is then replicated with some of the services put online and some held offline in reserve. Then periodically the offline copies are brought online with online versions being taken offline, scrubbed and ready to be brought online again. I don't do the technology justice so I recommend taking a look at his his website at http://cs.gmu.edu/~asood/scit/ and especially the listed publications.


Professor Sood also posted a great introductory article on Reavis Consulting Group's RiskBloggers.com on "Exposure Time - A Metric For Proactive Security Risk Management". The article discusses the critical consideration of the relationship between the intruder residence time (IRT) and the potential for loss as defined by the loss curve. The "exposure time" however, as Professor Sood asserts, is a more interesting metric. Unfortunately there is invariably a trade-off of cost (e.g. increased number of replicated services, additional equipment, etc.) and exposure time. If you are interested in services security and how to design low exposure time systems then this article and related links are worth reading.

For those of you who are dog lovers you will understand there is nothing like a dog that is so excited to see you that it's whole body is in motion, ready to launch up on to your lab, wet slobbering kisses to show how much you were missed, and maybe a few barks to make sure you are paying attention. What is even more amazing is when that dog isn't even yours and has that same excitement even when days, weeks or months may have passed since the last time you had seen them.

I can say that I was blessed to have known such a dog. A dog whose droopy eyes, long ears, short legs, etc. did not reflect the excited puppy at heart, tremendous intelligence and unexpected grace. Toby, the basset hound, died yesterday of cancer. What is stunning is that only two weeks ago he had all the excitment and energy mentioned above with absolutely no symptoms of his illness. Toby was strong until the end."

Toby will be missed by many. May he rest in peace! Condolences to his family and pack mates!

Toby as a puppy....


Toby at the dog park...

I find out recently that my favorite chocolate maker, Scharffen Berger, was purchased by the very commercial Hershey's and was stunned. Although it wasn't a life defining momment it did however give me pause to think about why Hershey's may have purchased a boutique chocolatier. Then I realized that many companies have been doing similiar acquisitions - beer companies buying up microbreweries, computer companies buying up gaming machine companies, soft drink vendors buying bottled water and specialized drink companies, etc. Unlike the trend with computer companies to provide customized solutions (e.g. configurations) as part of mass customization movement, this is more like "Mass Specialization".

Why would mass specialization be interesting? As is generally the answer it all has to do with money. People are willing to pay more for specialty products. In the case of Scharffen Berger people are willing to pay $4.00 to $7.00 a 3 oz. bar instead of $0.75 for a Hershey Almond Bar which likely means there is certainly better profit margin in the specialized bar. Then you factor in the economies of scale of Hersheys which can greater buying power, strong infrastructure, and infinitely better distribution channels. It is clear that the profit margin will only increase.

The most important question is ... "Will the quality suffer?" Only time will tell. However the large mass producing companies have a vested interest ensuring that the specialized product remains specialized to preserve their margin. I hope though that my $6.50 El Carmen Venezuela 75% Cacao Pure Dark Chocolate Limited Series Scharffen Berger chocolate bar will become slightly cheaper given the economies of scale. Unfortunately, or fortunately, I can't eat chocolate and hold my breath at the same time.

There appears to be two major (and interrelated) trends in networking cards (NICs) -- capabilities increasingly migrating from the "system" to the NIC and a convergence of previously distinct technologies.

The capability migration is seen with an increasing amount of security fucntionally and offloads (e.g. ipsec and linksec), IO (e.g. iSCSI), virtualization (chip level support for virtual machines via classification, multiple MAC address support, multiple TX and RX rings, etc), protocol offloads (LSO, LRO, Checksumming, Jumbo Frames, etc.), etc. capabilities. These allow commodity NICs to enhances performance, increase flexibility and increase system capabilities which can be implemented in a way that are largely transparent (e.g. requiring no human intervention or application changes).

The convergence aspect of the message is essentially that demarcation between classical technology boundaries such as security, virtualization, management, IO, networking, etc. are disappearing. The goal for Intel (and other vendors) to converge the technologies such that a single NIC can support multiple virtual machines on one end and _single_ fabric on the other end that also includes classical networking, plus IO (e.g. iSCSI), management, security, etc. Additionally they are trying to extend the system network further into the network using things like Linksec, BCN (essentially gathering stats to better tune congestion control, etc.).

The biggest challenges that we are faced with in the world of more intelligent NICs is first how to make much of that intelligence transparent to the consumer (people, applications, etc.) and secondly how to make the capabilities that are visible easier to manage.

Congratulations to the Nemo Project Team (including those who are no longer at Sun)! It was truly a team effort, required alot of hard work, and in the end paid off. I am glad I could be a part of a team who drives innovation that has a significant impact on our partners, customers and Sun's bottom line.

2006 Chairman Award Recipients

Eric Cheng
Nicolas Droux
Carol Gayo
Darrin Johnson
Sunay Tripathi

Innovation Title

Nemo: High-Performance, Feature-Rich, and Easy-to-Use Device Network Driver Framework

Innovation Description

Solaris 10 delivered a new framework for the network stack (project FireEngine) that made Solaris competitive with Linux and won the hearts and minds of our customers and developers. However, the FireEngine stack could not realize its full potential with the existing, and dated, DLPI-based device driver framework nor could it drive the next generation of network technologies (e.g. 10 Gigabit Ethernet, TCP-IP Offload, wireless, etc.). A radical new approach was needed. Enter Nemo.

The Nemo architecture (external name GLDv3) provides the traditional DLPI interface to the network layer services, while providing a high-performance, direct-function, call-based interface. It virtualizes the Data Link layer of the network stack. There is no longer a one-to-one correspondence between network interfaces and devices.

This leading-edge technology not only increased SMI's intellectual property (filing 12 patents), it lays the groundwork for innovative growth in the network stack virtualization and resource control areas.

Nemo improves SMI's bottom line because it:

• Dramatically increases ISV/IHV adoption of Solaris x86/x64, including Nemo-based drivers for Intel, Broadcom, Neterion (10Gig), Chelsio (10 Gig/TOE), Level5 Networks, Nvidia, etc.
• Demonstrates industry-leading through-put networking.
• Provides tangible out-of-the-box performance improvements critical to key markets (Financial, Telco, HPC, etc.) without any changes required by ISVs or customers.
• Significantly reduces the time needed to write, QA, and support device drivers.
• Provides marketing buzz around Solaris due to added performance and functionality.

In summary, John Fowler, EVP of NSG, best summarizes the value of Nemo: "Nemo kicks ***! Delivering performance, features, and making it easier for our partners to support us is a recipe for success."

I had a chance to visit the San Diego Zoo a couple of weeks ago and enjoyed it tremendously. For nostalgic reasons the Brookfield Zoo near Chicago is still my favorite but the San Diego Zoo is a very close second. The San Diego Zoo was very well laid out with alot of path, transportation, eating and resting options. The exhibits made every attempt to be "natural", allow you to feel close to the animals and their was a good variety of animals in general. This was my first chance to see a Panda which was great. Below are a couple of my best pictures from the visit. Enjoy!

Interesting links:

San Diego Zoo
Brook Field Zoo

Economics - study of how people use their limited resources in an attempt to satisfy unlimited wants.

I have always had a fascination with the Need for Speed and in particular to what extent people will do to attempt to push the boundaries of speed in it's many forms...throughput, latency, etc. However the first part of the definition regarding limited resources has had an increasing impact on the pursuit of performance. The "economics" of performance are being transformed.

Gone are the days of buying a multi-million dollar computer systems that are tuned for a couple of months to be optimized for a few applications that ran on it and also gone is the ability to spend months optimizing and tuning the application(s) for the particular idiosyncracies of the system they are being run on. Those specialized systems are being replaced with general purpose computers with "server consolidation" and "resource virtualization" becoming more and more popular. Additionally there has been a huge drive for horizontal scaling using lots of "cheap" (/disposable/replaceable/etc.) servers with cost to tune a system likely exceeding the value of the system in the first place.

The expectation has essentially shifted from buying a drag racing car to the out-of-the-box performance of a rally car or even a BMW M-series sedan. Customers have limited resources in terms of expertise, deployment time and engineering bandwidth but still expect to maximize the performance and utilization of the resource that they buy. Out-of-the-box performance can only be achieved by introducing flexibility and intelligence into the system to dynamically adjust to the needs of the customers applications including in hardware, drivers, operating system, compilers, applications, etc.

Also a critical element of out-of-the-box performance is eliminating the need for tunables. Can you achieve better performance for a particular application (or number of applications) with alot of tunables? We need only look back to the defination of economics to answer that the question itself is not relevant. Given the limited resources of customers and their unlimited wants the only answer is that the tuning must be dynamic to meet the many needs of many customers simultaneously. The economics for Sun is that "out-of-the-box" performance is becoming part of our engineering lifestyle and we strive to acheive the goal that out-of-the-box performance will exceed that of any manually tuned system.

Awhile ago my team and myself went a roadshow to a number of customers (60+) around the world to showcase "FireEngine" project (e.g. Enhanced TCP/IP stack) and Solaris 10 in general. We also took the opportunity to show case some of the projects we had planned (e.g. "Nemo" Network Interface Driver Framework, "Yosemite" UDP Performance Project), etc. with all of being very well received. However not wanting to rest on our laurels we proactively sought to understand their networking needs including the issues that keep them up at night, ways to enhance how they do business, and most importantly how could Solaris be a differentiator for them.

The distilled version of their collective requirements were fairly simple -- fairness, priority and resource control:

Fairness: Ability to allow various traffic types to share the bandwidth and associated compute resources in a fair manner
Priority: Ability to prioritize service, protocols or containers
Resource control: Ability to manage network bandwidth and associated compute resources for each service, protocol, or container

However figuring out how to deliver such capabilities in the networking side of the operating system was not that simple but that is why we like to get up and go to work every morning.

To address the needs of those customers we came up with the CrossBow project but once again we didn't rest on our laurels. We revisited many of the same customers (and more) to preview the CrossBow project and make sure it was appropriately vetted which it was. However we wanted to go even further than the vetting effort, oven normal Alpha and Beta programs, we wanted to take make it an open development effort.

Now is your chance to participate. Take a look at the OpenSolaris CrossBow project page at:

http://www.opensolaris.org/os/project/crossbow/

Subscribe to the CrossBow discussion alias:

Subscribe

And take a look at Sunay's blog:

Project Crossbow going live on OpenSolaris

Colm MacCárthaigh of RedBrick recently got one of the T2000 (CoolThreads, aka. Niagara-based) servers and has been taking it for a performance spin. Take a look at his outstanding blog at:

http://www.stdlib.net/~colmmacc/category/niagara/

His performance experiments and analysis are definitely worth a read. My hope to is to incorporate what he is doing into future development efforts for my Networking Performance Team.

Just want to give you a quick preview of some of the networking features that you can expect in next Solaris 10 update when it releases:

• Nemo – GLD v3 high performance and feature rich NIC driver framework (S10U1)
• 10 Gig Networking – Chelsio 10 Gig Driver, enhanced Intel 10 Gig Driver, etc.
• 1 Gig Networking – Significant enhancements to Broadcom, Intel and Nvidia drivers
• Yosemite – Enhanced high performance UDP/IP Stack
• CrossBow SoftRings – Enhanced interrupt distribution especially for CMT architectures (e.g. Coolthreads)
• Greyhound – Enhanced SSL cache accelerator (replaces NCAS)
• IPv6 IPfilter – Added IPv6 support in IPfilter
• Policy Based Routing (PBR) - Support an application to set a routing policy of individual TCP connections, taking precedence over the routing tables
• Transparent Proxy Interface – API for user applications to interact with IPFilter so it can be used by ISVs

As you can see that we haven't "rested on our laurels" after delivering the enhanced TCP/IP FireEngine stack in Solaris 10 or even the new GLD v3 (aka Nemo) driver framework in S10 Update 1. The great news is that the roadmap of future updates even looks more promising.

Although my formal education was in Genetics and Biochemistry (minors in Chemistry and Computer Science), I have always been "into" computers starting with owning an Apple II+, doing computational molecular biology in college, and ultimately going the route of software engineering and engineering management. I have had the great fortune of working directly or indirectly on many of the most significant operating systems:

Cray - Unicos, Unicos-MAX, Unicos/mk
Silicon Graphics - Irix
Microsoft - Windows NT/XP
Linux - Various flavors
Sun - SunOS, Solaris

I had alot of fun developing diagnostics for the Unicos flavors and participating on the release teams for each of them. One of the most challenging experiences was being the release program manager for SGI's Irix 6.5 (codenamed appropriately Kudzu). While at Adaptec I had got to build a team that worked onsite at Microsoft and helped Adaptec get involved in the Linux community (back when it was pretty much still just a community). I have had the greatest fortune of managing both Solaris kernel and networking teams.

I can confidentially say (and back it up with the fact that I have stayed the longest at Sun) that Solaris team and OS are the simply the best. Solaris is a total package that includes performance, quality, application adoption, scalability, reliability, etc., innovates in ways that directly benefits customers (simply look at the Solaris 10 features) and is developed/supported by passionate and devoted engineers and management. I take pride in what we have accomplished and am glad that we aren't resting on laurels with a whole bunch of great new innovative technology in the pipeline.

Darrin