Darrin Johnson's Weblog: Need for speed...

My wife and I decided to get away for the weekend and head over to Half Moon Bay to have some good seafood, see the sites, pick up some pumpkins and simply relax. We decided to splurge and stay at the Ritz-Carlton in Half Moon Bay in part because of the tremendous brand reputation of customer service. Ironically, customer service certainly wasn't their strength with a comedy of customer service errors (13 to be exact) that individually could be written off but as a whole was a stunning disappointment. The manager and assistant manager quickly acknowledge the problems as being fault of the staff and were apologetic. The both repeatedly asked what they could do (beyond trying to resolve each individual problem as they arose). This brought to mind the question..."What is good customer service worth?" or put another way "What kind of compensation would mitigate the bad service?". Other questions also factor in..."Should cost of services/goods factor into the equation?", "Should brand reputation be a factor?", "Does the severity or frequency issues factor into it?", etc. These questions however lead to a natural desire to negotiate for the maximum compensation possible.

I however take a different approach. Instead of negotiating, I prefer to let the person know that it isn't up to me to ask for compensation for the trouble but instead up to them to decide what value I am to them as a customer. They need to understand the value of good customer service with the consequence of their decision having a direct impact on whether or not I remain a customer (and tell others). It is my belief that ultimately the payout is far greater in that the company is more likely to make the changes needed to eliminate the service issues in the future. In the case of Ritz-Carlton in Half Moon Bay, I genuinely believe that the message was understood and that I will gladly be a customer again. Additionally, although I had insisted that I am not looking for compensation for the issues, they took care of the room charges which was completely unexpected.

Here are my favorite RSS feeds on the topic of chocolate:

Chefs.com - Candy and Chocolate
NYT > Chocolate
The Chocolate Blog (website)
Chocolate & Cioccolato World (website)
Chocablog (website)
Chocolate & Zucchini (website)
Chocolate Blog (website)
Chocolate Obsession (website)

Please comment if you can recommend any other chocolate RSS feeds.

Recently I have encountered a phenomenon that I thought was reserved for actors.....being typecast. In this case being typecast as "just a manager" with the perception that once one becomes a people manager they have a lobotomy that limits their ability to function beyond certain boundaries. Those boundaries of course are not well defined but seem to be centered around the common tactical managerial tasks of status reports, performance reviews, project management, etc. Being "just a manager" seems to preclude having technical depth, contributing to the strategic direction, or having a broader understanding of technology or business in which we work. Having been a people manager for nearly a decade, I have prided myself on trying to master both the classic skills of a people manager as well as going beyond those boundaries to leverage my experience as a scientist, engineer and business person to add greater value to my team/organization/company and for my own personal fulfillment.

Fortunately the typecasting is the exception, not the norm, and has helped me to reflect on what is important to me and what I think should be important for my fellow managers. Don't limit yourself to being "just a manager" and take it upon yourself to define your own boundaries. Take on the role of being a strategist, technologist, leader, business person, etc. if only for your own personal fulfillment. Take pride in breaking the typecast by being able to add to your own credits by being a scientist, author, change agent, entrepreneur, technologist, inventor, visionary, etc.

Sun has launched OpenEco.org which is an online community to provide free tools and resources for calculating, tracking, and comparing greenhouse gas (GHG) emissions. Dave Douglas, VP of Eco-responsibility at Sun, says, "This project came about because we were trying to calculate our own carbon footprint ... and we found that the tools out there were proprietary or expensive, or else companies were building spreadsheets (to calculate their footprint) on an ad hoc basis."

http://openeco.org/
http://weblog.infoworld.com/sustainableit/archives/2007/09/sun_delivers_op.html

In popular media and technical journals we frequently see reports of the latest security weaknesses, hackers breaking into systems, services and data being compromised, etc. Professor Arun Sood, at George Mason University, has taken a novel approach to confronting these security threats. The fundamental approach of Self Cleansing Intrusion Tolerance (SCIT) is to create a known good version of a service (e.g. identity, dns, webserving, etc.) which is then replicated with some of the services put online and some held offline in reserve. Then periodically the offline copies are brought online with online versions being taken offline, scrubbed and ready to be brought online again. I don't do the technology justice so I recommend taking a look at his his website at http://cs.gmu.edu/~asood/scit/ and especially the listed publications.


Professor Sood also posted a great introductory article on Reavis Consulting Group's RiskBloggers.com on "Exposure Time - A Metric For Proactive Security Risk Management". The article discusses the critical consideration of the relationship between the intruder residence time (IRT) and the potential for loss as defined by the loss curve. The "exposure time" however, as Professor Sood asserts, is a more interesting metric. Unfortunately there is invariably a trade-off of cost (e.g. increased number of replicated services, additional equipment, etc.) and exposure time. If you are interested in services security and how to design low exposure time systems then this article and related links are worth reading.

For those of you who are dog lovers you will understand there is nothing like a dog that is so excited to see you that it's whole body is in motion, ready to launch up on to your lab, wet slobbering kisses to show how much you were missed, and maybe a few barks to make sure you are paying attention. What is even more amazing is when that dog isn't even yours and has that same excitement even when days, weeks or months may have passed since the last time you had seen them.

I can say that I was blessed to have known such a dog. A dog whose droopy eyes, long ears, short legs, etc. did not reflect the excited puppy at heart, tremendous intelligence and unexpected grace. Toby, the basset hound, died yesterday of cancer. What is stunning is that only two weeks ago he had all the excitment and energy mentioned above with absolutely no symptoms of his illness. Toby was strong until the end."

Toby will be missed by many. May he rest in peace! Condolences to his family and pack mates!

Toby as a puppy....


Toby at the dog park...

I find out recently that my favorite chocolate maker, Scharffen Berger, was purchased by the very commercial Hershey's and was stunned. Although it wasn't a life defining momment it did however give me pause to think about why Hershey's may have purchased a boutique chocolatier. Then I realized that many companies have been doing similiar acquisitions - beer companies buying up microbreweries, computer companies buying up gaming machine companies, soft drink vendors buying bottled water and specialized drink companies, etc. Unlike the trend with computer companies to provide customized solutions (e.g. configurations) as part of mass customization movement, this is more like "Mass Specialization".

Why would mass specialization be interesting? As is generally the answer it all has to do with money. People are willing to pay more for specialty products. In the case of Scharffen Berger people are willing to pay $4.00 to $7.00 a 3 oz. bar instead of $0.75 for a Hershey Almond Bar which likely means there is certainly better profit margin in the specialized bar. Then you factor in the economies of scale of Hersheys which can greater buying power, strong infrastructure, and infinitely better distribution channels. It is clear that the profit margin will only increase.

The most important question is ... "Will the quality suffer?" Only time will tell. However the large mass producing companies have a vested interest ensuring that the specialized product remains specialized to preserve their margin. I hope though that my $6.50 El Carmen Venezuela 75% Cacao Pure Dark Chocolate Limited Series Scharffen Berger chocolate bar will become slightly cheaper given the economies of scale. Unfortunately, or fortunately, I can't eat chocolate and hold my breath at the same time.

There appears to be two major (and interrelated) trends in networking cards (NICs) -- capabilities increasingly migrating from the "system" to the NIC and a convergence of previously distinct technologies.

The capability migration is seen with an increasing amount of security fucntionally and offloads (e.g. ipsec and linksec), IO (e.g. iSCSI), virtualization (chip level support for virtual machines via classification, multiple MAC address support, multiple TX and RX rings, etc), protocol offloads (LSO, LRO, Checksumming, Jumbo Frames, etc.), etc. capabilities. These allow commodity NICs to enhances performance, increase flexibility and increase system capabilities which can be implemented in a way that are largely transparent (e.g. requiring no human intervention or application changes).

The convergence aspect of the message is essentially that demarcation between classical technology boundaries such as security, virtualization, management, IO, networking, etc. are disappearing. The goal for Intel (and other vendors) to converge the technologies such that a single NIC can support multiple virtual machines on one end and _single_ fabric on the other end that also includes classical networking, plus IO (e.g. iSCSI), management, security, etc. Additionally they are trying to extend the system network further into the network using things like Linksec, BCN (essentially gathering stats to better tune congestion control, etc.).

The biggest challenges that we are faced with in the world of more intelligent NICs is first how to make much of that intelligence transparent to the consumer (people, applications, etc.) and secondly how to make the capabilities that are visible easier to manage.

Change - To undergo transformation or transition.

In my experience, management is laregly about managing "change" rather than managing people. Also leadership is largely about driving "change" by leading people in a certain direction. Sun has seen a rather large number of high profile transitions:

• John Loiacano leaving his roll as EVP of Software
• Scott changing his role at Sun
• Jonathan becoming CEO
• Rich Green rejoining Sun as EVP of Software

The transitions have also been closer to home losing a couple of key senior engineers and a manager, but also gaining a great new college hire and some senior engineers. Some may lament at losing good engineers or revel in gaining new ones or even express frustration that there is just to much change. I however believe that change is a vital element of progress and success (however you define it) is result of embracing that change.

The "I Ching" has a quote (loosely translated) - "Before the beginning of brilliance, there must be Chaos." I have found this to be very accurate. Through the transitions outlined above I have seen junior engineers step up deliver as if they were seasoned engineers, senior engineers come to life to become leaders influencing not only their group but a much broader organization, and lastly I have seen myself step up to take on roles that I had never really considered.
At the heart of change appears to be opportunity.

Congratulations to the Nemo Project Team (including those who are no longer at Sun)! It was truly a team effort, required alot of hard work, and in the end paid off. I am glad I could be a part of a team who drives innovation that has a significant impact on our partners, customers and Sun's bottom line.

2006 Chairman Award Recipients

Eric Cheng
Nicolas Droux
Carol Gayo
Darrin Johnson
Sunay Tripathi

Innovation Title

Nemo: High-Performance, Feature-Rich, and Easy-to-Use Device Network Driver Framework

Innovation Description

Solaris 10 delivered a new framework for the network stack (project FireEngine) that made Solaris competitive with Linux and won the hearts and minds of our customers and developers. However, the FireEngine stack could not realize its full potential with the existing, and dated, DLPI-based device driver framework nor could it drive the next generation of network technologies (e.g. 10 Gigabit Ethernet, TCP-IP Offload, wireless, etc.). A radical new approach was needed. Enter Nemo.

The Nemo architecture (external name GLDv3) provides the traditional DLPI interface to the network layer services, while providing a high-performance, direct-function, call-based interface. It virtualizes the Data Link layer of the network stack. There is no longer a one-to-one correspondence between network interfaces and devices.

This leading-edge technology not only increased SMI's intellectual property (filing 12 patents), it lays the groundwork for innovative growth in the network stack virtualization and resource control areas.

Nemo improves SMI's bottom line because it:

• Dramatically increases ISV/IHV adoption of Solaris x86/x64, including Nemo-based drivers for Intel, Broadcom, Neterion (10Gig), Chelsio (10 Gig/TOE), Level5 Networks, Nvidia, etc.
• Demonstrates industry-leading through-put networking.
• Provides tangible out-of-the-box performance improvements critical to key markets (Financial, Telco, HPC, etc.) without any changes required by ISVs or customers.
• Significantly reduces the time needed to write, QA, and support device drivers.
• Provides marketing buzz around Solaris due to added performance and functionality.

In summary, John Fowler, EVP of NSG, best summarizes the value of Nemo: "Nemo kicks ***! Delivering performance, features, and making it easier for our partners to support us is a recipe for success."

I had a chance to visit the San Diego Zoo a couple of weeks ago and enjoyed it tremendously. For nostalgic reasons the Brookfield Zoo near Chicago is still my favorite but the San Diego Zoo is a very close second. The San Diego Zoo was very well laid out with alot of path, transportation, eating and resting options. The exhibits made every attempt to be "natural", allow you to feel close to the animals and their was a good variety of animals in general. This was my first chance to see a Panda which was great. Below are a couple of my best pictures from the visit. Enjoy!

Interesting links:

San Diego Zoo
Brook Field Zoo

Although there is some question about the authenticity of the quote it is still interesting from a management or organizational behavior perspective:

"We trained hard, but it seemed that everytime we were beginning to form up into teams, we would be reorganized. I was to learn later in life that we tend to meet any new situation by reorganizing; and a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency and demoralization."

Petronius Arbiter (210 B.C.)

Reorganizing is often done for many reasons including new leadership, failing to meet objectives, attempting to increase efficiency, eliminating redundancy, performance management, latest management fad, etc. Sometimes the reorganization is driven by a desire to create a "matrix", "flat", "functional", "hiearchical", "hybrid", etc. organizational structures.

To acheive a successful reorganization I think a fundamental question always needs to be asked..."What problem(s) need to be solved?" Then is it is paramount to understand what is the root cause of the problem. This is probably one of the hardest steps because it often takes time, requires effective measurement ("If you don't measure it, you can't improve it."), and often results in redefinition of the what the problem really is. The other challenge in understanding the need for reorginization is to validate the assumptions being made. Once it is understood "where the organization is coming from" it makes it far easier to understand where it needs to go.

Where the organization needs to go is often very subjective with numerous organizaitonal structures that can accomplish the goal. However regardless of which structure is choosen it is essential to evaluate progress towards the goal(s) of the new structure. All to often the "fire and forget" approach is taken resulting in a "rinse and repeat" every 6 to 12 months. It is far easier to make slight course changes as the reorganization is occuring rather than having to reorganized the reorganized organizations.

Economics - study of how people use their limited resources in an attempt to satisfy unlimited wants.

I have always had a fascination with the Need for Speed and in particular to what extent people will do to attempt to push the boundaries of speed in it's many forms...throughput, latency, etc. However the first part of the definition regarding limited resources has had an increasing impact on the pursuit of performance. The "economics" of performance are being transformed.

Gone are the days of buying a multi-million dollar computer systems that are tuned for a couple of months to be optimized for a few applications that ran on it and also gone is the ability to spend months optimizing and tuning the application(s) for the particular idiosyncracies of the system they are being run on. Those specialized systems are being replaced with general purpose computers with "server consolidation" and "resource virtualization" becoming more and more popular. Additionally there has been a huge drive for horizontal scaling using lots of "cheap" (/disposable/replaceable/etc.) servers with cost to tune a system likely exceeding the value of the system in the first place.

The expectation has essentially shifted from buying a drag racing car to the out-of-the-box performance of a rally car or even a BMW M-series sedan. Customers have limited resources in terms of expertise, deployment time and engineering bandwidth but still expect to maximize the performance and utilization of the resource that they buy. Out-of-the-box performance can only be achieved by introducing flexibility and intelligence into the system to dynamically adjust to the needs of the customers applications including in hardware, drivers, operating system, compilers, applications, etc.

Also a critical element of out-of-the-box performance is eliminating the need for tunables. Can you achieve better performance for a particular application (or number of applications) with alot of tunables? We need only look back to the defination of economics to answer that the question itself is not relevant. Given the limited resources of customers and their unlimited wants the only answer is that the tuning must be dynamic to meet the many needs of many customers simultaneously. The economics for Sun is that "out-of-the-box" performance is becoming part of our engineering lifestyle and we strive to acheive the goal that out-of-the-box performance will exceed that of any manually tuned system.

Awhile ago my team and myself went a roadshow to a number of customers (60+) around the world to showcase "FireEngine" project (e.g. Enhanced TCP/IP stack) and Solaris 10 in general. We also took the opportunity to show case some of the projects we had planned (e.g. "Nemo" Network Interface Driver Framework, "Yosemite" UDP Performance Project), etc. with all of being very well received. However not wanting to rest on our laurels we proactively sought to understand their networking needs including the issues that keep them up at night, ways to enhance how they do business, and most importantly how could Solaris be a differentiator for them.

The distilled version of their collective requirements were fairly simple -- fairness, priority and resource control:

Fairness: Ability to allow various traffic types to share the bandwidth and associated compute resources in a fair manner
Priority: Ability to prioritize service, protocols or containers
Resource control: Ability to manage network bandwidth and associated compute resources for each service, protocol, or container

However figuring out how to deliver such capabilities in the networking side of the operating system was not that simple but that is why we like to get up and go to work every morning.

To address the needs of those customers we came up with the CrossBow project but once again we didn't rest on our laurels. We revisited many of the same customers (and more) to preview the CrossBow project and make sure it was appropriately vetted which it was. However we wanted to go even further than the vetting effort, oven normal Alpha and Beta programs, we wanted to take make it an open development effort.

Now is your chance to participate. Take a look at the OpenSolaris CrossBow project page at:

http://www.opensolaris.org/os/project/crossbow/

Subscribe to the CrossBow discussion alias:

Subscribe

And take a look at Sunay's blog:

Project Crossbow going live on OpenSolaris

The cliche "It's not what you know, but who you know." seems to be more relevant in the ever increasingly connected world with web tools such as Plaxo, LinkedIn, etc. becoming ever more popular. Regardless of the tools available, "social networking" is becoming ever more important and especially given the potential of the "social capital" gained by having an extensive network. Social capital is simply the resources available in and through social (business and personal) networks including information, ideas, leads, business opportunities, power, influence, support, trust, etc. Some people have also called these shared assets.

There are those who believe in the myth of individualism in which everyone succeeds or fails on the basis of an individuals own abilties and efforts. However research, and certainly my own experience, shows that having a strong social network (and the corresponding social capital) increases the chance of getting a job, pay and promotion, influence and effectiveness, etc.

People are often surprised when the spend a little time thinking about who is in their network including family, friends, team members, employees, managers, partners, customers, admins, etc. Looking at who you interact with frequently, who are you dependent on, who is dependent on you, and who you just like hanging out with is another way to generate the list. Finally and most importantly thinking about who should be part of your network and pursuing those connections will even further expand the social network.

If you want to read more on the topic I can recommend "Achieving Success Through Social Capital" by Wayne Baker.