Managing on the Bleeding Edge Eric J. Ray's Weblog

Wow! It's great to finally get to talk about the OpenSolaris 2008.11 release and the tremendous accomplishment that it represents. I've been watching the OpenSolaris Install Development team (my team!) working on this for months--starting before the dust had even settled from the 2008.05 release--and they've just done a wonderful job. 

What's cool about this release? Well, there's a whole bunch, from Time Slider to the great accessibility features to vastly improved hardware support, but I'd like to focus on the Install developments, simply because that's closest to my experience (and my team's incredible effort). What did they do for this release?

Well, first, there's the AI (Automated Install) project, which delivered a great prototype of the hands-free install features that we plan to be supporting in the next release. It's a huge step and a vast amount of work, complicated by some staffing changes mid-stream. The team pulled together and really made huge progress. I do expect that there will be some rather substantial changes for the next release, because I keep talking to engineers who have an ever-growing list of "well, if I can, I want to fix" tasks for the next few months. That said, even in the current form, AI is a super achievement and a great foundation for our planned SPARC support and other enhancements. This project took the lion's share of the engineering effort from the Install team over the last several months, and it shows the commitment.

Next, there's DC (Distribution Constructor), which underwent a complete rewrite from the prototype code we delivered in May to final, supportable, and polished code for this release. Watching the engineers on this project was fun, because they just kept  pushing harder and harder to get the code done, and done early enough to allow some stabilization time before the rest of the code from all of the other parts of OpenSolaris hit. Obviously, as we're creating our own release engineering tools--and your tools, as a member of the community--it's a big task, and one that's very intolerant of mistakes. It's clear from how quickly the post-2008.11 improvements are being completed that this code is a significant milestone, and a great place for anyone wanting their own OpenSolaris-based distro to start. 

 ICT (Install Completion Tasks) is the un-sung hero here--the engineers who drove this simply saw some loose ends from the last release that needed attention, and they dove in and got it done. It wasn't sexy or exciting, but it really needed to be done, and the effort here shows in the overall fit-and-finish of the 2008.11 Install technologies. Thanks, guys!

 Snap Upgrade, finally, is my biggest "wow!" from this release, I think. (That changes, based on what I've looked at last, but for now, it's the one.) I just find it incredible that I can install a system with the 2008.05 release, and do a full upgrade, while the system is running, in only a few minutes, and easily--safely--choose to move forward, or stick with the old bits. After ~15 years with various Unix and Linux flavors, and the trepidation that system upgrades always bring with them, it's mind-boggling that upgrades now take virtually no time, accrue virtually no risk, and just work. From the engineering management standpoint, the most impressive part is that the engineers working on this project made it look utterly simple. No sweat, no drama, no hassle--just hg commit after commit after commit, and results in spades.

My comments here cannot be complete without acknowledging the rest of the folks who helped make these accomplishments possible. OpenSolaris Install benefited from having really good support from the QE, Tech Pubs, and HCI teams, and many other groups across the organization. Some of the team members really pitched in behind the scenes, in non-obvious ways, to make it possible for these teams to succeed. And the teamwork within the organization to ensure success just delighted me.

 Honestly, I cannot imagine being any prouder of a team than I am of this one--working with these very talented, hard-working, and diligent engineers is utterly delightful. 

 Thanks so much, to all of you--you've done a great job, and you're already delivering for the next release. Wow. 

For the last few weeks, I've been exposed to a remarkable number of communication situations that some have considered challenging. People (direct reports, colleagues, friends--all over the place) have pinged me to talk about their situations and ask how I thought they should handle them. Of course, the sub-text for "how should I handle this" reads like "how do I spin this". Perhaps we've just had too many months of political campaigns, but I really don't see why the obvious answer seems so elusive.

In every single situation over these last few weeks, I've counseled just laying it all on the line, and have been baffled about why that wasn't the obvious response in the first place. There's no harm, no shame, in saying no, declining an invitation, or explaining that things just won't work out.

"Thanks so much for the invitation, but I've just got too much on my plate to sign up for anything additional at this point."

"I appreciate the offer of help, but you're offering to help with something that isn't really important anyway--I bet your time could be better used elsewhere."

"I've love to participate, but have other plans already for that day."

"Sorry, but I'm just too damn busy."

"Wow, that sounds like a great idea. I guess I didn't make my objections sufficiently clear. Can we talk about this?"

"I'd love to offer you a better role, but this just doesn't look like a good fit."

"I'm really concerned about how things are going on your project."

I guess that I understand--somewhat--the reluctance to hurt feelings or be inappropriately blunt, but I think it's so much better to just be upfront with things than to hide them behind a veneer of fluff that obscures the real meaning.

Ask me how I spent my Saturday morning...go on, ask...

Well, if you insist, here's why I spent my morning reconfiguring my whole home email setup.

Yesterday morning, I noticed that my usual morning email supply (in my personal account) was rather sparse. All of the election email, kids grade reports, and daily reminders of stuff, all of it--it was conspicuously absent. After doing some investigation, it became clear that email just wasn't passing from the spam filtering service I use to my home network. After ruling out all other options, I ended up on the phone with Comcast, only to learn that "they're having problems with port 25". Sigh. I decided to let things sit for the day, and hope that they'd get it fixed. I'm tough. I can handle a day without email.

Well, they didn't fix it. After googling for "Comcast port 25", without quotes, it became pretty clear that I wasn't likely to get the problem fixed, and I'd have to work around it. (In brief, they seem to have started blocking all traffic on port 25, regardless of the direction or the purpose. I don't even send mail on port 25, but SMTP does use port 25 to get mail to me.)

I ended up having to configure a new mail server (Postfix) in a Solaris Zone on a system that we have at a colo facility. That system now accepts incoming mail from the spam filtering service and spools it. Then I had to set up a MTA (UWimap), with SSL, of course, to be able to get to that mail. I then set up fetchmail to retrieve that mail and feed it back into the home server, so we can all get our email again, from the home server under the stairs. (Of course, in the process, I sorta set up a mail loop that resulted in 40,000 extra emails for my wife--whoops. Had to fix that too.)

As I was tinkering, my wife asked "what do normal families do in these cases?" Well, I guess they just continue using Gmail or Hotmail or whatever, and life goes on. I dunno. Maybe I've been a geek for too long, but the idea of not running my email service myself just seems odd.

Although today, I'm thinking I might want to give that a try.

It's fun watching a great manager at work. From the first time we met, I knew this manager was out of the ordinary. Why? Well, when we first met, I'd really only expected that he'd have time for a prefunctory meeting, but we actually talked for nearly 2 hours. I surveyed the books on his bookshelf, and they were nearly all good managment books (as opposed to the more dated or subject-specific volumes I expected). 

I've now known this manager for nearly 2 years,  and I continue to be impressed. I see him in the hall, greeting his directs by name--and greeting people far removed from him, but still in his organization, by name as well. He knows what people like, what motivates them, and what their issues are, and his behavior reflects it.

Far more important, though, is how the overall organization reflects this manager's priorities and values. It's an organization where (it appears) that everyone really cares about all of their peers as well as the rest of the organization. It's a place where visitors are courteously greeted at the door, and welcomed into the building. It's a place where a minor concern or question is treated with respect and sincerity, and where other people from this manager's network are brought in to help drive resolution to the issues. 

Moreover, the results are demonstrably excellent. Customers are happy, stakeholders are satisfied, and deliverables exceed expectations. What more could anyone ask?

I had really never expected to see this kind of an organization outside of a hypothetical case study, or possibly some apocryphal story, let alone where I'd get to interact with it on a regular basis. For a lot of reasons, I'm gratified that Frank is as good as he is, and that he helps my son's elementary school be someplace special.

 Thanks, Frank!

Until a few weeks ago, I had over 20 direct reports, which is simply exhausting. I didn't realize how draining it was until relief arrived (thanks, Dana!), but since then I've been recuperating and trying to regain my bearings. I realized that things had been more out of control than I thought after I called one of my directs just to see what was happening, and he thought there must be a problem or I wouldn't have called. Sigh. Since then, in addition to having "just because" unscheduled chats with people, I've also been looking for different--supplemental--ways to "sharpen my saw", as it were. 

I tend to keep the local bookstores well in business with my regular business and management book purchases, and my RSS reader is full of the latest technology angles. Really, I thought I was pretty up to speed on what was happening in the world around me, and keeping up with the latest from Rand, Joel, and others, but I was surprised by a spectacularly good resource for management tips that I've apparently overlooked for years.

Enter Manager Tools.

 I was browsing podcasts, looking for something that wouldn't rot my brain or require a full-time commitment to keep up with--I just wanted something to encourage me to think a bit and look at things a different way. I tried four or five different ones, and rejected all of them within the space of a single walk with a dog. Then I hit Manager Tools, and was immediately hooked. (Oddly enough, I got hooked by the "How to pack for a trip" podcast, which isn't exactly the kind of information I was seeking....but it was awfully informative anyway.)

Since then, the guys from Manager Tools have come along for most of my morning walks, and I've sometimes had to walk around the block an extra time to get to the end of the show...

 What's good about it? Good question. First, I find their perspective simply validating--they say the same kinds of things that I say (or think), but better. Second, I haven't listened to a show yet that didn't give me new ideas, perspectives, or a better way to articulate something that is important to my team or me. Third, they've provided all kinds of tools and tips that I've been able to put to use immediately--same day, in some cases.

(One particular podcast left me with the nagging thought that there was something key that I hadn't retained, so I had to listen to it again. The phrase was "behavior is believable", in the context of walking the walk, not just talking the talk. "Behavior is believable" is such a succinct way to phrase it, though.)

On the down side, though, I have spent an awful lot of time promoting the site and podcast. But now I can just point people to this blog entry, right?

Yesterday afternoon I had a fascinating chat with a new acquaintance who also happens to be in the high tech field. Unsurprisingly, we spent a fair amount of time comparing notes and views on a number of different technology and management issues. 

 What was in fact surprising to me is how this team and organization has quietly changed over the last couple of years. I've--obviously--noticed the changes as they've happened, but talking to an outsider about what we're doing really drove home the point that our world has significantly changed, far more than I had realized. The cumulative effect is astounding.

 I'll be writing more about these changes later on, but just a sampling of the topics that had the "wow, you're doing that?" impact include:

• Moving our whole development process outside the firewall, including project team meetings, design discussions, code reviews, and the source repository
• Changing development languages from mostly C and shell scripts to Python
• Significant use of virtualization technologies for development, with investigation into further applications for build environments, test setups, and other angles.

As the Solaris Install team continues moving forward with the new install technologies, I expect that this list will continue to grow, but I'll be trying to step back far enough to see what's worth discussing here. Little by little, it didn't look significant at the time, but in aggregate, this is just remarkably cool.

In looking back at the last year, and reflecting (more on that later), I've come to a conclusion:

Far too many social networks are sucking up way too much of my time. I give up.

I'm certainly not in the 1000s of "friends" (or whatever) on MySpace, Facebook, etc., but the volume of invitations, pokes, and related stuff is simply too much, so I'm resolving to consign all invitations to new social networking sites to /dev/null. Heck, I may even do that with additional invitations and connections on existing sites. I see freedom just ahead....

Now, there are a few advantages to these sites--I've reengaged with several old friends and acquaintances that I might not have connected with had it not been for various social networks, but then again, I've also not sent thoughtful letters or emails to a whole lot of friends and family that I neglect. Hmmm.

While I'm at it, maybe I should give up on other time wasters. Let's see...as I write this blog entry, the Today show blaring in the background, we're on the third day of "tiger kills human" analysis, complete with insightful questions like "are you sure the tiger cannot jump out?" Meanwhile, Benazir Bhutto was assassinated about an hour ago, yet that hasn't even been mentioned. Good use of time? I think not.

Likewise, my Google Reader is overflowing with feeds and items that are arguably interesting...but not necessarily more valuable than the things that I'd be not doing while reading them. Looks like there's more housecleaning to be done there.

Well, I still have a week or so to finish up my resolutions, but this looks like a great place to start.





 

 As I look back at getting the home network server moved over to Solaris, one of the things that's  most remarkable is the fact that I didn't have to build anything from source. I'm pretty sure that's never happened to me before, even on some of the more popular Linux distributions.

Thanks, Dennis Clarke and Blastwave (http://www.blastwave.org/). Virtually everything I needed or wanted was either included in the Solaris distribution I choose, or was at the end of pkg-get. As a result, all of the time and effort in the process was in tweaking things like my Samba configuration and the like, to account for the fact that the kids are now users of the system, so it needs to be a bit more bulletproof than I've bothered with in the past. That's really cool, particularly given the time that I've spent in the past crawling through various Catch-22 permutations (on lots of different systems) to get a server in the state that I want it.

The one exception in terms of software availability was Unison, which is a file synchronization utility that I like for cross-platform file syncing. I considered setting up one of the Rsync clones on the kid's computers, but didn't feel like messing with the ssh agent complexity needed for that to work appropriately. Even getting Unison running under SMF wasn't a big deal, though--that's just cool.

 

After literally years of good intentions, I've finally managed to get my home network server flipped from an aging Linux distro to Solaris (Solaris Express, 12/06, to be specific)...whew. It's funny how well-entrenched technology gets, and the "if it ain't broke, don't fix it" rule of thumb takes over.

 

On thinking about it, I believe that we've had a home network server for about 12 years now, ever since we had a 10base2 network, Windows for Workgroups, and a honking Windows NT 3.51 server. That server was actually the impetus to getting into Unix in the first place-- we hit one of those nasty BSOD events that totally trashed the file system, and I had to rebuild the server and restore everything from scratch. Again.

I headed out to the computer store and found a copy of Yggdrasil Linux, along with a book to help out, and went home to get to work. It was a heck of an interesting learning process. I had done some sysadmin work on Ultrix and AIX systems in a previous life, and spent a little time with Sun products, including the spiffy SparcStation5, which just smoked everything else I'd seen. I hadn't, however, ever used a *nix system that was all mine. It was fun!

As I recall, it probably took a week or so to get things up and running sufficiently to have a network connection, X running, and Samba sharing files with the Windows boxes. Cool. (Funny how the time to configure doesn't change, although the scope of what you can configure in that time does...it's Moore's Law in reverse, I guess.)

Over the years, we migrated from one Linux distro to another, from Yggdrasil to Slackware to Redhat to SuSE, usually based on either cool new claims or hardware death that provided the opportunity to upgrade. Recently, though, life's just been too busy to upgrade, and the existing server wasn't broken, so just kept on going. (There's clearly a challenge here for Sun as a whole--when the existing systems are good enough and not actively failing, it's hard to get the needed catalyst to actually upgrade.) Fortunately, a fading Dell PIII server provided the impetus to put a used Xeon to work and reduce the server sound level by 30 dB.

With Solaris working beautifully on most x86-class hardware, not to mention that I've been working in the Solaris organization for about 7 years, it was clearly well past time to get the home network on Solaris. Over the next few postings, I'll try to share some of the things I did to get the server up and running, and happily serving the family needs. Stay tuned...

 

Well, yet again I was rejoicing about finally being Windows-free, and yet again my celebration was premature.

This time, the cause was my new Palm Treo 650 (using Verizon). (Thanks, honey, by the way--I really do appreciate it.)

Why? Well, I've been, as we say, edgemail-ed--my mail server and calendar server have been moved to the latest Sun ONE Mail Server and Sun ONE Calendar Server, and put on systems that are available both inside and outside the Sun firewall. Fundamentally, that's good, because it's possible to get calendar information and mail from anywhere, without requiring a VPN connection. However, it turns out that the only way I could find to synchronize the Treo with the Sun One Calendar Server is to use the Sun ONE Synchronization 1.1 Patch 2 download. And, to use it, I have to sync MS Outlook (really) with the calendar server on a Windows system at home, then the Palm Desktop to sync Outlook to my Treo. Sigh.

One option, of course, would be to fall back to Sun's traditional dtcm, which is an antique CDE-based calendar manager. On the other hand, it's important to keep poking at some of the newer technologies to help get them to the point of working with Solaris.

I hope I'm wrong--if so, please let me know. Otherwise, I'll keep complaining and trying to find someone who thinks this would be a cool problem to solve. (PilotManager and pilot-link work just fine for me on Solaris--I just don't have a conduit to sync my current calendar.)

Well, there's no time like the present to (finally) start blogging, right?

For context and perspective (it might come in handy), here's a bit about me.

I've been at Sun for over 7 years (yes, that's enough to hit that max vacation accrual that it doesn't look like I'll have time to use), and am just starting a new gig as the manager of the Solaris Install team. Previously, I had a brief stint managing labs for the Network Storage organization, and prior to that, I managed a variety of lab and QE (that's quality engineering==test development) groups within the Solaris organization. Before that, I was a senior technical writer, also working with the Solaris organzation (primarily working on Solaris Volume Manager documentation).

Before that? Wow...that's a whole different life. I worked with my wife co-authoring commercial computer books (Dummies books, etc.) as a full-time gig for about five years, after bailing out of a very unfortunate company called WorldCom.

What's it like managing the Solaris Install team? Good question--I just started about a week ago, but there's a lot of cool stuff happening, and, frankly, a tremendous opportunity to really drive change in Solaris Install and in our customer's experience with Solaris. More on that later.