Responding to telnet

It's probably pretty obvious that I don't write many blog entries, but our recent activities around a telnet vulnerability have left me feeling a bit inspired.

What happened -

Last Sunday, an OpenSolaris community member (thanks "skunsul"!) pointed us at a link to a website that demonstrated a pretty severe security vulnerability in our telnet daemon. A couple of engineers who participate in that community saw the post, and one of them, realizing the severity of the problem, coded a fix, had it reviewed, and put back into the OpenSolaris source.

Then, taking advantage of time zones, an engineer in Australia picked up the ball and made the necessary code changes to Solaris 10. He also built an IDR (Interim Diagnostic and Relief) to provide to folks while we waited for official patches. He also wrote the first draft of a Sun Alert to inform customers of this problem.

By the time he went to bed, engineers in the UK were reviewing his code and the Sun Alert. Some of these engineers are part of my Security Engineering & Coordination team and were aware of the emergency procedures we've put in place for turning around response to 0-day (no advanced warning) security vulnerabilities in Sun's products.

By the end of the work day in the UK, the IDRs had been converted to ISRs (Interim Security Relief) which are the only things we ever make widely available on sunsolve.sun.com before running them through our normal test routines (which can eventually turn an IDR or ISR into a patch).

There were some minor issues during the day on Monday in the US that caused a bit of a delay in the ISRs and Sun Alert making it to the external servers, but by the end of the day in the US (a little more than 24 hours from the first report on a weekend) we had posted a Sun Alert and ISRs to fix the problem.

Even better, by the end of Tuesday, we had official patches released that closed the vulnerability.

Looking back -

Almost everything worked exactly as it should as we responded to this fire drill. We'd put in a number of processes to allow us to do quick releases of ISRs and Security Sun Alert, and everybody knew their part and did it well. Sun took great advantage of having engineers located around the globe, and work progressed throughout our 24 hour response without needing to keep folks up past their bedtimes.

Going forward -

We did learn a couple of things from this experience. There are some aspects of the final push to external servers that we can get faster on (though I'm happy to report that these fire drills are fairly infrequent around here) and we're working on those.

Another interesting bit is the number of people who were looking to blogs.sun.com/security as their primary source of information. We put that blog together to mirror the primary Sun Alert page (entries get posted after a Security Sun Alert comes out), but there's no reason we can't post drafts there ahead of the official release. In the future, we'll do just that in emergency situations. In this particular case that would have meant a draft of the Security Sun Alert posted sometime before most of the US got into work on Monday. It would have mentioned an immediate workaround (shut down telnet) and a pointer to the place on sunsolve where the ISRs would appear later in the day.

It's also worth noting that this is the first time the OpenSolaris community was the source of us finding out about a security vulnerability in Solaris. As nice as it makes my job when we have sufficient time to fix things before going public, I understand that in an Open world, we've got to be able to react to public postings, and I think we did a pretty good job this time.

Posted at 03:36PM Feb 16, 2007 by drscholl in Security Coordination  |