Paul Roberts' Musings

Thursday February 15, 2007

in.telnetd vulnerability: fun with zero-days

Its been an interesting few days in the vulnerability team at Sun. As most readers of this entry will likely be aware, a serious vulnerability in the in.telnetd daemon shipped with Solaris was announced publicly on Sunday, which led to a great flurry of activity to get the issue resolved.

Thanks to work of a number of dedicated engineers who were on the ball on Sunday, and who ran with this, within a few hours of it being reported, Sun had a fix reviewed and integrated into Nevada (the current Solaris development release). Alan Hargreaves then jumped on this and started the backport to Solaris 10 (older releases are unaffected), and he has given a detailed summary of his experiences on his blog (well worth a read).

When I got into work on Monday morning Alan already had IDRs (a form of interim fix packaged up into a patch) available internally and a Sun Alert draft was in progress. In addition he was getting ready to submit his RTI (request to integrate) for the patch gates, he just needed a code reviewer and our team was happy to oblige.

The next step was getting Sun's official response published, in the form of a Sun Alert and the IDRs containing the fix. It was an advantage in this case that the problem was relatively easy to understand for someone familiar with Solaris, and as a code reviewer of the backported fix, putting together the technical details of the Sun Alert was relatively straightforward. We double checked that the IDRs worked and then passed them along with the Sun Alert to the next stage of the process, waiting for them to be published.

By Tuesday evening we had official patches tested and published.

Obviously when going through any experience like this, where our team is tested and where it's working on precisely the kind of problem for which it was invented, we learn lots of things. One of the most important, and perhaps surprising, things for me was the attention that the team's blog received.

We're still finding our feet in terms of the value that the blog can add, and it is currently effectively just a mirror of the security Sun Alerts published to Sun Solve. However, what surprised me was that for many people this is the first place they look to learn about security vulnerabilities in Sun's products. Unfortunately due to a bug in the tool the notification about the telnetd Sun Alert did not hit the blog until a good few hours after it was published on Sun Solve, and this upset a few people. We have now learned that this resource is important to people and we are hoping to raise its priority within the team.

But there's a deeper lesson to learn than that. It seems that people were following the blog because they wanted up-to-the-minute information about the progress of the vulnerability and its resolution. They wanted to actually follow the story. And not just the facts of the case, but the feedback that I have seen about Alan's blog entry has shown that people are interested in the personal voice as well. Understandably SunSolve, which is a very professional site supported by a huge infrastructure, is not designed for that sort of thing. Sun as a whole has already learned this lesson with the introduction of blogs.sun.com, and now it looks like this message is starting to filter down to the security team too.

So in hindsight it seems that our team could start filling in the blanks between the movements of the big SunSolve machine with less formal communications via outlets such as our blog and the many OpenSolaris discussion lists. An example of the kinds of things we're thinking about are potentially putting up a draft of the Sun Alert onto the blog as soon as we have finished writing it, which in this case would have meant it was available a good few hours before it appeared on Sun Solve. Another request we're exploring is allowing comments on the security blog, so that it will be more of a conversation than an information drop.

This is a learning process, and we've got a long way to go. Perhaps the readers out there can help us along the way!