Alan Hargreaves' Weblog

Open sourcing solaris article at CRN

Just came across the following on CRN:

Sources: Sun Plans to Open Nearly All of Solaris Source Code

Definitely a bit more of a tease. One of the more interesting paragraphs is:

Sources familiar with the company's plans told CRN at JavaOne 2004 that Sun is not going to simply open up bits and pieces of the millions of lines of code in Solaris, Sun's popular Unix-based operating system. The vendor plans to open up nearly all of the OS's source code, including, "all the rocket science," one Sun employee who requested anonymity said.

The comments attributed to Joe Lindsay look to me like he simply is not aware of what Sun has been doing with Solaris. Please Joe, go and have a look at the writeup on Solaris 10, even better yet, why not download and install Solaris Express, to see just what it is that we Sun Engineers are so excited about. While I would agree that "Linux would remain a popular alternative to Solaris", wouldn't it be nice to see how we have done some of the amazing things that are going in to Solaris 10? Surely a contribution such as Solaris to the open source community should not be so off-handedly written off. Oh, and by the way, in regard to the comment "even if Sun open sources versions of Solaris for all three of its hardware platforms, Sparc, AMD Opteron and Intel."; Solaris is the one source tree. It's all platforms or none.

The article also goes on to talk about the release of Looking Glass under GPL and a few other contributions.

OSnews.com also has a pointer to the article and (as of now) a few comments.

Posted at 09:48PM Jun 30, 2004 by Alan Hargreaves in Solaris Express  |  Comments[1]

When looking at crashdumps, never trust the source!

When I did my first advanced crashdump analysis course a number of years ago, the Instructor (George Hines) kept harping on one thing.

"Never trust the source."

What he actually meant by this is that what you see in the crashdump is not necessarily what you are going to see in the source code. There could be many reasons for this. I had this hammered home to me (again) just now.

We had a customer who had an XVR-100 graphics card in a Sunfire 280R running Solaris 9 with the latest kernel update and xvr-100 patch. This machine was consistantly panicing whenever the screen power management kicked in. Looking at the crashdumps showed that an address that should have contained a pointer to a stack variable (from further up the stack) contained rubbish. When we tried to dereference it, the machine panics. Reproducible 100% at the customer site.

After a few days of attempting to replicate this in our local lab after a few days of trying to follow code paths to see how this could have happened, I decided to revist the original crashdump. I went looking through pm_ioctl() to find any other reference to &devl just to check that I was looking in the right place. For this I went to the code and found that the following code fragment appears twice in pm_ioctl().

                                mutex_enter(...);
                                pm_enqueue_notify(...);
                                pm_enqueue_notify_others(&devl, ...);
                                mutex_exit(...);

OK, so lets go looking for any calls to pm_enqueue_notify_others(). The value I'm interested in will get put into %o0. Hmmm, I can find the calls to pm_enqueue_notify(), but not pm_enqueue_notify_others(). The assembly looks like:

pm:pm_ioctl+0x7dc:         call       genunix:pm_enqueue_notify
pm:pm_ioctl+0x7e0:         or %g0,  0x0, %o5  ( mov   0x0, %o5 )
pm:pm_ioctl+0x7e4:         call       unix:mutex_exit
pm:pm_ioctl+0x7e8:         or %g0, %l0, %o0   ( mov   %l0, %o0 )

Which set the alarm bells ringing. On checking the revision of the module in the crash we see

 id flags        modctl      textaddr     size cnt name
125 LI    0x30003cf0e28    0x780f4000   0x5466   1 pm (power management driver v1.101)

But the source was showing v1.104. Now, 1.104 should have been installed with patch 112233-12, which is what the crashdump reported the kernel as running. v1.101 was in 112233-11. It looks like 112233-12 may not have completed installing. To verify this I put the v1.101 binary onto my lab box and ran "xset dpms force off".

I got a panic identical to what the customer was seeing.

The fix should be to simply backout 112233-12 and re-apply it.

The moral? The assembly from the crashdump is what the system ran. The source code is a guide. Trust the crashdump first. Thanks George

Posted at 04:02PM Jun 30, 2004 by Alan Hargreaves in Solaris  | 

COO starts blogging

Wow, it's actually happened. Jonathan Schwartz has started a blog here on blogs.sun.com.

Welcome to the community Jonathan and thank you for the confidence in your staff to not only get behind this idea, but participate.

I look forward to seeing what you have to communicate.

Posted at 03:57PM Jun 29, 2004 by Alan Hargreaves in General  |  Comments[1]

Dtrace article and comments on OSnews

OSnews has listed an article from Sysadmin Magazine that came out earlier this month titled Dtrace -- Most exposing Solaris tool ever. There appears to be some good discussion on the article and on zealotry in the OSnews comments on it too.

The article is an interview between the Author and the architects behind dtrace (Bryan Cantril, Adam Leventhal and Mike Shapiro). They discuss the motivation behind Dtrace and give some examples of of it has been used inside Sun already to solve some hairy problems. The problem that they walk through in Chapter 9 of the paper is a beauty, and I don't think that we would have gotten to the root cause in anywhere near the time that we did without Dtrace, if indeed we could even have found root cause without it.

One of the folks commenting on OSnews also pointed to http://www.sun.com/bigadmin/content/dtrace/d10_latest.pdf which contains the latest version of the reference guide. I should also say that the forum hosting this is also worth a visit. It also contains a few other tidbits of interest such as the paper being presented at USENIX, a link to the original post by Bryan announcing Dtrace and a few other goodies.

It's also great to see that some folks still have faith in us as an innovator. Solaris 10 will justify this faith. Stay tuned, there is more to come yet :)

Posted at 01:39PM Jun 27, 2004 by Alan Hargreaves in Solaris Express  | 

Forbes follow up article on Open Sourcing Solaris

I just read an interesting article in Fortune where Jonathon Schwartz has responded to David Kirkpatrick on the Open sourcing of Solaris. The article is titled Why Open Source Doesn't Always Mean Free and Jonathon makes some interesting points and appears to have brought David around to thinking that this may actually be a good idea.

A couple of particularly interesting paragraphs were

What Schwartz's remarks underscore, and what I admittedly hadn't fully digested until now, is that open source means something quite different once it becomes a critical part of the enterprise infrastructure. Companies will pay a lot to ensure that their backbone systems are properly cared for, regardless of the development model under which the software was created. Nothing shows this more than the success of Red Hat. Its revenues for the fiscal year ended in February rose 39% to $126 million. And while it lost money the previous year, it made $14 million this time. And growth is accelerating. In the most recent quarter, ended in May, revenues grew 53%. More important, CEO Matthew Szulik has made a deft transition to a subscription and support model. Red Hat's enterprise subscription revenues are more than doubling annually. Since the renewal rates of these subscriptions exceed 85%, this is a great annuity. No wonder Sun wants a piece. In the past year Red Hat's stock has roughly tripled, and its market cap stands at $3.7 billion. Not bad for a company whose product is supposedly free. (Sun's market cap stands at $14 billion, and its stock is about 20% lower than a year ago.) Perhaps Red Hat will want to respond to Schwartz in a future column.

Posted at 08:51PM Jun 26, 2004 by Alan Hargreaves in Solaris  | 

Open Sourcing Solaris, where to discuss?

I just spent some time reading through some of the Comments to Andy Tucker's blog article Bigadmin forum similar to the one created for Dtrace or Zones.

Keep up the discussion folks, it's all good stuff and I look forward to seeing what is going to happen in the not too distant future.

Posted at 07:30PM Jun 26, 2004 by Alan Hargreaves in Solaris  |  Comments[2]

Got the car back!

With very little in the way of drame I got the car back yesterday (Friday) morning, gearstick now intact.

As the workshop who did the work was next door to my kid's school, I simply jumped on the school bus with them (the only adult apart from the driver), had the obvious upset of one child as I could only sit next to one of them (Jake got me this time, next time Lucy gets me) and wne to scholl with them.

Grand total spent getting it fixed was AUS$80, broken down into $60 labour and $20 for the gas and welding rod. I'm pretty happy with that.

Posted at 06:58PM Jun 26, 2004 by Alan Hargreaves in General  | 

Working the weekend

Well this weekend I get to cover the VOSJEC pager from the office during APAC hours.

A nice quiet day, I actually managed to catch up on a few of my own calls, including some more time on backporting some ufs fixes for Solaris 8. Got the first lot of testing finished on SPARC. I've still got to run the test suite on x86 and the full load test on both. Hopefully I can get this to be ready for code inspection within a few days.

I may also get onto some testing on another call involving power management on an f280r with an xvr100 video card and Solaris 9. I've got a similar box built up in the local lab; we just have to get the thing replicating the problem. Once I've got it replicated I can look at setting some kernel breakpoints and watchpoints with kadb. Unfortunately, booting with kadb disables power management if there is a graphical console on the machine, so I've had to find a way to force it back on. It looks like I can do this with

ok boot kadb -d
...
kadb: <return>
kadb[0]: pm_cfb_override/W1
pm_cfb_override          0x0        =         0x1
kadb[0]: :c
SunOS Release 5.9 Version Generic_112233-12 64-bit
Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.
Jun 26 18:43:07 f280r-d Use is subject to license terms.
...
WARNING:Kernel debugger present: see kadb(1M) for interaction with power management.
...

Posted at 06:42PM Jun 26, 2004 by Alan Hargreaves in General  | 

University antics and other associated humour

John Haslam reminded me of some of the things that we got up to in the days that I was at Uni.

The first one we did to a tutor who left himself logged in. This was in a lab that was using vt100 compatible terminals. The end result was that the tutor found that every third time he logged in, ten minutes after he logged in, his character set would change to graphics mode. He only twigged when he found a file in /var/tmp that he owned that had a number in it.

VMS and file versioning offered all kinds of opportunties. One such was to create multiple versions of login.com that would do their job and then remove themselves. One set of these would do the following.

1. Say something like "You shouldn't leave yourself logged in. Someone could come and remove all of your files... Like this", it would then proceed to display output that looked like all files and directories were being removed, then remove itself.
2. The next login.com would simply define dir as something like dir/exclude=*.*.* and then remove itself. No matter what kind of arguments they did to dir, it would show nothing.
3. The next one would simply write something like "Deleting account", remove itself and logout
4. The final one would simply remove itself then logout, leaving the account as if it had never been touched

By this time the frantic student would be contacting the admins to find out what had happened to their account. Of course, the account looked fine.

I was then reminded of the account of Robin Hood and Friar Tuck which I have reproduced below.

Back in the mid-1970s, several of the system support staff at Motorola discovered a relatively simple way to crack system security on the Xerox CP-V timesharing system. Through a simple programming strategy, it was possible for a user program to trick the system into running a portion of the program in `master mode' (supervisor state), in which memory protection does not apply. The program could then poke a large value into its `privilege level' byte (normally write-protected) and could then proceed to bypass all levels of security within the file-management system, patch the system monitor, and do numerous other interesting things. In short, the barn door was wide open.

Motorola quite properly reported this problem to Xerox via an official `level 1 SIDR' (a bug report with an intended urgency of `needs to be fixed yesterday'). Because the text of each SIDR was entered into a database that could be viewed by quite a number of people, Motorola followed the approved procedure: they simply reported the problem as `Security SIDR', and attached all of the necessary documentation, ways-to-reproduce, etc.

The CP-V people at Xerox sat on their thumbs; they either didn't realize the severity of the problem, or didn't assign the necessary operating-system-staff resources to develop and distribute an official patch.

Months passed. The Motorola guys pestered their Xerox field-support rep, to no avail. Finally they decided to take direct action, to demonstrate to Xerox management just how easily the system could be cracked and just how thoroughly the security safeguards could be subverted.

They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called `Robin Hood' and `Friar Tuck'. Robin Hood and Friar Tuck were designed to run as `ghost jobs' (daemons, in UNIX terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them.

One fine day, the system operator on the main CP-V software development system in El Segundo was surprised by a number of unusual phenomena. These included the following:

• Tape drives would rewind and dismount their tapes in the middle of a job.
• Disk drives would seek back and forth so rapidly that they would attempt to walk across the floor (see {walking drives}).
• The card-punch output device would occasionally start up of itself and punch a {lace card}. These would usually jam in the punch.
• The console would print snide and insulting messages from Robin Hood to Friar Tuck, or vice versa.
• The Xerox card reader had two output stackers; it could be instructed to stack into A, stack into B, or stack into A (unless a card was unreadable, in which case the bad card was placed into stacker B). One of the patches installed by the ghosts added some code to the card-reader driver... after reading a card, it would flip over to the opposite stacker. As a result, card decks would divide themselves in half when they were read, leaving the operator to recollate them manually.

Naturally, the operator called in the operating-system developers. They found the bandit ghost jobs running, and X'ed them... and were once again surprised. When Robin Hood was X'ed, the following sequence of events took place:

    !X id1

    id1: Friar Tuck... I am under attack!  Pray save me!
    id1: Off (aborted)

    id2: Fear not, friend Robin!  I shall rout the Sheriff
         of Nottingham's men!

    id1: Thank you, my good fellow!

Each ghost-job would detect the fact that the other had been killed, and would start a new copy of the recently slain program within a few milliseconds. The only way to kill both ghosts was to kill them simultaneously (very difficult) or to deliberately crash the system.

Finally, the system programmers did the latter --- only to find that the bandits appeared once again when the system rebooted! It turned out that these two programs had patched the boot-time OS image (the kernel file, in UNIX terms) and had added themselves to the list of programs that were to be started at boot time.

The Robin Hood and Friar Tuck ghosts were finally eradicated when the system staff rebooted the system from a clean boot-tape and reinstalled the monitor. Not long thereafter, Xerox released a patch for this problem.

It is alleged that Xerox filed a complaint with Motorola's management about the merry-prankster actions of the two employees in question. It is not recorded that any serious disciplinary action was taken against either of them.

Posted at 12:33PM Jun 23, 2004 by Alan Hargreaves in General  | 

Capturing the entire text segment in a corefile

Eric Schrock talks about corefiles in his blog. A few months back I wrote infodoc 72790 on how to use coreadm to capture the whole text segment within a corefile.

Usually when you get a core dump and you ask Sun to have a look at it, we will ask for a whole host of other things (e.g. all libraries that it liks with and the binary). Capturing teh text segment actually grabs all of this so we don't need to ask for it; it is simply available.

This functionality is only available in Solaris 10 and is certainly going to make looking at corefiles substantially simpler.

Posted at 08:30AM Jun 23, 2004 by Alan Hargreaves in Solaris Express  |  Comments[3]

Arrrgh

Decided to work from home today as the kids were going on a school trip down to Sydney to visit the powerhouse museum and have a ride on the light rail and monorail; and needed to be at school well before the normal bus would get them there.

Got them on the bus; went to drive home to start work and had the gearstick come off in my hand.

Sigh, at least it happened in front of a Service station with a workshop in it. They'll have a look at it today or tomorrow. Meanwhile, I ended up walking down to the shopping centre, grabbing a coffee and catching a bus home.

I've still got to work out how to get in touch with my wife to tell her that I can't give her a ride home from the station today

Maybe tomorrow will be a better day.

Update

Managed to get in touch with my wife, ... Apparantly I'm dead. :|

Posted at 10:22AM Jun 22, 2004 by Alan Hargreaves in Kids  | 

Wow, friends had a "little" boy

Just had an SMS on the way into work this morning. About 2 hours ago (6:10am Australia/NSW), one of my Uni and IRC friends from waaaayyyy back had a "little" boy; 11 lb 10 oz.

Welcome to the world Stephen William Eric.

Congratulations Leese and Ian.

Posted at 08:17AM Jun 21, 2004 by Alan Hargreaves in Kids  | 

On#Sun Articles I've written

After writing the previous entry pointing to an On#Sun article I wrote on /etc/system, it occurred to me that I should add some links to these in the public bookmarks on my blog. I just finished doing it. I hope that someone finds it useful or enlightening; entertaining would be nice too.

On#Sun is the magazine published by Sun Australia for Sun customers roughly every two months.

I've just submitted brief article on Solaris on x86 for the next issue and I'm currently working on a longer article that I'll keep the title under wraps for for a bit.

Posted at 08:02PM Jun 18, 2004 by Alan Hargreaves in General  |  Comments[1]

/etc/system viruses (virii?)

Richard Elling wrote about /etc/system viruses. I also wrote something about System tuning and /etc/system in the Australian customer magazine On#Sun that adds a little to this.

The bottom line is that nothing in /etc/system should be considered as boiler-plate. Just because a setting works well on one system does not mean that it will work well on another, let alone if the other system is running a different release of Solaris.

Posted at 11:34AM Jun 18, 2004 by Alan Hargreaves in General  | 

Student of the week!

My little girl (5 y.o.) was awarded student of the week for her kindergarten class yesterday. She managed to tell her big brothers but not mummy or daddy. Mummy found the ribbon on the floor this morning after Lucy went to school.

Written on the ribbon was the following:

Lucy is a joy to have in class. She is a quiet worker.

Go Lucy!

Posted at 06:32PM Jun 17, 2004 by Alan Hargreaves in Kids  |