Kool Aid Served Daily

I've got a bug for which I do not have a reproducible test case. But I'm pretty confident I found what is going wrong. I can run regression tests to show I haven't broken anything - but those same regression tests never tripped the bug in the first place.

The fix is:

------- usr/src/uts/common/fs/nfs/nfs4_stub_vnops.c -------

Index: usr/src/uts/common/fs/nfs/nfs4_stub_vnops.c
23c23
<  * Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
---
>  * Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
27c27
< #pragma ident      "@(#)nfs4_stub_vnops.c  1.3     07/10/25 SMI"
---
> #pragma ident      "%Z%%M% %I%     %E% SMI"
1751a1757,1769
>             * Someone is already working on it. We
>             * need to back off and let them proceed.
>             *
>             * We return EBUSY so that the caller knows
>             * something is going on. Note that by that
>             * time, the umount in the other thread
>             * may have already occured.
>             */
>            if (was_locked) {
>                    return (EBUSY);
>            }
> 
>            /*
1762,1763c1780
<            if (was_locked == FALSE &&
<                !mutex_tryenter(&net->net_tree_lock)) {
---
>            if (!mutex_tryenter(&net->net_tree_lock)) {
1814c1831
<            } else if (was_locked == FALSE) {
---
>            } else {

In English, the lock detection used to only handle when the lock was not being held.

What I want to do is force was_locked to be true at this point in both the original code (to verify I can trigger the panic at will) and also in my fix (to verify I have fixed the correct bug).

I can do that by adding the following code:

# wx diffs

------- usr/src/uts/common/fs/nfs/nfs4_stub_vnops.c -------

122a123,124
> int   nfsv4_mm_was_locked = FALSE;
> 
1750a1753,1755
>               if (nfsv4_mm_was_locked)
>                       was_locked = TRUE;

This is a global in the nfs module which by default does not force was_locked to be set. I can use mdb to change it on the fly:

# mdb -kw
Loading modules: [ unix genunix specfs dtrace cpu.generic cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci ufs mpt ip hook neti sctp arp usba fctl nca lofs cpc random zfs nfs fcip logindmux ptm sppp ]
> nfsv4_mm_was_locked::print
0
> nfsv4_mm_was_locked/W 1
nfsv4_mm_was_locked:            0               =       0x1
> nfsv4_mm_was_locked::print
0x1
> $q

Note that I do not want to add special code to check the environment, add something in /etc/default/nfs, or anything else which requires changing anything on a system. I leave it entirely in the kernel and I use mdb to control it.

I'm back from the week long Solaris 10 Kernel Internals course. I was talking to a co-worker after one of the classes, and he basically asked why I was in the class. I think the comment was that I could be teaching the course. While he was right, I've taught undergraduate OS courses and a graduate file system course, he was also wrong.

The main operating system I have experience with is Data ONTAP. And while it is based on NetBSD from the middle of the '90s, it isn't like any other Unix out there. In particular, it started out as a message passing OS (massive threading has since been added), it has no User mode, there is no concept of paging or swapping, and there are no processes. We spent the first 4 days in the class going over the transition between User and Kernal mode, paging/swapping, and process creation. I know about this stuff, but it felt strange. I also couldn't communicate the difference to my class mates.

The huge positive is that the OS model of Data ONTAP makes it quite fast. I don't know how many times the instructor commented on the need to quickly harvest process resources for reuse.

The huge negative is that Data ONTAP is not quite as flexible. You can't load any old application on it and run it. But as a security measure, this can't be beat.

The other main benefit of the course was the coverage of DTrace. The instructor used it to drill down into process creation. He wrote a bunch of macros and a bunch of printf statements. An example he used was in the fork/exec of a new process. He used several variants of a script to trace the various state transitions of both the parent and the child process.

When I taught, 8-9 years ago, Linux source code was available, but any kernel debugger was lacking. Okay, basically it was the equivalent of printf. Solaris source might have been available, but we used Linux because we were willing to support it at the cost. If DTrace had been available back then, I think it would have enabled more students to grasp what was going on.

I'd argue that while you can get to OpenSolaris source code, the associated kernel debugger, kmdb, still lacks. I want a symbolic debugger. I've had it before, gdb and Data ONTAP, and it is so nice.

I like DTrace, the technology behind it is simple and elegant. But, I think it overcompensates for the lack of a real easy to use kernel debugger. It provides on-the-fly insertion of printf into the code.

The Kernel Internals course went a long way to exposing me to using kmdb and DTrace with the core kernel. Where it went short for me was in that it only covered the core of the kernel. For example, the NFS code is in a different module. The course did not help me understand how to find out the module name, how to reference that module name, and how to make sure I could DTrace it.

But perhaps that is a need that only a small percentage of the students needed.

I don't consider the class to have been a waste of my time. I wish I had been able to attend it before I started fixing bugs. I even wish I had been able to take it as a NetApp employee - digging into any kernel is fun. And the lessons learned can apply to any other OS.

If you head over to SolarisInternals you would see that Jim and Richard are working on a second edition of their book. I almost bought the 1st edition off of Amazon, until I saw the 2nd was coming out.

Even better, Richard asked internally at Sun if people could serve as reviewers. So, I've been reading it in my free time.

Right now I'm working my way through the DTrace examples in Chapter 2. The nice thing is that I'm actually kinda getting paid to do it. I.e., my manager would love for me to know DTrace, mdb, etc, to be able to work on our bug backlog.

I haven't paid this much attention to detail on someone else's work since I was a reviewer for Morrison's Understanding Quantum Physics: A User's Manual. I actually sweated through 2 semesters of Quantum Mechanics first.

Anyway, the DTrace chapter is a breeze to go through - I like how they plan to walk you through observability tools before diving straight into the kernel. I must be too academic today, I keep on thinking of exercises they could add to help the reader really get a feel for DTrace's power.