By now I'm sure it's painfully apparent to anyone who has read my blog I am a hard-core geek. In fact, I'm not just your run-of-the-mill hard-core geek, but a kernel geek.

Aside from our being born socially inept, a common affliction amongst hard-core geeks is an inability to communicate our thoughts and ideas (and emotions -- but I ain't goin' there!) effectively. While there are a few rare folks I've met out there who seem to have stumbled upon the magic cure to this nasty affliction, the hard-core geek who can communicate effectively is a rare breed.

As a professional software engineer, one of my fundamental beliefs is that the dominant factor between a successful project and an unsuccessful project reduces to the effective dissemination of key information. As a battle-worn veteran of several software projects, I've seen other projects flounder because the members of the team didn't understand or buy-into the mission, major deliverables or objectives weren't clearly defined, team members didn't agree on the requirements, or team members grew differing perceptions of what the final product would look like!

The impacts of ineffective written communications within a project often are magnified by conspiring factors such as geographically dispersed teams, rushed schedules (i.e. we don't have the TIME to define WHAT we're going to deliver!), and in extreme cases, recalcitrant team members!

In this post I examine the role of effective project documentation in executing successful development projects, and share why I believe this is even more important in an Open software development model than in traditional, closed-source projects.

The Traditional Software Project Model

The traditional project development model is usually the classic waterfall model of software engineering. The project is conceived, funded, and staffed. At that point the conception phase moves into requirements analysis, functional specifications are developed, and the architecture is determined. Next, the design is specified, an implementation created, and the implementation is checked against the specifications of the architecture. If everything checks out, the project is accepted by the sponsors and deployed.

At each stage of the traditional software project, new stakeholders are introduced, and these stakeholders have progressively lower levels of focus. If the communication of the concept is not clear, the architects may come up with a proposal which doesn't match the original concept. If the architecture is over- or under-constrained, the implementation may be untenable or incomplete. If the validation specification does not match the architecture the acceptance may not match the implemented product. Finally, if the end-user documentation does not match the architecture and implementation, the end-user will find using the product to be time consuming, frustrating, and costly. Anyone who has played the telephone game and observed how garbled even a simple message can become can appreciate how even a small defect in the communication chain of a complex project can lead to catastrophy in the final product!

The Open Software Development Project Model

The Open software development model differs from the traditional model in several imporant ways.

First, the concept and requirements typically come directly from the end-user(s), and are usually accompanied by a rough implementation in the form of a prototype. At this point, the community critiques the proposal and implementation. Community members attempt to reverse-engineer any unspecified requirements, and suggest new concepts for inclusion. In the case of large software projects, the development may shift back into the traditional model after this point. With smaller projects, the process may be more iterative, with the traditional validation phase leading to revised requirements, and the cycle repeating itself.

Second, in the Open model the stakeholders at each stage are different. In the case of the traditional model, the needs of the business (the business case) ultimately drive the scope and requirements of a project; the concept phase is nearly set in stone by the time the implementors start to engage. In the Open development model, the focus is usually more on the desired outcome of the end-users of the product. The requirements, and even the high level concepts are more free to evolve over time as the project progresses.

The Role of Communication in Software Development

In my experience, most traditional software projects make it through the first few stages (concept and requirements) with relatively few problems. By the time the implementors are brought on-board, the functional specifications are baked, and they map back to the original concepts and requirements fairly accurately. The big issues such as projected development and scope have been settled by the project bosses. Due to the senior-level of involvement up to this point, the accuracy and quality of specification documents is usually very good.

The really successful software projects become successful because they give the right level of attention to clearly communicating the key concepts and requirements. Software engineering experts have differing opinions on how much of the total effort should be given to a project's early phases, but my gut feel is that it is well over 25%, and may be as much as 50%.

To make an Open project which is proposed in the form of a prototype successful, it is even more important to keep in mind that the key concepts and requirements need to be thoroughly documented for the community to agree on them. There are several reasons for this:

• Engineers on traditional development projects are usually geographically co-located, and utilize high bandwidth forms of communication such as meetings and whiteboard sessions to reach decisions and agreements. Open development projects are almost always geographically distributed. E-mail and project web pages, which are lower-bandwidth forms of communication, are typically the dominant forms of reaching decisions and agreements; this makes increased accuracy and completeness a necessity.
• Unlike in the traditional model where the stakeholders are the product bosses, Open projects have the entire community (and the end users) as the stakeholders! If the community can't agree on the high level objectives and functional requirements of a software project, the project is doomed from the start.

Documenting Concepts, Requirements, and Architecture

The concepts and requirements comprise the big picture view of the software project. These define aspects such as the major capabilies of the product and the intended target market. The architecture defines the input/output methods, interoperability features, and user interface.

The first nail in the road at the early phases of a project is that seemingly unimportant details such as how the requirements were gathered are often not recorded. High-level tradeoffs made in mapping the requirements into architecture are often not well documented (if at all). As we'll see, this is a disaster waiting to happen down the road when the implementors hit a roadblock in the design.

A less obvious trap at this point is that the requirements and architecture are usually baked by experienced software engineers with a good, albeit incomplete, idea of the requirements of the implementors. On the other hand, implementors are usually less-experienced, and have a more localized focus -- meaning that there is an inherent opportunity for information loss when translating the requirements into software constructs. If an implementor doesn't understand how flexible a requirement is, or why the requirement is there at all, the implementation may be inadequate or over-constrained.

The latter effect is magnified in the Open development model, where the implementors do not work with the software as their day job; if they do, they often don't have the benefit of years of experience networking with engineers inside a major software company. If an Open development project is going to be taken on by less-experienced implementors, then clearly communicating the concepts, requirements, and architecture of the software is of paramount importance.

Documenting the Design

To help avoid pitfalls during the implementation phase, I believe it is essential that the design team thoroughly document the aspects of their implementation blueprints. The apsects which should be thoroughly documented include any and all of:

• Assumptions
• Constraints
• Design Trade-offs
• Design Decisions

Assumptions

We have all heard the cliche' about how assumptions are bad. However, reality dictates that no matter how well the requirements are written and the architecture is specified, there will always be room for interpretation. The idea behind documenting all assumptions regarding the concepts, requirements, and architecture is that the folks who created them can review these assumptions and provide clarifications when necessary. In some circumstances there may be holes in the original specifications which result in assumptions about major requirements! If in doubt, and it's open to interpretation -- document your assumptions about it! The time spent up front will save you a lot of frustration down the road by preventing an implementation going to the validation phase, only for the end-user to come back and say this doesn't work the way I expected it to!

There's also an advantage to catching holes in our assumptions early which is particuarly helpful in the context of an Open development community. We're all human, and as such we have tendencies to blame others rather than ourselves when things go wrong. Having members of a community come back after an implementor has worked long and hard on a problem only to tell him/her that the hard work that person has done is incorrect is a big blow to the implementor's pride. In the worst case, flame wars may insue, and the contributor ends up leaving the community. A lot of frustration and wasted effort can be eliminated if the community has the opportunity to identify misguided assumptions and correct them early on.

Constraints

One major difference I've noticed between experienced, high-caliber software engineers and average software engineers is that experienced software engineers know how to leverage constraints in order to optimize solutions to difficult problems.

Let me give a concrete example from a project I worked on a few years ago: a previous team worked hard at solving a difficult kernel problem in the area of virtual memory management. The team came up with an approach, worked for about a year, and had a working solution -- only to have their approach nailed to the wall because of a design flaw which could result in data corruption.

when I took over the project it didn't take me long to decide that the systemic problem with their approach revolved around the constraints. Specifically, the project team's approach failed because they had designed around a constraint that a specific DDI call had to remain supported. Because the DDI call in question was part of an obsolete framework, removing support for the call was the best way to solve the problem, and it removed the potential hole where an obsolete or incompatible device driver might cause kernel data corruption when the new feature was enabled.

In a way constraints are assumptions -- but they are assumptions that aren't related to the specifications. Which functions a library supports, which platforms will best run an application, and which programming languages to use may all be constraints of an implementation on which the requirements and architecture are entirely mute.

Another example of leveraging constraints to your advantage in an implementation is creating a dependency on a particular system library, rather than developing your own equivalent functionality.

Design Trade-offs

Some folks look negatively on design-tradeoffs -- they refer to the practice of making trade offs in a design as cutting corners. However, software is like any other engineering practice -- we need to balance effort and complexity against functionality. Finding a happy middle ground is a constant battle.

Most high quality projects I've seen do a good job of articulating their design tradeoffs. This is because there are two paths to detecting an incorrect design tradeoff: in the first case, the tradeoff is documented, and someone detects the problem before it poses a major problem; in the second case, the product blows up in the hands of the end-user! I don't know about you, but I prefer constructive criticism over catastrophic failure any day!

Design Decisions

I prefer to think of the design decisions as the politics of software design. This position may not be popular, but I think it gets the point across well. If others don't understand the reasoning behind your decisions, they will be more inclined to disagree with the outcome. If you do not have a strong case, and others hold enough clout over you, they may convince others that their approach is right, and yours is wrong.

The best defense against showdowns based on emotion and past experience (versus the facts!) is to clearly articulate the reasons you decided something is best implemented a certain way. In many cases, the decision is unimporant and ends up being arbitrary (for instance, how often to poll a descriptor). In other cases, the decision is the result of a thorough analysis. In yet other cases, there may be mistakes that you made others can find that show your decision is flawed. Regardless of the scenario, clear and concise documentation of the reasoning behind a design, and not just the documentation of the final design, will lead to a better end product.

Putting Project Communication into Practice

Start out with a communication plan

Document each project phase

Refer to your documentation often

Examples

The minimalist approach

Suppose that I am proposing a bugfix. As long as there is no impact of the design of the system, and no change in the user's experience, only a small design document is necessary. My design may be as little as a few paragraphs to accompany the diffs in an e-mail, explaining the root cause of the bug, how I arrived at the fix, and what the fix does differently to correct the abnormal behavior.

Use common sense! If a bugfix is a one-line change to correct a misspelling in a code comment, it's obvious to everyone that no design documentation is required -- the diffs speak for themselves. In anything more complex than the most trivial, mechanical change, there was some thought process involved. The idea is to get that thought process out into the open, where others can review not just the fix, but what you were thinking when you wrote it.

The micro-project

Suppose I'm implementing a new function call in a shared library. The end-users will be software developers. The stakeholders will be whoever requested the interface. I'd start by collecting the requirements, which probably are coming from one or more persons who proposed the interface. Once the requirements fleshed out and agreed upon, I'd draw up an interface specification. Since all library calls require a man page, I would do this in the form of a mock-up man page, and send it out for wider review, making sure there are several experienced software developers among the reviewers. Once the interface was baked, I would sit down and draft up an implementation proposal, listing all of the tradeoffs I could think of. Once I had that in place I would request a peer review of my proposed implementation, and after iterating on it I would continue onto the code.

The macro-project

As usual a one-size-fits-all approach isn't going to work, so you'll need to consider many factors -- size of the team, importance of the functionality, dependencies, schedule, etc. A macro project may range from an RFE (request for enhancement) to a full blown software package.

At a minimum, I would suggest breaking up the project into the following stages, each paired with a set of project documentation deliverables:

• Concept:
  • List of key requirements and features
  • Component block diagram
  • Interface specifications
• Design:
  • Dependency and data flow diagrams
  • Detailed specifications for each component
  • Constraints, risks, assumptions, and trade-offs
• Implementation:
  • Code comments
  • Big theory statements (to describe algorithms)
• Validation:
  • Test plan
  • End-user documentation

If you are putting together a good-sized project and you lack experience, I would suggest seeking outside help. The OpenSolaris community (general discussion list) is a good place to start.

Lately I've been bugged to blog SOMETHING other than my usual rambling technobabble. In case anybody cares here's what's rolling in my iPod this month. No, I don't always mix them. :)

• 3 Doors Down
• Acoustic Alchemy
• Alan Jackson
• Brooks and Dunn (you can't live in Texas and not love this stuff!)
• Chicago (IX and 17)
• Chris Botti
• Coldplay (A Rush of Blood and X&Y, X&Y is one of my top plays)
• Copland's 3rd Symphony
• Dupre's Organ Symphony in G
• George Strait
• Hoobastank
• John Tesh
• Journey (Greatest Hits, also a top play)
• Oliver Messaien's Nativity
• Rachmaninov (Second symphony in E, truly an awesome piece)
• Remy Zero
• Sleepless in Seattle soundtrack
• Tears for Feares
• The Click Five (saw these guys live in Austin, they can rock it!)
• When Harry Met Sally soundtrack

This blog entry is a continuation of my previous blog entry, Observing the Solaris Kernel.

For my second demo I'll take a look at the anatomy of a simple process. As before, let's start out with a really simple program:

% cat simple2.c
int
main(void)
{
        sleep(99999);
        return (0);
}
% gcc -o simple2 simple2.c
% ./simple2
^Z[1] + Stopped (SIGTSTP)        ./simple2
%

In another window, as root, I fire up mdb in kernel target mode, and look up the proc structure of the process:

root@rutamaya > mdb -k
Loading modules: [ unix krtld genunix specfs dtrace pcplusmp ufs ip sctp usba fctl nca lofs random nfs logindmux ptm cpc fcip sppp ]
mdb> ::ps !head -1
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
mdb> ::ps !grep simple2
R 127247 127218 127247 127193 125946 0x42004000 fffffead763de3c0 simple2
mdb> fffffead763de3c0::print proc_t
{
    p_exec = 0xfffffe819091a840
    p_as = 0xffffffff899c1010
    p_lockp = 0xffffffff84a6db40
    p_crlock = {
        _opaque = [ 0 ]
    }
    p_cred = 0xffffffff923e14e8
...

Let's take a look at the context of the process in the kernel, to see what it looks like after I sent it a SIGSTOP using ctrl+z from the terminal:

mdb> fffffead763de3c0::walk thread
fffffe814473abc0
mdb> fffffe814473abc0::findstack -v
stack pointer for thread fffffe814473abc0: fffffe80007abb50
[ fffffe80007abb50 _resume_from_idle+0xde() ]
  fffffe80007abb90 swtch+0x241()
  fffffe80007abc00 stop+0xa68(5, 18)
  fffffe80007abc40 isjobstop+0xd7(18)
  fffffe80007abcf0 issig_forreal+0x48c()
  fffffe80007abd20 issig+0x28(0)
  fffffe80007abda0 cv_timedwait_sig+0x266(fffffe814473ad96, fffffe814473ad98,
  73de75e)
  fffffe80007abe20 cv_waituntil_sig+0xab(fffffe814473ad96, fffffe814473ad98,
  fffffe80007abe70, 1)
  fffffe80007abeb0 nanosleep+0x141(8047cb0, 8047cb8)
  fffffe80007abf00 sys_syscall32+0x1ff()

::walk thread takes a process structure pointer and "walks" all of the threads in the process (in Solaris there is a 1:1 mapping of user thread to kernel thread these days). Since our simple program is single-threaded this yields a single thread pointer, which we could have also gotten at by hand by looking at the p_tlist member of the proc structure:

mdb> fffffead763de3c0::print proc_t p_tlist
p_tlist = 0xfffffe814473abc0

::findstack takes a thread pointer and walks the thread stack in the kernel. We can see that cv_waituntil_sig() was called from nanosleep(), and it was interrupted by the SIGSTOP signal causing the process to call stop().

Using the pipe facility in mdb and a new dcmd I haven't yet used, ::pid2proc, I can also shorten everything I've done so far into an equivalent two-liner:

mdb> ::ps !grep simple2
R 127247 127218 127247 127193 125946 0x42004000 fffffead763de3c0 simple2
mdb> 0t127247::pid2proc | ::walk thread | ::findstack -v
stack pointer for thread fffffe814473abc0: fffffe80007abb50
[ fffffe80007abb50 _resume_from_idle+0xde() ]
  fffffe80007abb90 swtch+0x241()
  fffffe80007abc00 stop+0xa68(5, 18)
  fffffe80007abc40 isjobstop+0xd7(18)
  fffffe80007abcf0 issig_forreal+0x48c()
  fffffe80007abd20 issig+0x28(0)
  fffffe80007abda0 cv_timedwait_sig+0x266(fffffe814473ad96, fffffe814473ad98,
  73de75e)
  fffffe80007abe20 cv_waituntil_sig+0xab(fffffe814473ad96, fffffe814473ad98,
  fffffe80007abe70, 1)
  fffffe80007abeb0 nanosleep+0x141(8047cb0, 8047cb8)
  fffffe80007abf00 sys_syscall32+0x1ff()

Note the 0t prefix is needed because the PID is displayed by ::ps in decimal, just as if I had run the ps command in a terminal. In fact I could have done this from inside mdb using the shell pipe facility I've already used in this demo:

mdb> !ps -ef | grep simple2
   elowe 127247 127218   0 10:04:24 pts/1       0:00 ./simple2

In my first demo, I examined the user and group ID within a file; as we saw from running ::print to look at the proc structure, p_cred also contains a pointer to a cred structure, which holds the identity of a unique user on the system. Since I started the process from another terminal window under my normal UNIX account, the cred structure should contain the credentials associated with my UNIX login.

mdb> fffffead7ceb43c0::print proc_t p_cred|::print cred_t
{
    cr_ref = 0x32
    cr_uid = 0x1ebfa
    cr_gid = 0xa
...
mdb> fffffead7ceb43c0::print proc_t p_cred|::print cred_t cr_uid|::map =D
                125946

That's me. Since ::print likes hex more than I do, I used the ::map dcmd to pipe the dot from ::print cred_t cr_uid into =D, which is the mdb syntax to display a number in integer decimal format.

There are plenty of interesting things to explore on your own just by starting from the process structure, so this is a great start for folks who are just starting out learning about UNIX internals.

Technorati Tags: [OpenSolaris Solaris]

This past weekend I had the pleasure of presenting at the SIGCSE conference in Houston. For the unacquainted the audience is computer science educators, so my focus was on using OpenSolaris as a vehicle for teaching operating system internals.

For the conference I prepared some slides and a demo that attempt to peel off the first layer of the observability onion into the Solaris kernel. Aside from DTrace (which by now the whole world has heard about) I also make extensive use of the excellent mdb kernel debugging facilities. The demos in particular are worth sharing so I am posting them here for all to see and use as you see fit.

First, the slides (also in StarOffice; if you steal them for your own use all I ask is that you provide a pointer back to this blog entry). The main theme of my presentation is that OpenSolaris brings to the table free access to the source code, while the powerful observability tools allow quick insight into the dynamics of the kernel without mastering the source.

The first demo starts from a simple C program, follows the open() syscall flow down to the process file table with DTrace, and then drills all the way down (using the kernel debugger) into the inode at the file system level. This is accomplished by leveraging access to the source code, and for the demo I deliberately chose a part of the system to look at I had never been in before. This demo took about twenty minutes to create and about twenty minutes to walk through.

The second demo shows how mdb can be used to examine the anatomy of a stopped process. I ran out of time before getting to the second demo; I'll post the second demo in another blog entry in a few days, so keep an eye out for it.

First Demo

/*
 * A simple program which creates a file, writes one byte to it,
 * closes it, and exits.
 */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>

int
main(void)
{
        int fd;
        const char buf[] = "\0";

        if ((fd = open("simple1.out", O_CREAT|O_WRONLY, 0600)) < 0) {
                perror("Unable to create file");
                exit (1);
        }

        if (write(fd, buf, 1) < 0) {
                perror("Unable to write file");
                exit (2);
        }

        return (0);
}

% truss -t open ./simple1
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
open("/lib/libc.so.1", O_RDONLY)                = 3
open("simple1.out", O_WRONLY|O_CREAT, 0600)     = 3

#!/usr/sbin/dtrace -Fs
open:entry
/ execname == "simple1" /
{
        self->tracing++;
}
:return
/ self->tracing > 0 /
{
        trace(0);
}
open:return
/ self->tracing > 0 /
{
        self->tracing--;
}
:entry
/ self->tracing > 0 /
{
        trace(0);
}

% CPU FUNCTION
  0  => open                                          0
  0    -> open32                                      0
  0      -> copen                                     0
  0        -> falloc                                  0
  0          -> ufalloc                               0
  0            -> ufalloc_file                        0
  0              -> fd_find                           0
...

#!/usr/sbin/dtrace -wFs
open:return
/ execname == "simple1" /
{ breakpoint(); }

[console] root@rutamaya > ./simple1_files.d
dtrace: script './simple1_files.d' matched 4 probes
dtrace: allowing destructive actions
dtrace: breakpoint action at probe syscall::open:return (ecb fffffe8501e671e0)
kmdb: target stopped at:
kaif_enter+8:   popfq
[0]> :c
dtrace: breakpoint action at probe syscall::open:return (ecb fffffe8501e671e0)
kmdb: target stopped at:
kaif_enter+8:   popfq
[0]> :c
dtrace: breakpoint action at probe syscall::open:return (ecb fffffe8501e671e0)
kmdb: target stopped at:
kaif_enter+8:   popfq
[0]> <gsbase::print cpu_t cpu_thread->t_procp->p_user.u_finfo.fi_list
cpu_thread->t_procp->p_user.u_finfo.fi_list = 0xfffffea7aa34b800
[0]> 0xfffffea7aa34b800,4::print uf_entry_t uf_file->f_vnode
uf_file->f_vnode = 0xfffffe813aa86c80
uf_file->f_vnode = 0xfffffe813aa86c80
uf_file->f_vnode = 0xfffffe813aa86c80
uf_file->f_vnode = 0xffffffffb282ce00

%gsbase holds a pointer to the current CPU structure; from there I can get the running thread, access the process pointer, follow the u structure to the u_finfo which contains the array of open files (again indexed by file descriptor, as we discovered from the source). Once I have the pointer of this array I can follow the struct file to get the vnode for each open file.

Going back to the source and looking at the struct vnode I can see that the v_type and v_path fields are interesting so let's take a look at them..

[0]> 0xfffffea7aa34b800,6::print uf_entry_t uf_file|::grep >0|::print "struct file" f_vnode->v_type f_vnode->v_path
f_vnode->v_type = 4 (VCHR)
f_vnode->v_path = 0xfffffea79ece81b8 "/devices/pseudo/pts@0:1"
f_vnode->v_type = 4 (VCHR)
f_vnode->v_path = 0xfffffea79ece81b8 "/devices/pseudo/pts@0:1"
f_vnode->v_type = 4 (VCHR)
f_vnode->v_path = 0xfffffea79ece81b8 "/devices/pseudo/pts@0:1"
f_vnode->v_type = 1 (VREG)
f_vnode->v_path = 0xfffffe80ea212e40 "/home/elowe/proj/SIGCSE/simple1.out"

This prints each element of the array of uf_entry_t structures in the u structure if the pointer is not NULL, and pipes the dot (which is a struct file to the ::print dcmd to examine the fields we're interested in. The first three file descriptors are stdin/stdout/stderr and all point back to my virtual terminal; the fourth is the file being written to by the simple1 program.

Since I'm using NFS (and I can see from my earlier DTrace output that nfs3_open() is in the call path) I decided it might be interesting to poke at the underlying inode. The source shows that nfs3_open() gets an rnode_t by casting the vnode's v_data directly into the file system's internal inode representation. A little bit ago we had the vnode pointer in our mitts:

[0]> 0xfffffea7aa34b800,4::print uf_entry_t uf_file->f_vnode
...
uf_file->f_vnode = 0xffffffffb282ce00

So now we can keep digging:

[0]> 0xffffffffb282ce00::print vnode_t
{
..
    v_data = 0xfffffe83625e9d10
    v_vfsp = 0xffffffff885eaf00
..
    v_op = 0xffffffff86d24c40
..
[0]> 0xffffffff86d24c40::print struct vnodeops
{
    vnop_name = 0xffffffffc01c5a20 "nfs3"
    vop_open = nfs`nfs3_open
..
[0]> 0xfffffe83625e9d10::print rnode_t
{
..
    r_size = 0x1
    r_attr = {
        va_mask = 0xbfff
        va_type = 1 (VREG)
        va_mode = 0x180
        va_uid = 0x1ebfa
        va_gid = 0xa
..
[0]> 180=O
                0600
[0]> 0x1ebfa=D
                125946
[0]> a=D
                10
[0]> :c

% ls -l simple1.out
-rw-------   1 elowe    staff          1 Feb 27 11:34 simple1.out

Technorati Tags: [OpenSolaris Solaris]