By now I'm sure it's painfully apparent to anyone who has read my blog I am a hard-core geek. In fact, I'm not just your run-of-the-mill hard-core geek, but a kernel geek.

Aside from our being born socially inept, a common affliction amongst hard-core geeks is an inability to communicate our thoughts and ideas (and emotions -- but I ain't goin' there!) effectively. While there are a few rare folks I've met out there who seem to have stumbled upon the magic cure to this nasty affliction, the hard-core geek who can communicate effectively is a rare breed.

As a professional software engineer, one of my fundamental beliefs is that the dominant factor between a successful project and an unsuccessful project reduces to the effective dissemination of key information. As a battle-worn veteran of several software projects, I've seen other projects flounder because the members of the team didn't understand or buy-into the mission, major deliverables or objectives weren't clearly defined, team members didn't agree on the requirements, or team members grew differing perceptions of what the final product would look like!

The impacts of ineffective written communications within a project often are magnified by conspiring factors such as geographically dispersed teams, rushed schedules (i.e. we don't have the TIME to define WHAT we're going to deliver!), and in extreme cases, recalcitrant team members!

In this post I examine the role of effective project documentation in executing successful development projects, and share why I believe this is even more important in an Open software development model than in traditional, closed-source projects.

The Traditional Software Project Model

The traditional project development model is usually the classic waterfall model of software engineering. The project is conceived, funded, and staffed. At that point the conception phase moves into requirements analysis, functional specifications are developed, and the architecture is determined. Next, the design is specified, an implementation created, and the implementation is checked against the specifications of the architecture. If everything checks out, the project is accepted by the sponsors and deployed.

At each stage of the traditional software project, new stakeholders are introduced, and these stakeholders have progressively lower levels of focus. If the communication of the concept is not clear, the architects may come up with a proposal which doesn't match the original concept. If the architecture is over- or under-constrained, the implementation may be untenable or incomplete. If the validation specification does not match the architecture the acceptance may not match the implemented product. Finally, if the end-user documentation does not match the architecture and implementation, the end-user will find using the product to be time consuming, frustrating, and costly. Anyone who has played the telephone game and observed how garbled even a simple message can become can appreciate how even a small defect in the communication chain of a complex project can lead to catastrophy in the final product!

The Open Software Development Project Model

The Open software development model differs from the traditional model in several imporant ways.

First, the concept and requirements typically come directly from the end-user(s), and are usually accompanied by a rough implementation in the form of a prototype. At this point, the community critiques the proposal and implementation. Community members attempt to reverse-engineer any unspecified requirements, and suggest new concepts for inclusion. In the case of large software projects, the development may shift back into the traditional model after this point. With smaller projects, the process may be more iterative, with the traditional validation phase leading to revised requirements, and the cycle repeating itself.

Second, in the Open model the stakeholders at each stage are different. In the case of the traditional model, the needs of the business (the business case) ultimately drive the scope and requirements of a project; the concept phase is nearly set in stone by the time the implementors start to engage. In the Open development model, the focus is usually more on the desired outcome of the end-users of the product. The requirements, and even the high level concepts are more free to evolve over time as the project progresses.

The Role of Communication in Software Development

In my experience, most traditional software projects make it through the first few stages (concept and requirements) with relatively few problems. By the time the implementors are brought on-board, the functional specifications are baked, and they map back to the original concepts and requirements fairly accurately. The big issues such as projected development and scope have been settled by the project bosses. Due to the senior-level of involvement up to this point, the accuracy and quality of specification documents is usually very good.

The really successful software projects become successful because they give the right level of attention to clearly communicating the key concepts and requirements. Software engineering experts have differing opinions on how much of the total effort should be given to a project's early phases, but my gut feel is that it is well over 25%, and may be as much as 50%.

To make an Open project which is proposed in the form of a prototype successful, it is even more important to keep in mind that the key concepts and requirements need to be thoroughly documented for the community to agree on them. There are several reasons for this:

• Engineers on traditional development projects are usually geographically co-located, and utilize high bandwidth forms of communication such as meetings and whiteboard sessions to reach decisions and agreements. Open development projects are almost always geographically distributed. E-mail and project web pages, which are lower-bandwidth forms of communication, are typically the dominant forms of reaching decisions and agreements; this makes increased accuracy and completeness a necessity.
• Unlike in the traditional model where the stakeholders are the product bosses, Open projects have the entire community (and the end users) as the stakeholders! If the community can't agree on the high level objectives and functional requirements of a software project, the project is doomed from the start.

Documenting Concepts, Requirements, and Architecture

The concepts and requirements comprise the big picture view of the software project. These define aspects such as the major capabilies of the product and the intended target market. The architecture defines the input/output methods, interoperability features, and user interface.

The first nail in the road at the early phases of a project is that seemingly unimportant details such as how the requirements were gathered are often not recorded. High-level tradeoffs made in mapping the requirements into architecture are often not well documented (if at all). As we'll see, this is a disaster waiting to happen down the road when the implementors hit a roadblock in the design.

A less obvious trap at this point is that the requirements and architecture are usually baked by experienced software engineers with a good, albeit incomplete, idea of the requirements of the implementors. On the other hand, implementors are usually less-experienced, and have a more localized focus -- meaning that there is an inherent opportunity for information loss when translating the requirements into software constructs. If an implementor doesn't understand how flexible a requirement is, or why the requirement is there at all, the implementation may be inadequate or over-constrained.

The latter effect is magnified in the Open development model, where the implementors do not work with the software as their day job; if they do, they often don't have the benefit of years of experience networking with engineers inside a major software company. If an Open development project is going to be taken on by less-experienced implementors, then clearly communicating the concepts, requirements, and architecture of the software is of paramount importance.

Documenting the Design

To help avoid pitfalls during the implementation phase, I believe it is essential that the design team thoroughly document the aspects of their implementation blueprints. The apsects which should be thoroughly documented include any and all of:

• Assumptions
• Constraints
• Design Trade-offs
• Design Decisions

Assumptions

We have all heard the cliche' about how assumptions are bad. However, reality dictates that no matter how well the requirements are written and the architecture is specified, there will always be room for interpretation. The idea behind documenting all assumptions regarding the concepts, requirements, and architecture is that the folks who created them can review these assumptions and provide clarifications when necessary. In some circumstances there may be holes in the original specifications which result in assumptions about major requirements! If in doubt, and it's open to interpretation -- document your assumptions about it! The time spent up front will save you a lot of frustration down the road by preventing an implementation going to the validation phase, only for the end-user to come back and say this doesn't work the way I expected it to!

There's also an advantage to catching holes in our assumptions early which is particuarly helpful in the context of an Open development community. We're all human, and as such we have tendencies to blame others rather than ourselves when things go wrong. Having members of a community come back after an implementor has worked long and hard on a problem only to tell him/her that the hard work that person has done is incorrect is a big blow to the implementor's pride. In the worst case, flame wars may insue, and the contributor ends up leaving the community. A lot of frustration and wasted effort can be eliminated if the community has the opportunity to identify misguided assumptions and correct them early on.

Constraints

One major difference I've noticed between experienced, high-caliber software engineers and average software engineers is that experienced software engineers know how to leverage constraints in order to optimize solutions to difficult problems.

Let me give a concrete example from a project I worked on a few years ago: a previous team worked hard at solving a difficult kernel problem in the area of virtual memory management. The team came up with an approach, worked for about a year, and had a working solution -- only to have their approach nailed to the wall because of a design flaw which could result in data corruption.

when I took over the project it didn't take me long to decide that the systemic problem with their approach revolved around the constraints. Specifically, the project team's approach failed because they had designed around a constraint that a specific DDI call had to remain supported. Because the DDI call in question was part of an obsolete framework, removing support for the call was the best way to solve the problem, and it removed the potential hole where an obsolete or incompatible device driver might cause kernel data corruption when the new feature was enabled.

In a way constraints are assumptions -- but they are assumptions that aren't related to the specifications. Which functions a library supports, which platforms will best run an application, and which programming languages to use may all be constraints of an implementation on which the requirements and architecture are entirely mute.

Another example of leveraging constraints to your advantage in an implementation is creating a dependency on a particular system library, rather than developing your own equivalent functionality.

Design Trade-offs

Some folks look negatively on design-tradeoffs -- they refer to the practice of making trade offs in a design as cutting corners. However, software is like any other engineering practice -- we need to balance effort and complexity against functionality. Finding a happy middle ground is a constant battle.

Most high quality projects I've seen do a good job of articulating their design tradeoffs. This is because there are two paths to detecting an incorrect design tradeoff: in the first case, the tradeoff is documented, and someone detects the problem before it poses a major problem; in the second case, the product blows up in the hands of the end-user! I don't know about you, but I prefer constructive criticism over catastrophic failure any day!

Design Decisions

I prefer to think of the design decisions as the politics of software design. This position may not be popular, but I think it gets the point across well. If others don't understand the reasoning behind your decisions, they will be more inclined to disagree with the outcome. If you do not have a strong case, and others hold enough clout over you, they may convince others that their approach is right, and yours is wrong.

The best defense against showdowns based on emotion and past experience (versus the facts!) is to clearly articulate the reasons you decided something is best implemented a certain way. In many cases, the decision is unimporant and ends up being arbitrary (for instance, how often to poll a descriptor). In other cases, the decision is the result of a thorough analysis. In yet other cases, there may be mistakes that you made others can find that show your decision is flawed. Regardless of the scenario, clear and concise documentation of the reasoning behind a design, and not just the documentation of the final design, will lead to a better end product.

Putting Project Communication into Practice

Start out with a communication plan

Document each project phase

Refer to your documentation often

Examples

The minimalist approach

Suppose that I am proposing a bugfix. As long as there is no impact of the design of the system, and no change in the user's experience, only a small design document is necessary. My design may be as little as a few paragraphs to accompany the diffs in an e-mail, explaining the root cause of the bug, how I arrived at the fix, and what the fix does differently to correct the abnormal behavior.

Use common sense! If a bugfix is a one-line change to correct a misspelling in a code comment, it's obvious to everyone that no design documentation is required -- the diffs speak for themselves. In anything more complex than the most trivial, mechanical change, there was some thought process involved. The idea is to get that thought process out into the open, where others can review not just the fix, but what you were thinking when you wrote it.

The micro-project

Suppose I'm implementing a new function call in a shared library. The end-users will be software developers. The stakeholders will be whoever requested the interface. I'd start by collecting the requirements, which probably are coming from one or more persons who proposed the interface. Once the requirements fleshed out and agreed upon, I'd draw up an interface specification. Since all library calls require a man page, I would do this in the form of a mock-up man page, and send it out for wider review, making sure there are several experienced software developers among the reviewers. Once the interface was baked, I would sit down and draft up an implementation proposal, listing all of the tradeoffs I could think of. Once I had that in place I would request a peer review of my proposed implementation, and after iterating on it I would continue onto the code.

The macro-project

As usual a one-size-fits-all approach isn't going to work, so you'll need to consider many factors -- size of the team, importance of the functionality, dependencies, schedule, etc. A macro project may range from an RFE (request for enhancement) to a full blown software package.

At a minimum, I would suggest breaking up the project into the following stages, each paired with a set of project documentation deliverables:

• Concept:
  • List of key requirements and features
  • Component block diagram
  • Interface specifications
• Design:
  • Dependency and data flow diagrams
  • Detailed specifications for each component
  • Constraints, risks, assumptions, and trade-offs
• Implementation:
  • Code comments
  • Big theory statements (to describe algorithms)
• Validation:
  • Test plan
  • End-user documentation

If you are putting together a good-sized project and you lack experience, I would suggest seeking outside help. The OpenSolaris community (general discussion list) is a good place to start.