After multiple rounds of code review the netcat (or nc) test suite is now finally in the onnv-stc2gate. The test suite has its home in the OpenSolaris Networking community (see the networking tests page for the list of networking test suites).
The source code is present in the src/suites/net/nc/ directory and SUNWstc-netcat packages can be downloaded from OpenSolaris Download center.

Before I go further, this is how it looks like when the test suite is run (the output is trimmed a bit):

vk:honeymooners:/opt/SUNWstc-nc$ run_test nc
Validating Arguments...
New TET_ROOT for this run : /var/tmp/honeymooners_27828
The results will be available in /var/tmp/results.27828
tcc: journal file is /var/tmp/results.27828/testlog
12:45:57  Execute /tests/dflag/tc_dflag
12:46:04  Execute /tests/hflag/tc_hflag
12:46:05  Execute /tests/kflag/tc_kflag
12:46:11  Execute /tests/nflag/tc_nflag
12:46:15  Execute /tests/portranges/tc_portranges
12:46:23  Execute /tests/pflag/tc_pflag
12:46:26  Execute /tests/sflag/tc_sflag
12:46:35  Execute /tests/Uflag/tc_Uflag
12:46:36  Execute /tests/vflag/tc_vflag
12:46:43  Execute /tests/zflag/tc_zflag
12:46:46  Execute /tests/iflag/tc_iflag
12:46:59  Execute /tests/lflag/tc_lflag
12:47:29  Execute /tests/rflag/tc_rflag
12:48:16  Execute /tests/Tflag/tc_Tflag
12:48:33  Execute /tests/uflag/tc_uflag
12:48:50  Execute /tests/wflag/tc_wflag
##################################################
TC /tests/dflag/tc_dflag

TP 1 tc_dflag PASS
##################################################
TC /tests/hflag/tc_hflag

TP 1 tc_hflag PASS

...

##################################################
                 SUMMARY      
                 =======      
 
Number of Tests : 50

PASS            : 50
FAIL            : 0
UNRESOLVED      : 0
UNINITIATED     : 0
OTHER           : 0
 
##################################################

Test Logs are at /var/tmp/results.27828, Journal File = /var/tmp/results.27828/testlog 

vk:honeymooners:/opt/SUNWstc-nc$

It's been almost a year since I started developing the test suite last Christmas (see the initial blog entry about nc-tet). Since then, I have lost part of the source code in hard drive crash, had to redo the source tree structure, fix ksh style, fix numerous bugs in test suite code and make the test suite more robust. One might ask whether having test suite for such a simple program like nc(1) was worth the hassle. I have only one answer to that: absolutely. First, it gives a confidence of not breaking (most of; see below) existing things when changing/adding functionality and second it helped me (and I hope the others participating/observing the code review on testing-discuss too) to explore what it takes to write a test suite from scratch (I will not go here into details whether I prefer CTI-TET over STF and vice versa).

The Beautiful code book (which I really recommend for anyone tinkering with any source code) contains a chapter called Beautiful tests by Alberto Savoia. I hope that at least some of the test purposes in nc test suite have some degree of beautifulness of at least one of the ways highlighted by Alberto (1. simplicity/efficiency, 2. help making the software being tested better in terms of quality and testability, 3. breadth/thoroughness).

One of the important questions for a test suite is code coverage level. Obviously, for software adhering to the OpenSolaris interface taxonomy model it is important that the test suite exercises all of the Committed interfaces and execution paths around those interfaces. For nc(1) this means a subset of the command line options and their arguments (see PSARC 2007/389 for the actual list). The key is certainly to test the features which are likely to break with an intrusive code change.

Very crude view of test coverage for nc(1) test suite (counting test purposes gives only very remote idea about real coverage but at least provides visual image) looks like this:

       rflag: +
       Tflag: +++++---
       pflag: +
       iflag: +-
       vflag: ++
       kflag: +
       Uflag: +-
       dflag: +
       uflag: ++-
       sflag: +-
       hflag: +
       nflag: +-
       wflag: +
  portranges: +---
       lflag: ++++++++----------

One plus character stands for one positive test purpose, minus is negative test purpose.

Side note: the above ASCII graph was produced using test-coverage-graph.sh script (which presumes certain naming scheme for test purpose files). Just pipe a file listing into the script with test purpose filenames compliant to the scheme used in ontest-stc2 gate and it will spew out graph similar to the above.

In the above musing about code coverage I left out an important piece - why some of the features are not tested. For nc(1) the yet untested part is the SOCKS protocol support. Basically, this is because test suite environment does not contain SOCKS server to test against. There might not be many people using the -x/-X options but from my own experience nothing is more frustrating than discovering some old dusty corner which had to be fixed long time ago or removed completely. So for now, on my workstation which sits behind SOCKS proxy I have the following in ~/.ssh/config for a server outside corporate network which hosts my personal mailbox so it is accessed every day:

Host bar
  User foo
  Hostname outside.kewl.org
  # NOTE: for nc(1) testing
  ProxyCommand /usr/bin/nc -x socks-proxy.foothere.bar outside.kewl.org %p
  ForwardAgent no
  ForwardX11 no

This ensures (along with upgrades of the workstation to recent Nevada builds periodically) that SOCKS support gets tested as well. And yes, ssh-socks5-proxy-connect(1) and ssh-http-proxy-connect(1) are not really needed.

Now with the test suite in place, anybody modifying nc(1) (there are some RFEs for nc in the oss-bit-size list and other bugfixes or features are also welcome) can have pretty high confidence that his change will not break things. Yes, this means that more nc(1) features are coming.

As you might know, Netcat implementation is going to be part of OpenSolaris. The initial Netcat integration is based on a reimplementation from OpenBSD (here's why).

As Jeff Bonwick said, open sourced code is nothing compared to the fact that all design discussions and decisions suddenly happen in the public (loosely paraphrased). This is a great wave to ride and I have jumped on it when it was not really small so I have at least posted the webrev pointer for initial Netcat integration (CR 4664622) to the opensolaris-code mailing list (which is roughly the equivalent of freebsd-hackers, openbsd-tech or similar mailing lists) to get some code review comments.

Since then couple of things changed. Thanks to Dan Price and others it's now possible to upload webrevs to cr.opensolaris.org. I have started using the service so the new and official place for the Netcat webrev is

cr.opensolaris.org/~vkotal/netcat-webrev/

The webrev has moved location but what I said in the opensolaris-code post still holds true:

Any constructive notes regarding the code are welcome. (I am mainly looking for functional/logic errors, packaging flaws or parts of code which could mismatch the PSARC case)

The following things could help any potential code reviewer:

• Summary of the changes done to the original implementation in order to adapt it to Solaris environment..
• PSARC 2007/389 case covering interfaces delivered by this project. For more information about ARCs see Architecture Process and Tools community pages.
• SUNWnetcat package (x86) which contains /usr/bin/nc binary
• webrev of the differences between my version of Netcat and the one which is currently in OpenBSD.
  Only the *.[ch] files matter, of course. (This is very easy thing to do with distributed SCM since it only requires one to reparent and regenerate webrev against new parent workspace)
• Updated manual page
  This is slightly different from the man page in the PSARC materials because it contains new section about using nc with privileges and associated set of examples in the EXAMPLES section. The man page in the PSARC materials will not be updated because after a case is approved, the man page is updated only in case some architectural changes were needed. In the case of privileges, it is only addition describing specific usage, no architectural changes.

The conclusion for non code reviewers ? I hope it is clear the in (Open)Solaris land we value quality and transparency. Peer reviews and architectural reviews are just (albeit crucial) pieces which help to achieve that goal.