This will be a week of contrasts for me. I'm currently in Cannes-Mandelieu, France where I'm attending the annual Technical Plenary meeting of the World Wide Web Consortium (W3C). The W3C, of course, is the organization that's responsible for defining the standards and protocols on which the World Wide Web is built. Like most standards organizations it operates through a relatively small permanent staff and a much larger number of volunteers, most of whom (like myself) are sponsored by their employers to participate in standards development activities.

The organization functions primarily through Working Groups, each of which is responsible for defining a particular standard. (The W3C's Working Groups are the equivalent of the JCP's Expert Groups.) (If you're interested, you can compare the two organizations' process documents here and here.)

My involvement in the W3C goes back several years to the release of JAXP 1.2, which revised the spec to require that XML parsers incorporated into the Java SE platform support XML schema. Oops! Suddenly my team became responsible for verifying that XML parsers conform to specifications for a language that is arguably more complex than the Java language itself. Since we'd spent more than 50 staff-years developing tests for the Java language it seemed extremely unlikely that we'd be able to develop a comprehensive conformance test suite for XML Schema.

What I wanted to do was to simply revise our conformance requirements to say "plug in any parser you want so long as it passes version x.y of the W3C XML Schema conformance test suite." I looked for such a test suite, and didn't find one. The XML Schema Working Group had developed a large body of tests, but these were not packaged into a well-documented test suite with the kind of rigorous conformance requirements that we associate with our own TCKs. Consequently it would have been very difficult for our licensees to run the W3C tests and to definitively determine whether or not their parsers conformed to the specs.

As we have learned over the years, there's much more to conformance testing than developing a collection of tests. These must be packaged together with a test harness, documentation, and conformance requirements into a version-controlled test suite. There must be a well-defined process for users of the test suite to challenge the validity of a test and for invalid tests to be excluded or updated.

In the absence of such a test suite our only option seemed to be to incorporate the existing W3C Schema tests into our own TCK. (In a presentation I gave at last year's Technical Plenary on the re-use of test suites I called this testing by value rather than by reference.)

We did eventually incorporate several thousand W3C tests into the JAXP TCK and more recently we donated back to the W3C several hundred additional tests that we had developed for the JAXP TCK. With the intent of promoting more of this kind of collaboration between Sun's TCK developers and the W3C, and in the hope that we could influence the W3C's conformance test developers in a direction that would make it easier for us to reuse their work in the future, I joined what was then the QA Working Group (we have since re-chartered as an Interest Group).

This group has focused on developing guidelines for and promoting best practices in the development of specifications and conformance tests within the W3C. A complete list of the documents we have published can be found here here. The Test Development FAQ, in particular, addresses the difference between a "collection of tests" and a "test suite."

The Interest Group met yesterday and today to discuss how we should focus our efforts during the coming months. As always we have much more work to do than we have people to do it, so we would very much welcome assistance from people in the conformance testing community. Whether or not you're able to help, please join our mailing list and read our blog.

Tomorrow (Wednesday) is the actual Technical Plenary day, during which we all get together for a series of presentations and then on Thursday I'm leaving for a two week visit to St. Petersburg, Russia, where most of my team is based. More on that later...

Here at Sun we have a large QA organization that works hard to ensure that our implementations of Java technologies are of the highest possible quality. My team's focus, however, is very different.

Conformance-testing is a special kind of testing. Many of the areas addressed by "normal" testing (performance, usability, stability, fitness for purpose) are out of bounds for us - unless, of course, they are explicitly addressed in the spec.

Similarly, most specs deliberately - or perhaps inadvertently - leave some areas unsepecified. These too are off-limits for us. (In this sense our testing is more "black box" than "white box".)

Although we care passionately about quality, it is not our job to ensure that implementations of Java technologies are of high quality. We test only what is specified. If the spec requires a rotten implementation (hopefully somebody would have caught this at an earlier stage!) then we will faithfully verify that implementations are rotten in exactly the way the spec requires.

If it's specified then we test it. If it's not, then this is someone else's job.