Here at Sun we have a large QA organization that works hard to ensure that our implementations of Java technologies are of the highest possible quality. My team's focus, however, is very different.

Conformance-testing is a special kind of testing. Many of the areas addressed by "normal" testing (performance, usability, stability, fitness for purpose) are out of bounds for us - unless, of course, they are explicitly addressed in the spec.

Similarly, most specs deliberately - or perhaps inadvertently - leave some areas unsepecified. These too are off-limits for us. (In this sense our testing is more "black box" than "white box".)

Although we care passionately about quality, it is not our job to ensure that implementations of Java technologies are of high quality. We test only what is specified. If the spec requires a rotten implementation (hopefully somebody would have caught this at an earlier stage!) then we will faithfully verify that implementations are rotten in exactly the way the spec requires.

If it's specified then we test it. If it's not, then this is someone else's job.