Jonathan Gibbons
Minor updates for jtreg
jtreg 4.0 b03 is now available, and fixes a number of minor issues. You can find it on the standard jtreg download page
It includes the following minor fixes and updates:
- Do not follow symlinks when cleaning a directory.
- Avoid potential NPE when cleaning a directory.
- Allow annotation processing of class files in jtreg directives.
New option /process for @compile tag disables requirement that one or more *.java files must be given to @compile. - Set test.src and test.classes properties for @compile tasks.
These properties can now be used by annotation processors invoked by the @compile task. - Recategorize -e option as a general option instead of a JDK one.
- Set MKS to use case-sensitive compare in launcher script.
- Remove obsolete reference to JAVA_HOME from launcher script.
- (jtdiff) Avoid NPE in standard jtdiff if no output file given.
Posted at 07:38PM Jul 01, 2009 by jonathangibbons in jtreg | Comments[0]
jtreg update
I've been working on some bug fixes and features for the OpenJDK Regression Test Harness, jtreg. This new version has just been released in the standard place on the OpenJDK website. Here is a quick summary of the changes.
-
New option -noreport.
An issue with the underlying JT Harness is that it always generates reports about all the tests in the test suite, instead of just the tests you selected to run. This means that there may be a noticeable pause when the tests have completed while jtreg writes out reports. For those folk that never read the reports anyway, you can now disable the reports with this new option.
-
Fix -exclude option in -samevm mode.
This combination of options did not work correctly. Now, it should.
-
Improve version detection in jtreg.sh script.
The wrapper shell script tries to verify it is running on JDK 1.5 or later. The version detection used to be confused if you were running JDK in headless mode, when a spurious message could appear on System.out. Such messages are now filtered out.
-
Allow System.setIO calls in -samevm mode.
jtreg tries to insulate tests from each other, so that side effects in one test cannot affect the functioning of another. Previously, tests were not allowed to call System.setIO when running in samevm mode. This prevented a number of tests from running in samevm mode. However, since jtreg only supports running one test at a time, there is no reason to prohibit this call.
-
Fix the initialization of jtreg-specific system properties for samevm tests to use a
space separated list, not the default List.toString() format ("[a,b,c]").
It was an oversight that a suitable toString() method was not provided on an internal class, resulting in the default toString() for a List being used, instead of the intended format of a space-separated list.
-
Set the classpath for -samevm tests to include standard jtreg directories, the same as for
-othervm tests.
This is to reduce any unnecessary differences between othervm mode and samevm mode.
-
Enable XML report via system property.
While some folk don't want reports at all, other folk want more reports. The system property
javatest.report.kindscan be used to select which report types are generated by the underlying JT Harness report facilities. The property can be set to a space separated list of any of the values "text", "html", "xml". The default is "text html". The XML format will be useful for a possible upcoming new plugin for Hudson. -
Reduce false positives when setting keywords from test description.
Some users reported problems with -ignore:quiet and -k:!ignore incorrectly ignoring tests with -XDignore.symbol.file=true. This fixes that issue.
Recategorize the -e option in the "Main" group instead of the "JDK" group.
Update FAQ with reference to -e as a way to override the PATH setting.
(jtdiff) Add convenience 'jtdiff' script.
-
(jtdiff) Refactor and add new "super-diff" mode.
"Super-diff" mode is a way of analyzing a matrix of test runs, organized by platform and date. It runs a standard jtdiff for each row in the matrix, thus comparing test results across platforms for each date, and for each column in the matrix, thus comparing the history of the test results for each platform. See
jtdiff -help -superfor more details.
Posted at 02:38PM Oct 21, 2008 by jonathangibbons in jtreg | Comments[0]
jtharness vs jtreg
Now that the source code for the OpenJDK Regression Test Harness (jtreg) is available, this provides an overview of how jtreg relates to JT Harness.
In the beginning, there was a need to run compatibility tests on all Java plaforms, and thus was born the "Java Compatibility Kit" (JCK), and the test harness to run those tests (JavaTest). Not long after, there was a need to run a different type of test, regression tests, for JDK. They are fundamentally a different style of test (think white box and black box), but rather than write another, different harness, we adapted JavaTest, and thus was born "jtreg". Eventually, we needed to be able to run tests on different types of platforms, such as Personal Java, a precursor to the J2ME platform. JavaTest evolved to fill the need, and gave rise to the basic architecture you see today.
As you can see, JavaTest, now open sourced under the name JT Harness, is composed of a number of distinct components
- The core harness. This provides the general ability to represent tests, test results, and to sequence through the execution of a series of tests.
- Utilities. These provide support to the other JT Harness components.
- The user interface. JT Harness provides a rich graphical user interface, as well as a command line interface.
And, deliberately, there's a hole. By itself, JT Harness cannot directly run tests in any test suite. It needs to be given a plugin for each test suite, that define how to locate and read tests, how to configure and execute tests, and other additional information, often used by the GUI, to provide documentation specific to the test suite. The plugin code is not complicated and is often a direct use or simple extension of the utility classes provided by JT Harness. Note: most test suites do not need to use all of the JT Harness utility classes, which explains why JT Harness may have more build-time dependencies than are required to build and run any individual test suite.
Thus, the complete picture is more as follows:
This is the form used for most test suites, such as JCK and other TCKs. The code to plugin to the harness is bundled in the test suite, along with a copy of JavaTest or JT Harness itself. These test suites all leverage the user interfaces provided by the harness, both the CLI and GUI. When the user runs JT Harness, and tells it to open a test suite, the harness looks inside the test suite for details of the plugin to be used in conjunction with that test suite.
Because jtreg evolved from JavaTest early on, and specifically, before the JavaTest user interfaces evolved to what you see today, jtreg has a slightly different architecture.
jtreg still utilizes the harness' plugin architecture to configure how to read and execute the tests in the test suite, but it provides its own custom command line interface for setting up a test run. The jtreg CLI provides options tailored to running the JDK regression test suite. It can also invoke the JT Harness GUI, but this is just for monitoring a test run, and does not (yet) provide much support for reconfiguring a test run with different options from within the GUI, as you can do for other test suites.
jtreg is also different from other test suites in that jtreg is distributed separately from the tests that it gets to execute. TCKs are generally built and distributed as self-contained products in their own right, whereas the JDK regression tests do not get prebuilt: they exist and are executed in their source form in each JDK developer's repository. Per the The JDK Test Framework: Tag Language Specification, the tests are compiled and executed as needed.
For more information, see ...
Posted at 02:54PM May 05, 2008 by jonathangibbons in jtreg | Comments[0]
OpenJDK Regression Test Harness (jtreg) - open source
The OpenJDK Regression Test Harness, also known as "jtreg", is now available with an open source license.[Read More]
Posted at 08:54PM May 01, 2008 by jonathangibbons in jtreg | Comments[1]
Unit Testing Wisdom From An Ancient Software Start-up
JavaOne contained many gems of wisdom, of varying sizes. This one indirectly came my way during the week, and is fun enough to pass along. It is self-described as "Good advice on developer and unit testing, packaged as twelve cryptic bits of ancient Eastern wisdom." You can find it here. It is also available in PDF. Enjoy -- and thanks to Alberto Savoia for making it available.
Posted at 07:12PM May 14, 2007 by jonathangibbons in jtreg | Comments[0]
com.sun.javatest.*: JT Harness API
JavaTest provides some useful APIs for specialized usage ...[Read More]
Posted at 08:00AM Nov 16, 2006 by jonathangibbons in jtreg | Comments[0]
javatest.png: JavaTest in pictures
A look at the evolution of the JavaTest harness, over the years.[Read More]
Posted at 08:00AM Nov 15, 2006 by jonathangibbons in jtreg | Comments[2]
Way to go, JT Harness
A brief background overview of the harness underlying jtreg: the JavaTest harness, which is now available as JT Harness. [Read More]
Posted at 08:07AM Nov 14, 2006 by jonathangibbons in jtreg | Comments[0]
wwwww.jtreg: The Who What Where When Why of jtreg
A brief background overview of jtreg, the JDK regression test harness. [Read More]
Posted at 12:00AM Nov 13, 2006 by jonathangibbons in jtreg | Comments[0]