What is the best way to return status from a test case? This question has been asked more than once during the last week so it deserves a blog post.

Different test frameworks handle test results differently. Alexey has simple a side-to-side comparison of tests formats for JUnit-like tests and JT Harness in his blog. JUnit uses exceptions to signal test failure, JT Harness returns a Status object:

JUnit

    public void test() throws AssertionFailedException {
        // do something 
        ...
        // check result 
        assertEquals(expectedResult, result);
    }

JT Harness

    public Status test() {
        // do something 
        ...
        // check result 
        return (expectedResult == result) 
                ? Status.passed("OK") 
                : Status.failed("Unexpected result " + result);
    }

The exception-throwing approach appears to be the most straightforward, but it has its drawbacks. While for the simplest unit tests it may make the code easier to read, it quickly gets difficult in more complex cases. And exceptions affect test performance as well.

Here are a few other reasons to use the Status object.

First, use of exception to signal test status violates one of the important API design principles: "never use exceptions for flow control". Exception use is normally restricted to recovery from abnormal conditions such as resource failures or programming errors. A common argument is that test failure is caused by programming error and thus use of an exception is justified, but this is indeed not the case. The one and only reason to write unit tests is to detect bugs. While there is a bug in the underlying code, it should be considered normal flow of control for the test. On the other hand, if the test itself encounters a problem or the underlying code break down unexpectedly, then it is appropriate to throw an exception.

Second, use of an exception artificially limits test outcomes to just "Pass" or "Fail". Use of a return type allows to easily extend test format with additional status types. We found it convenient to have at least yet another status, "Error". "Error" is different from "Fail" as it signals a problem with test initialization or execution; "Fail" means that the test executed successfully and found a bug in the code. Throwing an exception in case of a test initialization problem is justified. In fact, JT Harness defines an exception class (MultiTest.SetupException) just for this purpose.

To summarize, use of a Status object is a cleaner API design. But if you like the brevity of the exception mechanism, it is fine to use as well. Just don't use both at the same time .