« Previous day (Sep 25, 2007) | Main | Next day (Sep 27, 2007) »

Download NetBeans!

20070926 Wednesday September 26, 2007

GUI Testing on the NetBeans Platform (Part 1)

Your wonderful Swing application is reaching completion, or is at some intermediate stage where you think it should be kind of solid, and you're wondering: "Hmmm. I have a very complex user interface. How do I know that all those menu items, toolbar buttons, dialogs, palettes, windows, and wizards actually work? Some of them come from modules provided by others, some come from modules provided by people who no longer work here, some come from modules that I wrote in a hurry... I really don't know how reliable all of it is anymore. Should I maybe get an intern to click every piece of u.i. in order to find out whether everything is working as expected?"

And the answer is, of course, "Yes". Just kidding. The answer is, in fact, of course, "No". If you're building your application on the NetBeans Platform, you'll be able to type this on the command line: 

ant -Dxtest.testtype=qa-functional

And then your application will start up and little tests such as the following will be run: 

public void test1() {
    new Action("File|New", null).perform();
}

So, above, after starting up the application, your test infrastructure will automate the choosing of the "New" menu item under the "File" menu. If you're around at the time, it's quite fun to watch, because you'll see it happening in front of your eyes, without you doing anything at all. If you're not around, and you're running the Ant script as part of some batch process, then that's also okay, because the results of the test are written to an HTML file, that looks like this:

Tests such as these are called functional tests, as opposed to unit tests. Typically, with a functional test you're trying to establish whether a scenario works or not. Clicking the New menu item under the File menu isn't really a scenario. A scenario is a complete user task, such as "Creating a music sheet" (which means, clicking the New menu item under the File menu, filling in the wizard page, clicking Next, filling in the next page, and clicking Finish) or "Adding notes" (which means, clicking a button in a palette, dragging it to the music sheet, dropping it on the music sheet), etc. So, you're trying to simulate a user task by means of small tests that the test infrastructure processes for you. My earlier blog entry Testing... Testing... One, Two Three introduces the various test tools available to you in the NetBeans world. When it comes to functional testing, one should immediately think two words: "Jemmy" and "Jelly". This blog entry is about these two, how to set them up, and how to write your first Jelly test, which will invoke the "New" action, by choosing the "New" menu item under the "File" menu in the JFugue Music NotePad. You will be able to run the test either from the IDE or from the command line, using Ant.

  1. Start up NetBeans IDE 6.0 Beta 1.

  2. Go to the Plugin Manager and install the Testing tools:

    It's really cool that they're now in the Plugin Manager. Previously, as I explained here, you had to go elsewhere.

  3. Once you've installed them, go to the IDE's installation folder. Not the user folder! The installation folder i.e., the parent folder of the folder where the IDE's launcher is found. You will see a brand new folder called "testtools":

    Notice that it also contains the xtest-distribution folder.

    When you expand the modules folder, you'll see all the JAR files containing the functionality you will need for testing:

  4. Now, in the IDE, let's set up the testing infrastructure, after which we'll write our first test. In the Projects window, right-click on the module project that you want to test. In the New File wizard, choose "XTest Infrastructure", in your new "Testing Tools" category:

    When you do so, the IDE shows you where everything that it will create for you will be put:

  5. Click Finish. Now open the Files window. You should see that your module has a new subfolder called "test":

    That's your testing infrastructure.

  6. Let's now create our first GUI test. Back in the New File wizard, choose "JellyTestCase Test", as shown here:

  7. On the next page, make sure that you select "Functional Test Packages" in the "Location" drop-down, as shown below:

  8. Click Finish. You now have a skeleton for a new Jelly test. In the Projects window, you have logical nodes for your testing locations, giving you easy access to your test case, i.e., without needing to dig around for it as you need to do in the Files window:

  9. Now add the single line shown at the start of this blog entry to the test1() method, and get the correct import statement, both of which are shown in bold below, all the rest was generated for me by the wizard mentioned above. 

    package org.netbeans.modules.musician.test;
 
import org.netbeans.junit.NbTestSuite;
import org.netbeans.jellytools.JellyTestCase;
import org.netbeans.jellytools.actions.Action;

/**
 * A Test based on JellyTestCase. JellyTestCase redirects Jemmy output 
 * to a log file provided by NbTestCase. It can be inspected in results. 
 * It also sets timeouts necessary for NetBeans GUI testing.
 *
 * Any JemmyException (which is normally thrown as a result of an unsuccessful
 * operation in Jemmy) going from a test is treated by JellyTestCase as a test
 * failure; any other exception - as a test error. 
 *
 * Additionally it:
 *    - closes all modal dialogs at the end of the test case (property jemmy.close.modal - default true)
 *    - generates component dump (XML file containing components information) in case of 
 *      test failure (property jemmy.screen.xmldump - default false)
 *    - captures screen into a PNG file in case of test failure (property jemmy.screen.capture - default true)
 *    - waits at least 1000 ms between test cases (property jelly.wait.no.event - default true)
 *
 * @author gw152771                       
 * Created on September 26, 2007, 10:08 AM
 */
public class NewJellyTestCaseTest extends JellyTestCase {
    
    /** Constructor required by JUnit */
    public NewJellyTestCaseTest(String name) {
        super(name);
    }

    /** Creates suite from particular test cases. You can define order of testcases here. */
    public static NbTestSuite suite() {
        NbTestSuite suite = new NbTestSuite();
        suite.addTest(new NewJellyTestCaseTest("test1"));
        suite.addTest(new NewJellyTestCaseTest("test2"));
        return suite;
    }
    
    /* Method allowing test execution directly from the IDE. */
    public static void main(java.lang.String[] args) {
        // run whole suite
        junit.textui.TestRunner.run(suite());
        // run only selected test case
        //junit.textui.TestRunner.run(new NewJellyTestCaseTest("test1"));
    }
    
    /** Called before every test case. */
    public void setUp() {
        System.out.println("########  "+getName()+"  #######");
    }
    
    /** Called after every test case. */
    public void tearDown() {
    }

    // Add test methods here, they have to start with 'test' name.
    
    /** Test case 1. */
    public void test1() {
        new Action("File|New", null).perform();
    }

    /** Test case 2. */
    public void test2() {
    }
}

  10. Next, I needed to close the IDE and restart it (a known bug, because the right libraries are not put on the classpath until the IDE is restarted). Without doing this, compilation of the test case, shown above, failed. So, if it fails for you, you will, for now, need to restart the IDE.

  11. Now... right-click the project node, choose XTest, and then choose "Run qa-functional Tests", from the menus shown below:

    The tests defined as functional tests (based on their location, which you defined when you created the JellyTestCase Test in the New File wizard) are run. Alternatively, go to the command line, browse to the module's "test" folder and run the Ant script, as shown earlier: 

    ant -Dxtest.testtype=qa-functional

    Note: When you run the Ant script, make sure that the application is built. If it isn't built, the application will not start up. Instead the module will be tested within the development IDE.

    Results are written to the 'results' folder, accessible in the Files window:

  12. Finally, read Writing Jelly Tests in NetBeans, as well as all other documents under "Version for NetBeans Dev" on the Jellytools Documentation page. (The FAQ is excellent.) Remember, as pointed out in earlier blog entries on testing tools support for the NetBeans Platform, that a very handy overview can be found here. Also, have a look at Writing tests for XTest. For another sample, go to the New Project wizard and then inspect the two samples under Samples | Testing Tools, especially the second one, which is a NetBeans Platform application.

Thanks a lot to the NetBeans team's testing guru, Jiri Skrivanek, for helping me understand all of the above. While writing this blog entry, I received an e-mail from him containing a complete test for the JFugue Music NotePad. Apparently it automates the playing of the "famous song for children, 'Dog jumps over oats'". Hmmm. Not famous in Holland, where I'm from, anyway. Read this blog tomorrow and you'll find out what it's all about. :-)

In other news. This blog entry is dedicated to Tom Wheeler... who frequently asked me to research this and I apologize for taking so long to do it.

Sep 26 2007, 03:56:25 AM PDT Permalink