News, updates, tips, tricks NetBeans Core QA

The goal when producing new version of NetBeans Platform is to be backward compatible, unless otherwise stated. This means that if you developed a module/plugin for version A of the platform then it should work even in platform version B that was developed a few years later. If there is a module for which the previous statement is not true, then the compatibility is compromised. Discovering such modules is generally bad sign. It can either mean that there is an unwanted incompatible change, and in such case we shall fix it prior to release, or this change is desirable and highly justified, and in such case it should be properly documented. In either case, backward compatibility is the strongest commitment of the NetBeans platform. Undesired incompatible changes have to be eliminated.

As the testing of the compatibility in an artificial testing environment might be helpful it wouldn't meet our goals. Therefore we(QE) started a community program that should give us more reliable results.

We received reports for 5 different products. You can look at full reports at http://wiki.netbeans.org/NB65BackwardCompatibilityTestingReport. I have to say that we expected more reports then "only" three. On the other hand, these reports are from people and companies that build real product on NetBeans Platform for years therefore we are happy that they haven't found any incompatibilities in NetBeans Platform - platform9 cluster of NetBeans IDE 6.5.

We will start similar compatibility testing for NetBeans IDE 7.0 when it will reach its Beta.

• Backward Compatibility Testing of NetBeans Platform

It seems that it repeats every release. The worst bugs appear a second before the end of the release cycle. I always think "did we forget to test this situation?". the answer is "No". We did our best. Unfortunately, we are not able to cover all the different configurations and usages that the users can create. As written in various books and papers it's mission impossible

We have a P1 in NetBeans 6.5 FCS candidate that looks really serious. The IDE starts to delete users' files on the disc. Real stopper, right?
Fortunately, it isn't easy to get into this situation. When you could face it? You need a project in a version control system (most likely SVN) then you have to "too many open files". It means exceed the limit for open files in your system. The it can happen that the IDE goes mad and deletes some of the files.

Good news is that we have fix and we are testing it. It will take a time... The fix is in Filesystems which is one of the core IDE functionalities. Everybody builds on it.

• P1 stopper
• one line fix

UPDATE: thank you to all who helped with the final testing of the latest fixes in NetBeans 6.5! We are done. No regressions, no new stoppers.

We are testing another build of candidate for Release Candidate 1. There were some stoppers that had to be fixed however it seems that we are finally close to the right build. If everything will go fine AND no new stopper will be found then we will publish the RC1 early next week.
Update1: there is maven cluster in the build AND it shouldn't be. That means another re-built. And one build takes 5-6 hours. What it all means together in Friday afternoon -> we'll finish it no Monday.

• Fixes in RC1

Note: the RC1 candidate build isn't available for download yet. We don't publish the candidates for candidates because we don't want to confuse our users with "several RC1 builds". When RC1 will be ready then it will be published on netbeans.org site and mailing lists.

There was recently a very hot thread on nbusers mailing list, discussing performance of upcoming release of NetBeans 6.5. To prove that we really care about voices from community NetBeans team contacted users, that were suffering with performance issues, and were willing to help us identify weak areas.
NetBeans team appreciates such feedback and cooperation from those users. And very big thanks to people, who shared their projects with us! It's really helpful, not all bugs are reproducible in "helloworld" project with a few classes, sometimes is needed huge application with large sourceroots. Whole this action resulted into couple of serious performance bugs, which are being evaluated/fixed.
Two cent's from Core QA Team to this action so far:

• issue 145546
• issue 145404

...and lot of others (performance unrelated), that would be really hard to reproduce without big projects provided by users.

We are close to finishing of testing of next patch for the NetBeans 6.1. It seems that the patch doesn't cause any regressions and fixes the bugs that it promised. Unfortunately, some of the bugs are almost unverifiable because they appear only in some unknown conditions. Anyway, we haven't found any regression during the testing that could be caused by the fixes.

The patch should be released on August 27 if there won't be any other issues reported by other teams.

• full list of the bugs that are fixed in the PATCH 3
• Sustaining pages about patch3

If you are not patient then you can try the patch now. On your own risk, of course. Add new update center and set up the URL field to this link. The updates will appear in the Plugin manager or in bubble.

Please, be aware of this bug when you are installing a patch - Bubble can install different set of update than via Plugins.

About month ago, NetBeans did a big change. The change was kind of low-level, so if you are an ordinary user, you didn't probably notice. But if you are a developer on NetBeans platform, or somehow actively involved in developing NetBeans modules, or IDE itself, you are certainly interested in this new approach to testing. So what was the change about? We got rid of Xtest - the harness that was used by NetBeans to run automated tests for our modules. Reasons why we did this? Xtest was kind of complicated harness, lot of Ant scripts including other scripts. Poor integration with IDE. Quite complicated configuration (which mostly ends by "I'll copypaste config from somewhere where it's working and change this and that, and it will probably work"). Our new approach is called "simpletest". It means mostly simplification, easier test setup and simple usage of test distribution. Nice change is that you can you can run now qa-functional test directly from IDE in the same way you were used to run unit tests. Test results are now produced in both HTML and Junit XML format - yes, you can take a benefit from this and run your tests on Hudson, you get a nice charts, results history and some other stuff. In overall, qa-functional and unit tests are now more aligned.

For more information, see Simpletest wiki. I contains:

• Migration guide - howto change your tests to be "simpletest compliant"
• New properties for running tests from Binary test distribution
• How to setup test configurations
• Discussion

This is a response for not a nice post about quality of NetBeans at nbusers mailing list. I thing that the original post is more misunderstanding and a email from an upset user.
To make some of the things clearer I'd like to describe how we test the Milestones. And how the milestone process should look like. I'm using word "should" because the world isn't perfect...

Milestone process

Let me describe it in words. Release engineers(RE) build the NetBeans builds. When the build is ready quality engineers(QA) start the testing of the milestone. Developer focus on fixing issues during that time. After a time (usually one week) new branch is created. QA identifies show stoppers these are bugs that have to be fixed before the milestone build can be published. When all the show stoppers are fixed the milestone is published and announced to the NetBeans community.

• Milestone process description on wiki.netbeans.org

We are testing accessibility this week. It means to check all the dialogs/menus/wizards/etc. in IDE that they implement Accessible interface, that the provide AccessibleName and Description. We also check the correct mnemonics, tab traversal. This kind of testing is required by Section 508 - more details.

We use a tool for our testing. It is UI Accessibility Tester. We install it as NB module and then run it with icon from NetBeans IDE. But sometimes you need to start the A11Y tester before the application is started. It's needed in cases when you are testing dialogs that appears right after the startup of NetBeans - e.g. Registration wizard.

Start UI Accessibility Tester before the application

1. Copy a11y.jar from a11y.nbm/netbeans/modules/ to $JDKHOME/jre/lib/ext directory under your JDK installation
2. Modify (create one if it doesn't exist) your $JDKHOME/jre/lib/accessibility.properties file to include the following line - assistive_technologies=org.netbeans.a11y.tester.UIAccessibilityTester
3. Start IDE

More detailed instructions at a1y.netbeans.org.

There is nothing like bugless software.


And NetBeans is not exception. We in QA do our best to find all the bugs that we can. However the bugs are hidden very well. Sometimes they appear in a special configuration. Sometimes we just miss them (yes, we are humans). The live cycle of the release can be long therefore some users don't want to wait for next release to get their bug fixed. They want them fixed immediately. That was the reason why NetBeans started to produce patches for the releases. That was just a background info for the patches in NetBeans.

We produced patch 2 for NetBeans 6.1. QE is responsible for the testing of the patch. If the fixes in patch doesn't break anything in the NetBeans that the users have been using for while. This time the problems appears at the end of the testing cycle. All it started with problem with rights of updater.jar on Mac OS. Unfortunately, the Mac users couldn't update the jar because of wrong setup of the rights. There is always a workaround. And they(.JARa and Jirka) found one. But then the real fun started. Another problem with user/admin rights. WTF! I don't know if you ever tried to test this scenario but it is enormously boring - install as admin, update, switch user, test it. This kind of scenarios cannot be automated :( My thanks to .JARa who took care about it.
The result was "we cannot provide update for everything". There are un-updatable parts of the IDE. This must be fixed for next release.

We are glad that the patch 2 is out (and we don't have to test it anymore, till the next patch). Enjoy the fixes!

• http://wiki.netbeans.org/NetBeans6.1PatchesInfo