Roumen's Weblog

How We Do Usability Studies of NetBeans

Yesterday I went to see one of our usability studies. NetBeans UI group has arranged two days of usability testing of the development version of NetBeans with emphasis on it's editor. It was an exciting experience so I'd like to share some of my insights.

At first, how does it look like. There are two rooms - one with many LCDs, panels and audio-video technics. This is the operator room. The second room is almost empty, there is only a table with a computer, a chair and two cameras. This is for the person I'll call for lack of more appropriate terms as a victim of the study, who is observed and recorded. I'll definitely take a camera next time to make photos of the rooms, they're on Czech technical university (Sun opened the lab there last year - see bottom of the article).

The operator speaks with the victim using by a microphone, he cannot hear us unless we press a button. We see and hear everything the user is doing and saying and see his screen which is recorded, too. We'll have a lot of material to study once we make videos out of this. The victims had 8 tasks to finish, they didn't have to do all of them. Some were rather of type "write a hello world!" and some included more advanced programming - e.g. rewriting of existing code.

We had a bunch of different people with different backgrounds, all experienced Java developers, some of them using Eclipse, some tried 3.x versions of NetBeans before and some used IDEs. How to recognize an Eclipse user? When he gets an error in the code he immediately clicks on the red "x" button in gutter and is irritated that he doesn't get assistance. The interesting part is that they do not learn from this experience and even they get a lightbulb as a hint elsewhere, they still click on the red sign. The only thing you can do when they click on the red "x" for 10th time is to laugh - I know this should be taken seriously, but I just could not help it (hopefully the victim cannot hear us). It's incredible to experience how much are people connected with their tools - they do lots of the tasks unconciously.

At the end each victim was thanked (it takes an hour or so) and interviewed. They also got a free t-shirt and a reasonable amount of money. One of the victims was really cool - he was happy about the t-shirt but was refusing to take any money. We were trying to persuade him - he did a good job... but he resisted so we had to keep it (we didn't spend it in the pub, honestly). The guy was so helpful that he has sent us an additional e-mail with what he forgot to tell us in the interview. What an amazing participation!

The results of the study will be published on netbeans.org, like the older ones (see usability reports on ui.netbeans.org) once processed. I made some conclusions for myself about the editor:

• Developers rely on the IDE when writing and editing code in a large way - any extra assistence we can add is good.
• Probably the most crucial part is what I call the "write - get error - fix" cycle. That's typical for Java, you get errors like uncaught exception, missing import, missing cast, etc. all the time. The developers are becoming used to this assistence so features like editor hints are very important.
• Developers do not remember and do not want to remember details of Java APIs. They're too complex, so they rely a lot on code completion and built in javadoc. If they need to recover manually from an error the compiler is writing, some of them can hardly do so without the IDE. For example, there needs to be a functionality to add an additional exception into the try-catch block, which is added according to the error - developers don't remember which exceptions need to be caught and it's time consuming to look it up. Some of them almost cannot do this manually today!
• Number of exchanges of keyboard and mouse needs to be minimalized. Especially for fixing inline errors - we should not force users to use mouse to find out which error is there. We need to provide a facility to go over errors and fix them one by one from a selection of fixes using keyboard.
• Discoverability matters, especially if we want other users to use NetBeans. I've seen it too many times - the developer is searching for a feature, he is hovering above it, you want to shout at him: "just click there!"... but he doesn't find it, gives up and does it manually. This is not an issue if people are used to the IDE but if we want users of other IDEs to take advantage of NetBeans, we need to work on discoverability.
• We need to remove anything that obstructs developers, because such things are really frustrating. All automatic features (adding of brackets, quotes, etc.) are very useful but need to be non-obtrusive.
• Developers want to use their own way of formatting.
• Majority of developers are most productive if they don't have to think about the language. They just want to add their features and the IDE needs to do the language-related things automatically (handling exceptions, adding imports, templates - even for if() and while(), adding casts, ...). These features need to be offered instantly.
• Refactoring matters. Developers are becoming used to it and start to use it for even basic tasks like renaming a variable in a for cycle.

The participants who know NetBeans from 3.x versions were very much surprised how much NetBeans has improved and became more intuitive and good looking. They find work with the IDE as quite a pleasant experience - nobody was frustrated. But they would really welcome improvements in how the IDE helps write java code to make them more productive.

The results of the usability study will be filed into Issuezilla as concrete suggestions how to improve the editor. Bigger features/changes will be planned for next releases. I think this study was very useful and shows how important the productivity-related editor features are.

[Side remark] I was half a day out of the office and when I came back I found out that they've published my CVS flash demo on netbeans.org... things happen so fast lately it's really hard to be the first one to blog about the hottest news :-)