
This past May, I attended my first
CHI conference, which was held in exotic San Jose, California :) CHI is short-hand for the
ACM special interest group on Computer-Human Interaction, and the annual conference is the premier spot for presenting academic work and reports on what has been newly created or discovered in the field. I'd
co-authored a paper that was published at
CHI 2005, but didn't attend -- my colleague,
Nicole Yankelovich presented our paper that year.
About this time last year, I was busy writing up what my class-mates and I had worked on as a project for our Spring 2006 semester Master's class, Measuring the user Experience. The class was offered at Bentley College and taught by Dr. Joe Dumas, who is world-renowned for his work in the field of usability. Our class was experimental, in that it hadn't been run before, and Joe brought in real clients from industry for whom we could do work. The team that I was on was assigned to perform a massive heuristic review, but we faced a couple of problems: how could we combine the results of our reviews to present a single face to the client, and how could we prioritize the issues that we uncovered?
While several papers had been written on classifying the severity of a problem uncovered by usability testing, there was nothing that we could refer to regarding how to classify the severity of a problem uncovered by heuristic evaluation. The other challenge we faced was how to combine our ratings -- after all, we needed to evaluate more than 25 website features across 8 websites -- but the only guidance we found was from Nielsen, who recommended averaging the scores of the reviewers. We found two problems with that approach: first, to produce a reliable result, Nielsen recommended using 15 evaluators, and we had only 3; second, when we tried averaging our scores, we found that we had a result that none of us could defend.
The resulting method that we (okay, mostly my colleague, Hanna Yehuda) came up with, and its benefits were the topic of our Experience Paper for CHI 2007. I presented the material both at CHI, and then 3 weeks later at the Boston UPA Mini Conference on Usability and User Experience. Both times, the paper was well received, but the discussions that arose from the presentations were very interesting; so interesting, that a former professor of mine and I will be submitting a paper to CHI 2008, to discuss the topic fully and openly.