Skrocki's Blog: Sun, Social Networking, Random Thoughts...

I observed a usability study today in Sun's usability lab. Martin's group is conducting a usability study on a number of new sun.com search ideas that are being kicked around. I began program managing sun.com's search a few months ago, so I figured this was a great opportunity to not only learn more about the science of search (a LOT to learn), but also learn more about usability studies and user input on search. I've initiated a number of usability studies on various projects, but never watched them real-time.

The study consists of several one hour sessions with a single interviewer asking a single interviewee scripted questions. The types of questions depend on the type of study (branding, functionality ease of use, new concepts, etc.). In this case we are interested in gathering ease of use information for existing functionality as well as input on some new concepts that are not yet deployed.

The person conducting the interview is a non-Sun person and clearly states this to the users up front. The idea is they'll feel comfortable stating their honest opinions vs stating what they think we want to hear. She did a great job of drilling down on responses to ensure the users comments were clearly understood, but avoided influencing their opinions with repetitive or leading questions like they do in the movies until some poor sucker finally confesses to a murder s/he didn't commit.

The lab is a lot like what you see in the movies during an interrogation scene - bright room with the interviewer/interviewee on one side separated by a sound-proof one-way mirror with observers sitting in a dark room on the other side. Once I got over the "Wow this is cool! What does this do?" phase of getting familiar with the lab and began focusing on the test in-progress, I found the user responses quite interesting, surprising and at times rather humorous. For the most part, the validation of ideas was good, but I found the negative responses far more intriguing.

One of the specific points this study validated was users often abandon site navigation if search results are present. The study also validated that a few features of sun.com's search that we suspect are ambiguous are in fact so and via the study we were able to gather a few ideas on how to improve them.

Overall, observing a live usability study was an enlightening experience. It was good to observe a non-Sun person using our stuff and nice to see search meeting the expected needs for the most part.