Katy Dickinson

How to Survey

Introduction

This is an updated external-to-Sun version of a web page I first created five years ago to bring together in one location key information and resources for how to conduct surveys. The initial audience for this information was the Sun Sigma (Six Sigma) professional community.

How do I know about surveys and data collection? I was certified as a Sun Sigma Black Belt in 2002 and have been serving as a Master Black Belt since 2002. I was also in one of the last classes that Dr. Deming taught on statistical management methods, in 1993.

Key Questions

Questions to ask yourself before starting to create a survey:

1. Why survey?

A survey is one of many good ways to collect information from customers. It may or may not be the best way for your situation. Have you considered other options such as field studies, baseline research, interviews, and focus groups? Many customer groups get surveyed over and over and get very tired of questions: what do you know about the data already collected from the target group?

2. What are the rules?

Information protection, security, and privacy are some policy and legal areas to consider before developing your survey. Local laws about sweepstakes and contests also need to be considered for some survey incentives. If you work for a company, there may be different policies for internal corporate surveys and external customer surveys.

3. What questions?

Developing survey content is as much an art as it is a science. How do you form questions so that the results can be usefully analyzed? What will you do with the answer to each question? How many questions will your target audience answer before abandoning the survey? Is the way you ask the question clear to people from other contexts and countries? An excellent survey takes time and testing to perfect. If you are new to surveying, consider asking a consultant expert for support.

4. What tool?

If you are not having someone else create your survey, there are a number of tools available to you, details are available below under Tools and Services.

Tools and Services

• Sample Size Calculator, Creative Research Systems web site tool
"This Sample Size Calculator is presented as a public service of Creative Research Systems. You can use it to determine how many people you need to interview in order to get results that reflect the target population as precisely as needed. You can also find the level of precision you have in an existing sample."
• 3-way Percent Calculators
• Web-based surveys are a best practice. When I create a web based survey, I do not use a third party survey tool. I work with a Sun Engineer to create and test my survey using HTML, PERL, and CGI scripts. For those to whom these are not easily available, here are three survey tool companies which have been recommended to me. Before starting to use a third party survey tool, be sure to consider who owns your survey data, how the privacy of your data is protected, and whether the tool company charges for larger numbers of responses.
• Zoomerang "Create custom web-based surveys and get rapid results. Start using the #1 online survey tool today!"
• VTSurvey "A web-based tool which enables end users to autonomously create and run online surveys, feedback or registration forms."
• SurveyMonkey.com "Intelligent survey software for primates of all species. SurveyMonkey has a single purpose: to enable anyone to create professional online surveys quickly and easily."

Reading

• "Ask Them Yourself" - How to survey your customers on the cheap, By Ellyn Spragins, FORTUNE - Small Business - Innovation, From the Dec. 2005 Issue of FSB
• "Keep Online Surveys Short" by (former Sun Distinguished Engineer) Jakob Nielsen - Alertbox, February 2, 2004
  "To ensure high response rates and avoid misleading survey results, keep your surveys short and ensure that your questions are well written and easy to answer."
• "Raising Your Return on Innovation Investment" By Alexander Kandybin and Martin Kihn, Booz Allen Hamilton, 2004
  "There is also is a flaw in the methods by which most companies go about developing new products. Focus groups and surveys elicit consumer opinions, but people can't know what they don't know."
• "Listening to the Voice of the Customer" by Mark Federman, Chief Strategist, McLuhan Management Studies, McLuhan Program in Culture and Technology, University of Toronto - November 28, 2001 (9 pages, PDF format)
• "Getting the truth into workplace surveys" by Palmer Morrell-Samuels, "Harvard Business Review", February 2002 - Reprint R0202K
• How to Conduct Your Own Survey by Priscilla Salant, Don A. Dillman. John Wiley & Sons (1994) ISBN: 0471012734
• An alternative to the survey:
  • "Field Studies: The Best Tool to Discover User Needs" by Jared M. Spool, Originally published: March 13, 2007
    "While techniques, such as focus groups, usability tests, and surveys, can lead to valuable insights, the most powerful tool in the toolbox is the 'field study'. Field studies get the team immersed in the environment of their users and allow them to observe critical details for which there is no other way of discovering."
  • "Risks of Quantitative Studies" by (former Sun Distinguished Engineer) Jakob Nielsen - Alertbox, March 1, 2004
    "Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research."

See my 1 May 2008 blog entry How to Survey, Part 2 (Best Practices) for more.

Posted at 04:00PM Apr 16, 2008 by katysblog in Mentoring & Other Business  | 

PreSEED-2 Application Status

Since the application web pages opened for use on 14 April, we have received 25 submissions, 19 of which are complete. The PreSEED-2 pilot mentoring term for Sun Software Members of the Technical Staff will accept up to 50 participants; it will run from June-December 2008. The initial applicant group is remarkably diverse geographically. We have submissions so far from China, Czech Republic, Germany, India, Ireland, Israel, Russia, and the USA. The due date for application submission is 21 April.

On 3 April, we announced PreSEED-2, the second pilot mentoring term aimed at helping Sun Engineering staff who have been getting almost all "Sun Standard" (2 or Standard-level) performance ratings onto a path which may lead them to higher engagement. The first PreSEED pilot term is currently under way, running from March-September 2008. The PreSEED-1 metrics and feedback so far are good and the same or better than metrics of a regular SEED worldwide mentoring term. We are now collecting the first formal feedback from PreSEED-1 mentees, managers, and mentors.

Software Chief Technologists Bob Brewin (Distinguished Engineer and Vice President) and Tim Marsland (Fellow and Vice President) are PreSEED's pilot term Champions. Greg Papadopoulos (Chief Technology Officer and Executive Vice President of Research and Development) is the SEED program executive sponsor.

PreSEED is a pilot of the SEED worldwide Engineering mentoring program. More information on SEED is available at http://research.sun.com/SEED/

Posted at 02:30PM Apr 16, 2008 by katysblog in Mentoring & Other Business  | 

SAMA Auction

John and I ran the checkout table at the SAMA (St. Andrew's Medical Assistance) event last Sunday night. Thanks to our many generous volunteers, donors, and guests, SAMA's Middle Eastern Feast and Auction were a delightful success. We are still counting up expenses and collecting payments.

For auction catalog and bidder management and check out, we used the Auction!, Version 3 product of Auction Systems, Inc. in Colorado Springs, CO. In general, Auction! worked well for our relatively-small charity auction (about 200 people and 70 items). Below is my view of the positives and negatives of the product. Bear in mind that I am a 26 year veteran UNIX user and I love my Mac laptop. I have been publishing hypertext documents for 16 years. I only use a PC when there is no alternative. Auction! is a PC-based product.

• Pro
• The Auction! product was easy to buy and was shipped promptly. A down loadable version was available immediately.
• The Auction! product worked well out of the box. That is, I was able to make it work without more than a quick read of the documentation.
• The design incorporates many good ideas on managing a charity auction that I would not have otherwise known.
• Auction Systems' tech support staff pick up on the first ring and are cheerful and helpful once they understand the question.
• Tech support was available by pager on the event day. We did not have time to ask them any questions but I liked knowing I could if there was a total mess up.
• I liked the Hint button - a good error checker.
• Con
• The design has many assumptions which are not documented about how auctions are best run. For a first time auction manager, this is a challenge. The information is implied but not explicit. There is minimal introduction to running an auction and why the product features are set up as they are.
• There is no way to export the catalog to HTML for web page display. It is possible to export a PDF version. I ended up typing auction item information twice - once into Auction! and again for the SAMA web page.
• The catalog does not allow the inclusion of photos. We prepared a JPG photo of each item for our SAMA web page. When I asked how to include these in the printed catalog, Auction Systems' tech support staff actually suggested that I use scissors and a glue stick!
• Document formats can only be changed within tight limits. For example, it was very convenient to be able to print labels with the bidder names and numbers to put on the back of each catalog. However, I could not resize the bid numbers to be really big. So, one of the SAMA team had to write the bidder number by hand on each catalog under the label. This allowed the catalog to be used as a bidder paddle.
• Sending a print command is a lengthy process that took several minutes each time: 1st - create the document or report. 2nd - click Print. 3rd - review the preview print document. 4th - click Print. 5th - pick the target printer. When printing checkout receipts, the minutes-long print cycle required by Auction! became a major bottleneck.
• The single-user software version was a challenge to use even for our small auction - many activities required multitasking. (For example, I could not answer a question about who won an item at the same time as printing out a checkout receipt.) We needed the higher-priced network version. I am not sure the single-user version is appropriate for any auction.

John guessed that the Auction! software was written 5 to 10 years ago and is "good enough" - so a wholesale rewrite has not happened. The design is showing its age. Nonetheless, I plan to use Auction! again now that I have some experience.

Posted at 01:54PM Apr 16, 2008 by katysblog in News & Reviews  |