Wednesday August 29, 2007

Cold Fusion Moments I can't count how many times in grad school I discovered the equivalent of Cold Fusion. 

It was a fairly regular habit of mine to slip under my professor's door late at night a chart displaying freshly acquired data that would change the world, complete with an elaborate explanation, lots of exclamation points, and the opening sentences of my Nobel prize acceptance speech. 

I'd then walk home, blurry eyed and exhausted, but proud of my great discovery and looking forward to being showered with praises the next day. 

The next day I'd arrive back at the lab and look at my notes from the previous night and shout "Oh shoot!" (or something very similar to that) realizing the night before I had a cable disconnected, or had misplaced a decimal point by three orders of magnitude.  I'd retrieve the chart from the professor saying "Never mind!" which he never did because he had already figured out my mistake.

It is very easy to discover Cold Fusion buried in Web Analytics data.  The key is knowing when to publish it, and when not to.  In other words, if you find something very startling in your data, good or bad, as an analyst you need to be sure of your finding before you blow the horn and get everyone in the business worked up.

To that end, here are the rules of thumb I have learned to follow:

1.  Find at least one other way to make the same measurement.  Run collaborating reports to test and support your finding.  The first comment will always be "There must be a problem with your data", so you better be prepared to respond to that.

2. If possible, explicitly test the measurement.  Go through the user's experience yourself on the Web site and validate that the data is being produced as you assume it is.  (Many times it is not.  Surprise!)

3. Once you are confident in the data, anticipate the next two questions and start answering them.  Once others believe your data they will want to know "Now what?", "Why is that?", or "How did this happen?".  Pursuing these follow up questions not only helps the business take action, but also helps you build confidence in your original finding.

Bottom line is you need to do enough to be confident you're right, but not so much that you never share your results and take action.  Discovering that balancing point is key to being a successful analyst. 

( Aug 29 2007, 04:58:51 PM MDT ) Permalink Comments [2]