Wednesday August 29, 2007

Cold Fusion Moments I can't count how many times in grad school I discovered the equivalent of Cold Fusion. 

It was a fairly regular habit of mine to slip under my professor's door late at night a chart displaying freshly acquired data that would change the world, complete with an elaborate explanation, lots of exclamation points, and the opening sentences of my Nobel prize acceptance speech. 

I'd then walk home, blurry eyed and exhausted, but proud of my great discovery and looking forward to being showered with praises the next day. 

The next day I'd arrive back at the lab and look at my notes from the previous night and shout "Oh shoot!" (or something very similar to that) realizing the night before I had a cable disconnected, or had misplaced a decimal point by three orders of magnitude.  I'd retrieve the chart from the professor saying "Never mind!" which he never did because he had already figured out my mistake.

It is very easy to discover Cold Fusion buried in Web Analytics data.  The key is knowing when to publish it, and when not to.  In other words, if you find something very startling in your data, good or bad, as an analyst you need to be sure of your finding before you blow the horn and get everyone in the business worked up.

To that end, here are the rules of thumb I have learned to follow:

1.  Find at least one other way to make the same measurement.  Run collaborating reports to test and support your finding.  The first comment will always be "There must be a problem with your data", so you better be prepared to respond to that.

2. If possible, explicitly test the measurement.  Go through the user's experience yourself on the Web site and validate that the data is being produced as you assume it is.  (Many times it is not.  Surprise!)

3. Once you are confident in the data, anticipate the next two questions and start answering them.  Once others believe your data they will want to know "Now what?", "Why is that?", or "How did this happen?".  Pursuing these follow up questions not only helps the business take action, but also helps you build confidence in your original finding.

Bottom line is you need to do enough to be confident you're right, but not so much that you never share your results and take action.  Discovering that balancing point is key to being a successful analyst. 

( Aug 29 2007, 04:58:51 PM MDT ) Permalink Comments [2]

Thursday August 16, 2007

Inaccurately Precise Sometimes this job makes me crazy. 

I was discussing some web analytics reports the other day with a colleague who was fairly new to analytics.  He wanted to know  what pages were referring traffic to his web page.

"Do you want to know what external sites sent you traffic or  internal sites?"

"Well, both I suppose!", he responded.

So I explained that there were different reports for each.

We then went on to discuss external referrers that sent traffic directly to his page as an entry page, or referrers that  sent users to his page after entering on  another page and navigated to his page.  And that search engines were a different question altogether.  After a bit of going in circles I paused to explain to him that half the battle with analytics is precisely determining what you want to measure.  There are many subtleties.

"Gee, it's amazing how accurate these tools are." he commented.

To which I had to explain , that, no, they are actually rather inaccurate given issues around JavaScript being disabled, blocking of third party images, blocking of cookies (but not all cookies), deleting some cookies (with some unknown frequency and probability), surfing with multiple browser tabs and windows and computers, improperly tagged sites, dropped tags, non-html content, RSS syndicated content, and on and on.

"We  inaccurately measure very precise things!", I boasted with a heavy degree of schizophrenic pride and distain for my own profession.

I haven't heard from him since.

( Aug 16 2007, 08:31:59 AM MDT ) Permalink Comments [1]