Monday January 26, 2009

Unique User Measurement Error My father used to tell me, "Son, never confuse monthly unique users with people."

Well, actually, he never said anything remotely like that as he was a baker by trade and never touched a computer in his life.  But he did used to tell me never to mix yeast and salt, if that is of any use to you.

Unique users, of course, are not necessarily people.  They are simply counts of unique cookies (more precisely, unique visitor IDs) who visited your web site.  If people tend to use multiple computers to visit your web site, or delete their cookies, your unique user count will be inflated relative to the actual audience size.  (Stop me when I say something new!)

So, I figured it was high time we go measure this on our own Web sites and see how wrong our data is.  Fortunately, the analysis process is fairly straightforward.  You need a site that requires user log in.  You count the number of unique login IDs in a month, and divide it by the number of monthly unique users from your Web analytics tool for that same month.  This ratio, unique login IDs/unique users, is what I like to call the unique user correction factor.

To be precise, I'm ignoring the effect of multiple people sharing a single login ID, or a single person having multiple login IDs.  So, sue me.

On one of Sun's developer sites I measured a correction factor of 78%, meaning that I should take my monthly unique user measurement and multiply it by 78% to get a better estimate of my true audience size.  This, by the way, was for a site which at the time was using the 2o7.net cookie, but more about that later.

I was a bit surprised that the error was not bigger.  The well known comScore study of a couple years ago found correction factors more like 40% on the sites they studied, i.e. a much more serious matter.

The thing is, of course, is that you can't really extend the comScore results (nor my results) to your own web site.  This is because the correction factor is a complicated function of cookie deletion, multiple computer use, and visit return frequency for users on your site.  These factors will likely be different for your web site than other web sites.

Just to illustrate, if your user population deletes their cookies every day, but visit your web site only once per month, your monthly unique users measurement will be unaffected.  Or, if your audience visits your site every day, but never deletes their cookies, your data will also be accurate.

The error gets big when users delete frequently and return frequently.  (Similarly, if they use multiple computers and return frequently, your error will also be larger.)  Imagine one guy who visits your site every hour and deletes his cookies between every visit.  He is only one person, but will single handedly rack up erroneously huge user counts.

The message here is that you need to measure this for yourself so you can defend your own data within your own company. 

The next question is what happens if we switch from the 2o7.net cookie to a first party cookie?  I'll discuss that in my next entry.

( Jan 26 2009, 09:12:53 AM MST ) Permalink