Monday April 27, 2009
All
|
General
Monday April 27, 2009 Effects of Moving to a First Party Cookie
Picking up where I left off yesterday (okay, three months ago, but who's counting)...We took the plunge and made the switch from the 2o7.net cookie to a first party sun.com measurement cookie. Other than the self-serving benefits of improved data, it seemed like an improvement to have a browser image request go to metrics.sun.com rather than the mysterious 2o7.net.
The outcome was an improvement in several data quality metrics.
The correction factor for monthly unique users improved from 78% to 83% (see last entry for what this means).
Percent of users who delete their measurement cookie at least once per month decreased from 20% to 14%.
Percent of users who block the measurement cookie dropped from about 5% to less than 1%.
As part of this analysis we also backed out what percent of our users use more than one computer in a month to visit our site: 30%. Of course, this was unaffected by the cookie change. (It would have been remarkable if it had been. Its consistency was a bit of a control in our calculations.)
Accurately determining the cookie deletion rate and multiple computer use is an inexact science based on comparison of IP addresses, user O/S and browser version profiles for a given visitor ID. So, I don't bet my life on the accuracy of these breakdowns. Only the correction factor can be accurately measured; the constituent mechanisms behind it are slipperier to divine.
We've watched these numbers over nine months now since we made the change, and although they do vary a little from month to month, we have not seen a definite trend either up or down in any of them.
My assessment is that it was worthwhile to make the switch from third to first party cookies, although I would characterize the improvement in quality of data as modest. The observed difference between deletion rates for first and third party cookies is not dramatic. Add to that the fact that much of the error is driven by multiple computer use, and the bottom line ends up being that the change is favorable, but not eye popping.
Nevertheless, I'm much more confident now having evaluated these parameters for myself on our own site. I no longer have to rely on published data from other sources which don't transfer to our situation, and I am in a better position to defend the level of accuracy in our data.
Monday January 26, 2009 Well, actually, he never said anything remotely like that as he was a baker by trade and never touched a computer in his life. But he did used to tell me never to mix yeast and salt, if that is of any use to you.
Unique users, of course, are not necessarily people. They are simply counts of unique cookies (more precisely, unique visitor IDs) who visited your web site. If people tend to use multiple computers to visit your web site, or delete their cookies, your unique user count will be inflated relative to the actual audience size. (Stop me when I say something new!)
So, I figured it was high time we go measure this on our own Web sites and see how wrong our data is. Fortunately, the analysis process is fairly straightforward. You need a site that requires user log in. You count the number of unique login IDs in a month, and divide it by the number of monthly unique users from your Web analytics tool for that same month. This ratio, unique login IDs/unique users, is what I like to call the unique user correction factor.
To be precise, I'm ignoring the effect of multiple people sharing a single login ID, or a single person having multiple login IDs. So, sue me.
On one of Sun's developer sites I measured a correction factor of 78%, meaning that I should take my monthly unique user measurement and multiply it by 78% to get a better estimate of my true audience size. This, by the way, was for a site which at the time was using the 2o7.net cookie, but more about that later.
I was a bit surprised that the error was not bigger. The well known comScore study of a couple years ago found correction factors more like 40% on the sites they studied, i.e. a much more serious matter.
The thing is, of course, is that you can't really extend the comScore results (nor my results) to your own web site. This is because the correction factor is a complicated function of cookie deletion, multiple computer use, and visit return frequency for users on your site. These factors will likely be different for your web site than other web sites.
Just to illustrate, if your user population deletes their cookies every day, but visit your web site only once per month, your monthly unique users measurement will be unaffected. Or, if your audience visits your site every day, but never deletes their cookies, your data will also be accurate.
The error gets big when users delete frequently and return frequently. (Similarly, if they use multiple computers and return frequently, your error will also be larger.) Imagine one guy who visits your site every hour and deletes his cookies between every visit. He is only one person, but will single handedly rack up erroneously huge user counts.
The message here is that you need to measure this for yourself so you can defend your own data within your own company.
The next question is what happens if we switch from the 2o7.net cookie to a first party cookie? I'll discuss that in my next entry.
Monday December 08, 2008 Esther Duflo is a professor of economics at the Poverty Action Lab at MIT. At the risk of oversimplifying her research, basically what she does is apply multivariate testing to economic aid for impoverished countries. Paraphrasing Tim Heffernan's Esquire article, say you spend a million dollars in Africa supplying textbooks, or treating students for parasitic worms. Which investment is more successful at improving academic performance?
Turns out the answer is treating for worms so that students attendance improves and thus they can learn more, even with outdated or no text books.
To quote the Esquire article summarizing the aid problem, "When you apply for funds to keep your project going next year, all you can say is...money helps. So please give us more of it." To address this problem, Professor Duflo and her team are applying data analysis to large scale, complicated economic and societal issues to cut through arguments over where funding for economic aid should be spent.
Sound familar?
So, if they can apply MVT for big problems like this that really matter, seems like running some simple A/B tests to improve my calls to action on my Web site should be something I can pull off.
( Dec 08 2008, 08:44:08 AM MST ) Permalink Comments [1]
Thursday October 23, 2008 I was fortunate to attend the Emetrics Summit this week in Washington, DC. As always, it was a great conference. Here are my general impressions.
An unavoidable theme was concern over the economy. Attendance seemed a bit down, and many conversations included some sighs and rolling of eyes about upcoming budgets and staffing. Other than those happy thoughts, there was much of interest.
Major themes I picked up on were:
Where are the case studies and success stories of major business decisions being driven or influenced by Web Analytics results? Plenty of examples of web optimization based on WA data, but has this work become a significant driver of the core business?
The "Web" in Web Analytics is fading with the topic being more simply "Analytics", i.e. Marketing Analytics or Business Intelligence, or whatever you want to call it. It is getting harder to put a fence around "Web Analytics" as it is just part of the marketing data ecosystem.
There was an interesting panel discussion about privacy issues: pending and potential litigation; inconsistencies between privacy in different environments (e.g. people willing to share everything on their Facebook page, but also deleting cookies to preserve their privacy); privacy vs. value to the customer (no problem if my grocery store tracks everything I buy as long as I get 50 cents off mayonnaise, but don't you dare give me a metrics cookie as I get no value from it). Policy and standards development here is slow coming, but still seems like the elephant in the room.
Seems to me to be a flood of A/B and MVT vendors and solutions. Seems everybody was touting their version of a solution. Yet, I did not hear too many examples in conversations about successful MVT testing. Maybe I was just talking to the wrong people, or went to the wrong talks. Or maybe it is harder than people are willing to admit.
I noticed more attendees from the public sector (e.g. government) than in the past. Individual conversations with these folks is always mind shifting to a person in the private sector as they have special challenges around quantifying and measuring their goals which are not as simple as, say, counting revenue. How exactly does one measure, for example, if one's Web site has contributed to responsible policy formation for United States energy resources. I would like to see a pubic sector topic raised to keynote presentation status at a future Emetrics Summit.
More talks than I've seen in the past about artificial intelligence. Still seems a bit on the forward edge, but quite a bit of interest at the talks.
Speaking of forward edge, went to a talk on metrics in the virtual world, e.g. Second Life. For starters, I would probably be corrected for calling it a virtual world, but my ignorance aside, the talk made some interesting points about the ability to build brand with avatars, challenges this space is having as it matures, and an assertion, probably true, that this will be an environment which will require involvement from the marketing and measurement community as a matter of course before long.
( Oct 23 2008, 04:23:38 PM MDT ) Permalink Comments [1]
Friday April 11, 2008 Please Enter Though the Garage
Quick! What percent of your web traffic enters though your home page?Wrong.
Take a close look at the entry pages to your web presence and see what percent of your visitors actually enter on your home page. I bet it's a lot less than you would assume before looking at the data, especially if your site is for a large enterprise.
Why is this? Because home pages are like the front door of your house. It's where strangers and people who have never been there before probably go. But friends, frequent visitors--they come to the side or back door because they know that's the best way to get inside. The quickest way to the kitchen where you keep the beer.
Your front enterence should make a good impression, of course, and be welcoming to visitors.
But it's an easy pitfall to put too much emphasis on your home page, and overlook what most of your customers (your best customers) are probably doing.
Thursday February 07, 2008 Are You Smarter Than Homer Simpson
My favorite episode of The Simpsons has a scene in which Homer goes into the witness protection program and the agents have to coach him to learn that he now has a new name. After many frustrating hours of them trying to get him to respond to his new alias, Homer leans over and whispers to the agent sitting next to him, "I think he's talking to you."Sometimes I'm not sure I catch on a lot quicker than Homer.
My group and I spend tremendous effort in Sun trying to enable the latest and greatest Web analytics measurements-- custom events and special variables and bells and whistles to try to make campaign measurement more insightful or efficient. Marketers and web area owners are constantly asking for more and increasingly complicated measurement features so they can optimize their area of the business better. We try to please. Sometimes we do.
But then the CEO pops out a 10 second email asking a simple thing like how many downloads of X we had over the last two years. Frenzy! Panic! Fire drill teams spring into action! I can readily tell you the clickpaths and fall out rates of users coming from search key word "xVM" last month in Germany vs. Kazakhstan. But I'm not ready to answer a simple question from the CEO.
Thus, the lesson that I am as slow to learn as Homer was his new name. Do the simple stuff first. Do it well. Make sure it works. Make sure it's right.
( Feb 07 2008, 11:22:57 AM MST ) Permalink Comments [1]
Friday November 16, 2007 Every so often it needs to get pruned down, then built back up with the latest, greatest metrics. (I think that's a mixed metaphor.) Corporate and department goals and priorities change and evolve, and metrics reports need to do the same.
As part of the redesign I've asked one of my senior analysts to propose some new metrics. Using best practices she's been very conscientious about including only "actionable" metrics. However, as we progress through the process, and think about the audience that will be reviewing the report (Directors and VPs) I wonder what "actionable" really means at that level?
Online revenue is a great example. I can't imagine not having this metric in the report, but how actionable is it really? If revenue is down, you certainly want to take action, but I suspect the action would be "Paul, go figure out why revenue is down!".
I think the right answer will be that some metrics are "bottom line" metrics and need to be included, but they should be surrounded by actionable metrics which tell me what to do if the bottom line (or top line in this case) slips.
Thursday September 27, 2007 You can find a podcast of this interview at
http://www.webanalyticsassociation.org/en/art/?433
in which I think I manage to say a few insightful things, a lot of maybe not so insightful things, and hopefully no outright bone headed things.
At any rate, if we were to run into each other at a conference and you asked me what Sun is doing about measuring Blogs, this is how the conversation would have gone.
Many thanks to Jennifer Day who conducted the interview and was patient and fun to engage with, and Jim Humphrys of the Research Committee of the WAA for inviting me.
Wednesday August 29, 2007 It was a fairly regular habit of mine to slip under my professor's door late at night a chart displaying freshly acquired data that would change the world, complete with an elaborate explanation, lots of exclamation points, and the opening sentences of my Nobel prize acceptance speech.
I'd then walk home, blurry eyed and exhausted, but proud of my great discovery and looking forward to being showered with praises the next day.
The next day I'd arrive back at the lab and look at my notes from the previous night and shout "Oh shoot!" (or something very similar to that) realizing the night before I had a cable disconnected, or had misplaced a decimal point by three orders of magnitude. I'd retrieve the chart from the professor saying "Never mind!" which he never did because he had already figured out my mistake.
It is very easy to discover Cold Fusion buried in Web Analytics data. The key is knowing when to publish it, and when not to. In other words, if you find something very startling in your data, good or bad, as an analyst you need to be sure of your finding before you blow the horn and get everyone in the business worked up.
To that end, here are the rules of thumb I have learned to follow:
1. Find at least one other way to make the same measurement. Run collaborating reports to test and support your finding. The first comment will always be "There must be a problem with your data", so you better be prepared to respond to that.
2. If possible, explicitly test the measurement. Go through the user's experience yourself on the Web site and validate that the data is being produced as you assume it is. (Many times it is not. Surprise!)
3. Once you are confident in the data, anticipate the next two questions and start answering them. Once others believe your data they will want to know "Now what?", "Why is that?", or "How did this happen?". Pursuing these follow up questions not only helps the business take action, but also helps you build confidence in your original finding.
Bottom line is you need to do enough to be confident you're right, but not so much that you never share your results and take action. Discovering that balancing point is key to being a successful analyst.
Thursday August 16, 2007 I was discussing some web analytics reports the other day with a colleague who was fairly new to analytics. He wanted to know what pages were referring traffic to his web page.
"Do you want to know what external sites sent you traffic or internal sites?"
"Well, both I suppose!", he responded.
So I explained that there were different reports for each.
We then went on to discuss external referrers that sent traffic directly to his page as an entry page, or referrers that sent users to his page after entering on another page and navigated to his page. And that search engines were a different question altogether. After a bit of going in circles I paused to explain to him that half the battle with analytics is precisely determining what you want to measure. There are many subtleties.
"Gee, it's amazing how accurate these tools are." he commented.
To which I had to explain , that, no, they are actually rather inaccurate given issues around JavaScript being disabled, blocking of third party images, blocking of cookies (but not all cookies), deleting some cookies (with some unknown frequency and probability), surfing with multiple browser tabs and windows and computers, improperly tagged sites, dropped tags, non-html content, RSS syndicated content, and on and on.
"We inaccurately measure very precise things!", I boasted with a heavy degree of schizophrenic pride and distain for my own profession.
I haven't heard from him since.
( Aug 16 2007, 08:31:59 AM MDT ) Permalink Comments [1]
Friday April 13, 2007 Quantum vs. Classical Web Analytics
Let me drag you through your science lesson for the day. I promise you won't need your white lab coat or safety goggles.
There are two basic scientific models of the world. The older one, dating back to Sir Issac Newton is commonly referred to as "Classical Mechanics" and describes the world in terms of continuums. That means that measures of the physical world can be infinitely and arbitrarily broken down into smaller and smaller units. A unit of energy, for example, can be divided in smaller and smaller pieces with no limit to how you divide it or how small you make it.
This model worked fine to describe things for hundreds of years, up until around the early 20th Century when scientists began observing phenomenon that the Classical model could not explain. They found that in reality, the world can not be broken down into arbitrarily small units, but rather, was better explained by a model consisting of small, discretely sized units, or "quanta". This became known as quantum mechanics.
So, what the heck does this have to do with Web Analytics?
Web Analytics is going the opposite direction. We are transitioning from Quantum Web Analytics to Classical Web Analytics.
The paradigm that Web Analytics has been based on is a "quantum" model of page views. The user experience has been conveniently described as a series of discrete steps, from page view to page view. However, this model is starting to break down. With the introduction of new rich internet applications (AJAX, videos, etc.) the user experience is no longer a herky-jerky succession of steps from one page to the next. It is becoming a smoother flow from one activity to the next--more of a continuum rather than a series of steps.
How this change in paradigm will ultimately impact the way we think about web analytics is yet to be seen. However, we might be able to draw upon hundreds of years of scientific models to provide guidance.
Thursday April 12, 2007 The Three Headed Monster. My experience has shown me than an Emetrics Group has three major internal clients: Web Marketing, Web Design, and Web Management. To fully add value you need to effectively serve each of these clients. If you ignore one, the whole value proposition breaks down. Here's why.
My simple minded view of a Web business is like this. Step 1: Bring in lots of qualified prospects. Step 2: Convert them. Step 3: Count your money. (I should teach at Wharton.)
The internal clients, obviously, map to these three steps.
Web marketing's job is to bring in gobs of potential customers itching to buy your products. But doing so is worthless if when they get to your site, the Web design is so lousy that they can't accomplish (convert) their goals. And the accomplishments of both of these organizations is diminished if Web Management doesn't have proper visibility to the bottom line, and the things that lead up to it. If you put too much focus on one area while neglecting the others, you might as well not bother with any.
So, those are five things I've learned so far. Hope you've found them useful.
Friday March 30, 2007 You Need a Customer for Your Analysis: I think there are different schools of thought about how to choose topics to investigate. One is to just start exploring the data and see what you can uncover. Identify those big new opportunities that had been unknown and then advocate for change based on this new intelligence. The other approach is to identify internally where the current business focus, existing resources and funding are and dive into that area to provide guidance and insight to make these existing efforts more successful.
Maybe I'm just too pragmatic, but I've seen more value come from analytics by supporting existing initiatives than trying to create and advocate new ones. I know that sometimes you need to search out the new ideas, but there is always (around here anyway) so much near by opportunity to contribute to and see immediate value that I tend to steer my group to working with people who are ready to take action and accept help.
Web Analytics is a Profession, not a Project: I think this fact has become rather obvious in the last few years with the creation of the Web Analytics Association and the boom of the industry.
My point, however, is to bring people into your group who have internalized this and are committed to the field. I like to tell people that "Nobody ever said Web Analytics is easy" which is rather ironic because people actually say it all the time. It's just that they're wrong. It's not easy and it takes a real commitment to learn how to be good at it. And if the people in your analytics group are not viewing this as a career, they are unlikely to last and be successful.
Sunday February 25, 2007 I was going to write an entry listing the top five things I've learned so far about managing a emetrics group, but I'm feeling kinda lazy, so I think I'll start with two. I can add the other three next time just to increase the suspense (and after I think of three more).
Punchin' Out Chryslers: The beginning of the month is a very busy time for my group. We have numerous reports that we need to produce, each one targeted for a different audience and containing mainly different data. And everybody wants their reports done right away.
Of course, there's only so much you can do all at the same time, and only so much you can automate. Sure, you can create pretty automated dashboards from web analytics tools, but not every piece of data spits out of such a tool, plus the real value is in taking the time to review the data, interpret it, and boil its essence down to a one sentence business friendly sound bite. No analytics tool will do that for you.
Thus, you need to think in terms of metrics production, like you were running a factory. Reports need to be production friendly, there needs to be schedules that everybody knows they need to meet, and all hands need to pitch in to spread the work around so it can be done quickly. And you have to keep looking for ways to make this all run like a better oiled machine because requirements keep changing and new metrics requests keep coming. Which leads me to point number two...
Prune The Tree: It has become apparent to me that people are good at asking for more metrics, but rarely volunteer to let you stop producing any.
There is a constant desire to know the next thing, to know more, to follow the latest new project or initiative. Quite quickly a five page report becomes fifteen and too much for any reasonable person to read (and a production burden...see Point One above).
So, you have to prune back the charts and data that have lost their novelty and impact. Alas, attention spans are short and charts that show an essentially flat trend line just don't garner much interest or action. Identifying these less useful metrics and trimming them away is a necessary and sometimes delicate action you have to take.
Wednesday January 03, 2007 It was clear from the data that there has been a fall off, and we were even able to point to a particular page where users fell out of the process. Click path reports told us where the users were going.
"Thanks, Paul, but why are they doing that? That's not what we expected them to do."
"Well, Joe, Web Analytics are voyeuristic, not omniscient."
I'm not sure he liked that answer, but it was true. We can observe what people are doing, but we can only guess at why.
This is the stage, of course, where team work and collaboration kick in. We looked at the page and postulated a few possibilities, but really the experts in design and usability take over here. I offered to help run some A/B tests to validate design tweeks, but beyond that, I had to accept the limits of my abilities.
| Archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Language | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Referrers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Today's Page Hits: 68 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||