Sunday February 25, 2007
All
|
General
Sunday February 25, 2007 I was going to write an entry listing the top five things I've learned so far about managing a emetrics group, but I'm feeling kinda lazy, so I think I'll start with two. I can add the other three next time just to increase the suspense (and after I think of three more).
Punchin' Out Chryslers: The beginning of the month is a very busy time for my group. We have numerous reports that we need to produce, each one targeted for a different audience and containing mainly different data. And everybody wants their reports done right away.
Of course, there's only so much you can do all at the same time, and only so much you can automate. Sure, you can create pretty automated dashboards from web analytics tools, but not every piece of data spits out of such a tool, plus the real value is in taking the time to review the data, interpret it, and boil its essence down to a one sentence business friendly sound bite. No analytics tool will do that for you.
Thus, you need to think in terms of metrics production, like you were running a factory. Reports need to be production friendly, there needs to be schedules that everybody knows they need to meet, and all hands need to pitch in to spread the work around so it can be done quickly. And you have to keep looking for ways to make this all run like a better oiled machine because requirements keep changing and new metrics requests keep coming. Which leads me to point number two...
Prune The Tree: It has become apparent to me that people are good at asking for more metrics, but rarely volunteer to let you stop producing any.
There is a constant desire to know the next thing, to know more, to follow the latest new project or initiative. Quite quickly a five page report becomes fifteen and too much for any reasonable person to read (and a production burden...see Point One above).
So, you have to prune back the charts and data that have lost their novelty and impact. Alas, attention spans are short and charts that show an essentially flat trend line just don't garner much interest or action. Identifying these less useful metrics and trimming them away is a necessary and sometimes delicate action you have to take.
Wednesday January 03, 2007 It was clear from the data that there has been a fall off, and we were even able to point to a particular page where users fell out of the process. Click path reports told us where the users were going.
"Thanks, Paul, but why are they doing that? That's not what we expected them to do."
"Well, Joe, Web Analytics are voyeuristic, not omniscient."
I'm not sure he liked that answer, but it was true. We can observe what people are doing, but we can only guess at why.
This is the stage, of course, where team work and collaboration kick in. We looked at the page and postulated a few possibilities, but really the experts in design and usability take over here. I offered to help run some A/B tests to validate design tweeks, but beyond that, I had to accept the limits of my abilities.
Tuesday December 12, 2006 Consider again the issue of "new visitors". When this data was presented at a meeting an audience member said "So, this is just the number of people coming to our web site who do not yet have our cookie?" Well, yes indeed. Well simplified.
Somebody else could have looked at the same piece of data and asked, "If a user is using tabbed browsing, will he show up as a new visitor if he visits from two different tabs? What if he is using two different browser profiles? Then what?" Good questions as well. I suppose we ought to understand that.
So, let me ask. Is a web analyst more like the first person or the second?
( Dec 12 2006, 12:45:57 PM MST ) Permalink Comments [3]
Monday December 11, 2006 I think he'll thank me.
So, why would I be such an unscrupulous wretch, deliberately passing untruths to the person who has so much power to make my career successful or painful? Because that's my job.
I better start explaining. I was preparing some web data for him to use at a presentation he needed to make to his peers and his executive VP. Part of the materials included data about new and returning visits as well as visit number (i.e. is this the user's first, second, Nth visit?), and as I was assembling the slide I found it to be horribly complicated, burdened with nuance and caveats Were cookies deleted? What if the user used multiple computers? How long had we been capturing data for that web site compared to the other we sites? How do I explain the difference between a return visit and a return visitor? (I'm still struggling with that last one!)
So, I simplified the whole mess and gave him a simple sound bite.
The next day I was fortunate enough to be in the meeting when he presented this information. It was received by a smattering of nodding heads which subsequently proceeded to some valuable discussions about marketing strategy, rather than a rat hole trying to figure out what in the world that chart was trying to say. Thus, I successfully removed some trees so that they could see the forest.
So, am I advocating fudging of web data because it is too complicated? No! In fact, if an analyst on my staff were to pull the same trick with me I'd run him up the flagpole for not understanding the data (even if he actually did). My point is that as analysts you need to remember that it is your job to translate web data into business information and that process sometimes requires a judgment call if some details can be safely left out.
( Dec 11 2006, 09:44:25 AM MST ) Permalink Comments [1]
Tuesday November 21, 2006 We recently spent some time on vacation in Tuscany, exploring the beautiful country side and hill towns, visiting churches and museums, and of course, sampling the great food and wine of the region.
One day we found ourselves at lunch in Montepulciano at Osteria Acquacheta (Via del Teatro, 22). If you're ever fortunate enough to be there, don't miss the opportunity to dine there, assuming you can even get in. It's a real authentic Italian experience; family run, over flowing with locals, raucous and bursting with abbondanza. The kind of place where you slam your fist on the table and shout "Bene!" when they drop off your steaming plate of eggplant Parmesian. The kind of place where they serve a small miracle of simple bruschetta drenched in local Tuscan olive oil and enough garlic to deter Mussolini, but to try to recreate at home results in a humbling waste of bread and hope. The kind of place where the wine comes by the carafe rather than the bottle.
The kind of place where the darn waiter won't give you a wine glass for your vino. You have to use your water glass.
Wait! What's wrong with this picture? Why would such a perfect place be too lazy to give you a glass for your wine? They even carefully explain to you that it is a tradition at their restaurant that you only get one glass. Cheap skates!
But as you progress through your meal and juggle the competing priorities of drinking water and wine, you are forced to alternate between glasses of one or the other. And thus, you end up getting less inebriated as you slow your wine consumption and switch off with water in between. Brilliant!
It turned out to be a natural regulating mechanism and we left the meal perfectly delighted with the food, comfortably soothed by the wine, and well hydrated.
( Nov 21 2006, 04:35:00 PM MST ) Permalink
Saturday November 18, 2006 As I see it there are two fairly clear levels of privacy on the web. If a user anonymously visits a web site, then they should be treated anonymously. This seems to be the equivalent of window shopping.
That said, if I own a conventional bricks and mortar store, I feel I have the right to watch how users look in my windows and observe what displays catch their attention the most. I distinguish one person from another by an anonymous attribute. The guy in the green sweater is interested in the golf clubs at 10% off. The woman in the dress likes the brand of Italian shoes I'm offering.
The other situation is when a user comes to a site and logs in, telling me who they are. Now I know that Joe Smith is in my store and is interested in the sale on golf clubs. I might even give Mr. Smith an extra discount because I know he shops at my store often.
But there is this gray area on the web I'm not sure what to make of.
Let's say at one point Joe Smith told me he wants me to send him emails when I have some kind of promotion. I send a personalized email to Mr. Smith and he clicks on a link that brings him to my web site. He has not logged in but I could easily know it was him on my site. That seems a bit sneaky to me if I were to track him by name.
Let's make it even grayer. (Hmm. Can things actually be more gray or less gray?) I send Mr. Smith an email which he clicks to come to my site. I direct him to a personalized portal customized for his needs. The top of the page says "Welcome, Mr. Smith!". I have let him know that I have identified him by name. He didn't really ask me to, but he went along with the attraction to come to his personalized site. If I follow his actions now on my web site, I know exactly who is looking at what.
But what is his expectation? Does he have an expectation of privacy in this case? I really don't know. I kinda think he does. Until he takes an explicit action to log in and tell me who he is, he has not taken the initiative to identify himself. Yes, he chose to come to the personalized portal, but did he choose to become no longer anonymous?
So, I'm full of questions at this point and not so many answers. I welcome your thoughts on the matter.
( Nov 18 2006, 02:03:37 PM MST ) Permalink Comments [1]
Monday September 18, 2006 Our first deliverable has been to corral the herd of metrics reports roaming the business plains. (Alert: Heavy handed metaphor!) We have had plenty of data getting produced around the organization, but it has not been collected in any meaningful way, nor has it been regularly reviewed nor discussed. So, our first order of business has been to produce a monthly department emetrics report which gets reviewed each month with senior web management.
That is a great first step and, for now at least, we have the ear of management who actually seem to pay attention at the monthly staff meeting as I pour over charts and graphs and try to point out trends that somehow predict where the business is going. It's a fine start and has been successful at stimulating discussion. But I doubt that the interest will last as trend charts are slow moving creatures and for the most part are either pretty flat or so noisy that you can't tell what is going on.
But what is coming from this are questions. A trickle at first. But one question leads to another and each month I get to come back with some analysis following up on last month's questions. And thus the snow ball rolls.
I can see the wheels turning in the directors heads. We are making emetrics part of the conversation of the business. Now to just keep that conversation going!
( Sep 18 2006, 08:54:12 AM MDT ) Permalink
Wednesday August 16, 2006 Where Can I Get My Pointy Haired Toupee?
We have made enough progress with our efforts here that the company is willing to make a regular organization of us. Coalescence and critical mass ought to take us to the next level of helping to drive the business with analytics.
I can already see that insights I'll be able to offer in this blog will be changing. Formerly I worked directly with marketers and other first level practitioners to optimize their sites and campaigns. Now I'll be dealing a lot more with management, engaging them directly in the conversation of emetrics, seeing what kinds of data and messaging register, figuring out how to actually motivate organizational action based on emetrics.
If I actually make any sense of it I'll let you know! ( Aug 16 2006, 08:54:31 AM MDT ) Permalink
Tuesday June 06, 2006 I've written a couple entries below on dealing with conversion latency on a web site which is indeed a tricky problem. I don't pretend to have all the answers, but to provide some insight for myself I have instituted a customized measurement that is proving to be rather interesting. And because it is all already visible on our web site for anybody with the where-with-all to find it, I'm not really sharing any secrets here. (That should stave off the lawyers, I hope.)
My business question was "How long is the latency period between a user viewing our product information and purchasing a product online?"
Note that at least for now I am not asking about quantity, i.e. what percent of users convert from a product view to a purchase. I just want to build a histogram of the time lag (latency) between the two steps.
I have no choice but to explain this in the language of our vendor, Omniture, so bear with me. I suspect there are analogous ways to do this with other vendors but you'll need to do the translation yourself
The approach I took was to set an eVar on the product view page to capture the current date. I have linear allocation set for the eVar, an expiration of six months, and renamed the eVar "Product Research Date". I also set a custom event that I renamed "Product Research" and the s_products variable so that I can segment the data by product.
The code on the page looks like this (I X'd out the product name below so I don't reveal too much):
<script language="JavaScript">The outcome of this is that I can run a report for product XXXX, broken down by Product Research Date where my selected metric is "Purchase".
<!--
var s_events="event11";
var s_products=";XXXX Server";
var s_eVar22=new Date().getFullYear()+"-"+(new
Date().getMonth()+1)+"-"+new Date().getDate();
//-->
</script>
The trick is that the report must be run for a single calendar day at a time, say June 1. Then I see a list of all the days on which a user who purchased XXXX had previously viewed the product research page. The report looks kind of like this:
Research Date Purchases
2006-6-1 156
2006-5-31 34
2006-5-28 27
etc.
Then in a spreadsheet I subtract the research date from the day of the report and determine the latency:
Latency Purchases
0 days 156
1 day 34
4 days 27
etc.
Then, repeat the report for another day, determine the latency, and average enough data together until you have enough to build a histogram.
Showing real data would get me in trouble, but I can say that this leads to interesting analysis. Two teasers for you are that different products show clearly different latencies (like a server vs a simple software product), and that histograms of products with significant latencies often show bumps at 7 days, 14 days, and 30 days.
( Jun 06 2006, 10:19:09 AM MDT ) Permalink Comments [2]
Tuesday May 02, 2006 Thoughts on Multivariate Testing
One theme that stood out to me this year was a lot of talk about multivariate testing to optimize a web page. There were some good case studies presented and just the general buzz seemed to be that this next year would see MVT becoming common place rather than leading edge.
But as I listened and learned it became more and more clear to me that as useful of a tool as MVT is, it must be used appropriately. After all, when you run a MVT to optimize a page you are optimizing the page for a particular, explicit outcome. Often the targeted outcome is to increase conversion of a particular call to action. But while optimizing for this outcome, you may be de-optimizing other outcomes. And these de-optimized outcomes might not be obvious.
For example, you might optimize a page to be a lean, mean revenue generating machine, but inadvertently de-optimizing a softer metric like customer satisfaction. The user experience might become very efficient and utilitarian but end up leaving a bad impression. You might increase the likelihood of the current sale while negatively impacting your brand.
An interesting study would be to run a parallel MVT study with two outcomes, one being a practical conversion outcome like a click through, and the other being customer satisfaction or brand impression measured with a survey tool. Then compare to see if the same recipe optimizes both outcomes.
( May 02 2006, 08:24:03 AM MDT ) Permalink
Friday April 07, 2006 I think this technology convergence is inevitable and a good thing, but I left the meeting with a nagging feeling that I hadn't heard the phrase "user experience" nearly often enough in the last two days. Surely the user experience is core, but where does it fit into this cold language of emarketing and web analytics?
Here's one way to think about it.
A Web business breaks down to this simple model:
Demand Creation -> Conversion -> Business Results
which is influenced by these underlying factors:
eMarketing -> User Experience -> Customer Success.
eMarketing effectiveness is usually described in terms of ROI, but at its root should be viewed as a function of demand creation (both volume and quality) and conversion. And conversion is driven by the user experience. You can have a great emarketing program driving lots of well qualified customers to your site, but if they have a lousy experience when they get there, they are unlikely to convert.
Taking this dissection one step further, the user experience is comprised of the objective and the subjective. The objective piece being pure utility in that did the user accomplish what he wanted to, and the subjective piece being did the user enjoy the process. The objective, utility piece is well measured by traditional web analytics (conversion), and the subjective piece is measured by customer satisfaction (usually via a survey).
So really, although emarketing and web analytics are converging, I hope the discussion does not become obsessed with optimizing emarketing effectiveness as the end goal while forgetting that at the root of all of this is the user experience.
( Apr 07 2006, 09:23:18 AM MDT ) Permalink
Friday March 10, 2006 In Chapter 13 of Eric Peterson's book, "Web Analytics Demystified", he discusses conversion and defines conversion as
COMPLETIONS/STARTS = CONVERSION RATE.
Question: Duh or Not Duh?
Answer: Not Duh.
Although this definition seems obvious enough, the subtleties come out a few pages later...
"...when you are making a conversion rate measurement you want to use 'like' metrics, that is, visits for both the numerator and denominator, and not visits for the numerator and page views for the denominator."
"...also be careful not to mix and match metrics, that is, do not use 'completions' divided by 'respondents' as the former is a non-unique metric, the equivalent of a page view, and the latter is a unique metric, the equivalent of a visitor."
These points are worthy of discussion. Conversion is always expressed as a percent. This is a dimensionless quantity, a ratio of identical units which cancel, like visits/visits, not like submissions/visits or registrations/page views. (Having algebra flashbacks, yet?)
I bring this up because recently I had a discussion with some pretty smart marketing folks who got themselves all confused working through a conversion calculation. The calculation went something like this:
Impressions
Click throughs 1%
Visits
Response rate 1%
Leads
Qualified leads 25%
Proposals
Successful proposals 30%
Sales
They thought they needed to figure out how many unique users they needed to hit their sales target, but couldn't figure out how to work users into the equation. The thing is, unique users don't really factor into the equation.
Falling back to Eric's definition of conversion, you would deconstruct the above model like this...
# impressions completed (i.e. clicked)/# impressions started (i.e. served) = impression conversion = 1%
1 visit/clicked impression
# visits completed (i.e. successfully)/# visits started = visit conversion = 1%
1 lead/successful visit
# leads completed (i.e. qualified leads)/# leads started = 25%
1 proposal/qualified lead
# proposals completed (i.e. sales)/# proposals started = 30%
1 sale/proposal
# sales
When looked at this way, it becomes more clear what is being converted at each stage, that is
# "somethings" completed / # "same somethings" started.
The catch was to break out the assumptions about the transformations between dimensions--the change in units when the user moves from step to step. In the example, we are assuming that one clicked impression corresponds to one visit, one successful visit corresponds to one lead, and so on.
Of course, you must consider if the transformation assumptions are sensible. For example, might a user click on two banners and refer himself to the same visit twice? Probably not. Will a user submit two "contact me" requests to generate two leads in a single visit? Doubtful, but not out of the question. Could a single lead produce two proposals? Maybe, but not likely.
The bottom line is to know what you are converting (starts of something to completions of that same thing), and what assumptions you are making about the dimensional transformation from step to step.
( Mar 10 2006, 03:48:51 PM MST ) Permalink
Friday February 03, 2006 Same Visit Conversions
Assume a process which has four significant steps A,B,C,D. And for this example let's say that the same visit conversion from step to step looks like this:
A --90%-> B--5%-> C--90%-> D
or viewed as the composite
A --4.1%-> D
Granted these are exaggerated step to step conversions, but it helps make the point.
What is going on here? Well, it could be that Page B is poorly designed and really stinks at converting users to the next step. Or it could be (because we are talking about latency here) that Step B naturally leads to latency due to its place in the buying cycle (e.g. maybe it is a quoting page).
Let's say that it is indeed a quote generating page and the fact that users are ending their visit here seems reasonable, even necessary, if you were in their shoes. Is saying your conversion from A to D is only 4.1% have the business meaning that is useful to you? Let's look at this situation a little differently.
Population Snap Shot Conversion
In a previous blog entry I discussed the possibility of measuring "conversion" by taking the ratio of traffic at Step B to that of Step A. (Give that a read if you haven't because there are some significant assumptions to keep in mind here. ) Below is some example data that one might see:
A B C D
10K 9K 5K 4.5K (visits)
Taking the ratio of traffic at each step gives a "conversion" like this:
A --90%-> B --56%-> C --90%->D
This ratio approach I call the "Population Snap Shot Conversion". At any given instant you are taking a snap shot of how many users are at the different stages of the buying cycle. It is a way to gloss over the latency problem that affects the traditional "same visit" conversion, and provided that you are clear in communicating what you are measuring, can provide more meaningful business insight to how users are converting through your buying process when you have a big latency gap. The example above suggest that rather than 5% of your users converting from Step B to C that actually 56% who visit Step B eventually come back to visit Step C at some point. Of course, these are different users at B and C at this instant, but by making some assumptions about the population you get a different view of your buying cycle conversion.
But we're not done with the caveats yet! One of the assumptions to this measurement being somewhat meaningful is that you didn't just send a boatload of people to Step A because of a new promotion you launched.
Buying Cycle Equilibrium
The Population Snap Shot Conversion assumes that at any step in the buying cycle there is not a significant transitory spike due to some perturbation such as a promotion. The concept is that the buying cycle is in more or less equilibrium, meaning that there is not a bubble of users at any particular stage. Such a bubble would temporarily inflate the number of users at a certain stage and throw off the ratio of users who make it from stage to stage as motivated by the value of your products and usefulness of your web site. That unperturbed ratio is what you care about.
Thus, this approach is unsuitable for certain businesses. But for businesses where effects of promotions are damped out by stronger factors that drive the natural buying cycle, Population Snap Shot Conversion can help you look past latency and give insight into what percent of your users are coming back to move to the next step of the process.
Still more to discuss on this. Stay tuned.
( Feb 03 2006, 06:10:57 PM MST ) Permalink
Monday November 28, 2005 What is Web site "conversion"? It is typically simply stated as the percent of visits which progress from one step in a Web process to another step. For example, if the shopping cart gets 1000 visits and 100 of those visits proceed to the "Thank You" page, that's a 10% conversion rate.
But what I never hear stated when conversion rates are discussed is the time frame of the conversion. It is usually assumed to be within the same visit (which could be any time frame as long as there is not a 30 minute dead time to terminate the visit.) However, such a simplification is not always warranted, especially if your business is complicated enough that it requires your users to return to your site multiple times over multiple days to complete the process. In this more complicated case, the notion of "latent conversions" must be invoked, and a time frame must be specified.
Latent Conversions.
Consider the case where customers generate and save quotes for a product on your web site, then need to go and use this quote to get a P.O. approved by their management, and then come back a few days later to the Web site and proceed from the saved quote to completing the purchase. How do you correctly measure and specify this conversion?
The stated conversion only makes sense if you specify a latency time frame.
Simplifying the example above, say 500 users save a quote on a given day, 100 of them return to make a purchase, and by some miracle every user who comes back to make a purchase does so in exactly seven days. Then you can state a "seven day or less latent conversion" of 100 divided by 500 which is 20%. The traditionally stated "same visit" conversion would be 0% because nobody progressed from Step A to Step B in the same visit.
Measuring Latent Conversions
Assuming you really want to measure latent conversions, how would you actually do this? Modern Web Analytics packages actually allow for this kind of measurement by letting you store previous activity of a user in a variable that is tied to a cookie, so that if a (unique but anonymous) user takes several visits to convert, the tool records this. Of course, this is all only as accurate as the user's cookie, but that is a whole other topic.
Another approach is to make some assumptions about your users. If you want to measure the latent conversion of your users from Step A to Step B, you could simply on a given day measure how many users completed Step A and how many (probably totally different) users completed Step B. Then divide the number of occurrences of Step B by the number of occurrences of Step A and call that the "conversion".
The big assumption here is that the users who are at Step A on that day are statistically indistinguishable from the users who are at Step B. Although the users certainly are distinguishable, you could choose to make the approximation that their characteristics and biases and motives are not distinguishable and thus the statistical assumption is defensible. How accurate this assumption is certainly depends on your Web site.
(Note that this approximated conversion is different from the "traditional" definition of web conversion which considers contiguous paths of specific users progressing from Step A to Step B. If your latency is seven days, the same visit conversion is zero!)
Other than the major assumption of indistinguishability of your users, this approximation approach has the major drawback that the latency period is unknown and unspecified. You would need to state clearly the assumptions you are making in this conversion approximation.
There is more to consider on the topic of latent conversions, such as multi step processes and distributions of latency, but I'll save that for another blog entry.
( Nov 28 2005, 11:08:46 AM MST ) Permalink Comments [6]
Thursday November 03, 2005 Who's Your Daddy--er, I mean, Customer?
And like any teaching experience, the teacher typically learns as much as the students do. I really enjoy working with the students each year (generally seniors and graduate students in the computer science department) because they ask questions that I would never hear around my normal work environment. The questions are often so fundamental that they come like a slap in the face--challenging concepts that I have long since taken for granted. And I'm refreshed by having to scrape the crust off of thoughts that I have considered settled and closed.
Each year is different, but a theme that arose last night went kind of like this.
"Business metrics? My Web site is not about business."
"I'm not sure who my customer is?"
A little background. These students are working on senior projects that revolve around redesigning a user interface of some sort. They pick their projects from proposals submitted by organizations or individuals outside the class who have a web site that needs improvement. So, it's a win-win. The sponsor gets a spiffed up web site, and the students get real work experience.
The twist is that many of these projects tend to be outside of regular "ecommece" sites. A few class project examples are: an application that displays geographical location of sensors worn by park rangers doing wilderness work to help with search and rescue if they go missing; a site that provides interesting weather data and maps for teachers to use in the classroom; a tool that makes interpretation of earthquake data easier to interpret. Really cool stuff! No boring old sites trying to coerce some chump into buying whatever widgets you happen to be selling today.
Thus, as I proudly instructed the students to (1) Identify a business goal, (2) Identify the customer web activity that demonstrates this business goal is being achieved; and (3) Determine attributes (measures) that quantify or characterize that activity, a lead balloon came in for a crash landing.
You see, many of these students had never really considered the fact that their projects represent a business, and that even though they are not selling anything, they indeed have customers.
The result of the evening, I realized, was not so much that the students created a comprehensive metrics plan for their web site, but that being forced to think about appropriate metrics drove them to ask and answer the most fundamental questions needed to be successful in their business.
And I, as the teacher, learned that metrics is not always about optimization; it is often about something much more basic. Metrics is often just about forcing clarity in how we think about our business.
( Nov 03 2005, 09:42:50 AM MST ) Permalink Comments [0]
| Archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Language | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Referrers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Today's Page Hits: 96 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||