If you’re one of the many organizations moving toward a more project-based approach to operations perhaps even with a Project Management Office (PMO) then this post is for you!
It’s often difficult to fully integrate a web analytics methodology into projects and some companies even have to abandon trying.
It isn’t that anyone has a secret hate-on for measurement it’s because of Eric T. Peterson’s now famous catch phrase: “Web analytics is hard!”
Let’s assume that you have a “typical” project process that looks something like the one below:
Initiation
During this time period the PM group is engaged by a business unit looking to get some kind of initiative in the pipeline. Often at this time, a business unit may not know exactly what what the final project will look like, but the basic ideas are there for what they’d like to accomplish.
It’s at this point however when analytics can play a very large part! The designated analytics person on your team (assuming of course you have one) can play a consulting role here.
Depending on the site type (content-based, e-commerce, lead generation / marketing), any initiative should be tied towards improving the key performance indicators of the site. If, for example, we’re dealing with a content-based site and the strategy is Growth, Loyalty and Engagement then it’s the job of an analytics resource to do a quick check that the current initiative maps to what the data is showing.
If for the past few months, engagement metrics have been in serious decline and the initiative proposed is largely targeted to bringing in new visitors (growth), then perhaps there should be some discussion at this level of the project before serious time and money goes into it any further.
Plan
After the initiation phase, it’s assumed that everyone has signed up for the project (no turning back now!). Your analytics specialist (which may be the current reader) has given things a look and decided that the initiative makes sense in light of current trends exhibited by key performance indicators.
As per usual in this type or project process, Business Analysts normally get involved to start documenting requirements usually in the form of a Business Requirements Document (BRD). The BRD is largely meant as a communications tool where by all stakeholders on the project can clearly understand what’s required of the final outcome including a broad list of the feature set planned by the new initative. Some examples of what typical outcomes may be:
- Redesign tool/site with an easy-to-use and engaging interface
- Findability – Improve findability of pages from Web search engines
- Improve browsing experience of the site
- Increase revenue
None of the above goals are bad per say, but how will we know that we’ve had any success towards them when the project is done? For example, is there currently any way to measure that “Findability” has improved by 10% for example?
And what is the goal threshold anyway – when will the project team know to head out for champagne and celebrations? Is it a 5%, 10%, 15% increase in “Findability”?
Clearly what we’re lacking here are clearly defined project success metrics as well as goals for those metrics. Here are a few reasons for why success metrics and goals are a good idea:
- Defined metrics and goals that are signed off on by stakeholders mitigate the risk of those questions at the end of a project when people begin mentioning those pesky words, “Return on Investment”
- Defining goals for success metrics forces stakeholders to have realistic expectations for what the project will and won’t do from a results side, not a feature side
- Provide an easy way to communicate the success of your project to stakeholders
So let’s take two of the goals listed above and try to make them measurable!
Redesign tool/site with an easy-to-use and engaging interface
Notice that this goal already mentions two “fuzzy goals” being a tool / site that is “easy-to-use” and “engaging”. It’s the job of your web analytics resource to:
- Translate “easy-to-use” and “engaging” into metrics and
- Help clients set goals for these metrics
So let’s get to it.
“Easy-to-use” is a difficult goal to evaluate using the traditional web analytics metrics so instead, let’s employ the use of a qualitative survey here. Before making any changes to the tool, I would employ the use of any of the online survey providers and place a survey either throughout the entire site or tool being redesigned. Although surveys of this sort can be farily complex, let’s assume its a few simple questions (*Note: I am not a professional survey designer, I’m sure there’s tons of bias in the questions below!):
Q1: “How would you rate this website in terms of ease-of-use?” 0 – not easy to use, 1 – somewhat easy to use, 2 – easy to use, 3 – very easy to use
Q2: “Were you able to complete your task on this site / tool easily?” Yes / No
Q3: “Would you recommend this site / tool to others?” Yes / No
This simple qualitative survey now provides metrics that I can directly apply to understand the success of the project in terms of improving ease-of-use:
- Average point score from Q1
- Percentage of Yes responses Q2
- Percentage of Yes responses Q3
That covers “ease-of-use” now onto engagement. Although many people will define engagement very differently, I feel a good number of generic metrics to group together in order to get a general sense of engagement are:
- Average pageviews per visit
- Bounce rate
- Average time on site
Note: In addition, I would actually recommend adding in certain conversion events you may already have tracked in your tool. For example, signing up for an e-mail newsletter or sending a page to a friend are, in my opinion, also key examples of engagement metrics.
Another Note: If you’re only evaluating a tool it can be tough to grab something like “average pageviews per visit” for just the tool itself. What I recommend in this case, is work with your analytics vendor (or a consultant in the case of Google Analytics – I strongly recommend the guys at EpikOne) to create a segment based on all visits that used the tool in the course of their visit and then pull your regular engagement metrics for these visitors.
Segmentation in general is a key way to evaluate one area of the site in terms of its influence on other areas. Once a segment of those visitors is created, you can compare your key performance indicators of that segment versus the entire site and see if the area is worthwhile.
Setting goals for the above metrics is an exercise that requires involvement of your client group. A great idea at this point however is to think about the monetary implications (if any) of the goals you’re setting. For a site with paid advertisers, expectations around average pageviews per visit will seriously affect ad inventory levels and revenue for the site. Just something to keep in mind!
Findability
You’ve got to love that word don’t you? How do we measure “findability” in the context of: “Improve findability of pages from Web search engines”?
Most analytics tools out there should have the ability to segment your visitors (or visits) by traffic source and the fancier ones can do groupings like “organic search”.
If you’re in the lucky group that has a tool that will group your search results in this way, then the simple success measure is “Percentage Growth in Organic Search Visits / Visitors (depending on your tools capabilities)” defined as (Total Organic Search Visits After Project – Total Organic Search Visits Before Project) / Total Organic Search Visits Before Project.
The timeframes for “Before” and “After” project are arbitrary but they have to be the same length and I wouldn’t recommend anything less than a month in order to allow search engines to properly reindex all your content.
Hopefully those above examples gave you a bit of an idea of how to translate a fuzzy goal into a definable metric. There’s no exact science to it so I just recommend relying on experience to improve on it!
Design
In the design portion of a project a Functional Requirements Document (FRD) is produced and teams fully plan and develop how the solution will look with illustrations documents such as wireframes.
It’s important at this stage that the analytics resource works with technical personnel (either your vendor or internal resources or both) to identify any gaps between data you’re collecting now and the data you need to collect for your success metrics.
In some cases, you may be required to make changes to the site before the site launches in order to have a benchmark to compare metrics before and after launch. This is an important reason for web analytics personnel to be involved as early as possible in projects. It’d would be a pain to delay a project due to measurement restrictions.
I’d also recommend incorporating any technical instructions related to measurement in the technical document for the project. Creating separate documentation just adds one more thing for your developers to have to refer to when coding so why not make their lives a bit easier?
Develop
Development is a low key time period for an analytics resource. The main job here is to develop the test cases required to ensure that any technical changes recommended in the Design phase are working. Outside of that, there’s a little break here. 
Deploy and Test
Another hopefully low-effort time for an analytics resource. Ideally the solution proposed in the Design phase worked without any problems and passed quality assurance (QA) test cases created in the Development phase. If not however, revisions will need to be done to documentation and coding iteratively until the solution works and key metrics are able to be captured.
Close and Maintain
The close and maintain portion of the project represents the end of things for most resources, but not for an analytics person!
During maintain and close the analytics resource is finally able to have a look at the changes in key success metrics and perform the most valuable contribution to the entire project cycle.
Reporting on the success of the project is relatively easy as the key measures of success have already been determined, but where the true value comes in of having an analytics resource on the team is in the analysis and optimization.
During this phase, I strongly recommend the creation of a Key Insights Analysis Document. This document reports on the “Before and After” of all the key success metrics previously determined but also includes an analysis section of the results observed as well as suggestions for further optimization.
It is the Key Insights Analysis Document which should communicate if the project overall was a success or not and what lessons were learned. Did an assumption about a tool’s effect on a metric like Average Pageviews per Visit prove to be false? Is this information properly communicated to both the project team and the organization as a whole? A well written Key Insights Analysis Document should be able to spell these things out in order for anyone to be able to pick it up and quickly understand what worked and what didn’t.
In addition to the Key Insights Analysis Document, the analytics resource should produce a Web Analytics Operations Document. This document should act as a living guide for operations teams to be aware of any technical changes made to the site that need to be maintained for further updates.
Occasionally collecting specific data requires some pretty nifty hacks and once their on the site, there’s no guarantee that they’ll be maintained so that site owners can monitor these metrics after the project is over. The operations document should red flag what was performed, how it needs to be maintained and who to contact if anyone isn’t sure.
Wrapping up…
Incorporating web analytics into web projects isn’t always easy, but if you can’t prove the value of a very expensive and time consuming project, then what is the point in doing it in the first place?
In addition, being far more organized on projects like this allow you to quickly communicate successes and failures so that an organization as a whole doesn’t end up making the same mistakes twice.
First thing’s first though…you’ll need a dedicated or semi-dedicated web analytics resource so go out there and hire one!
