For my final post on benchmarking, I want to address when and how often you should be conducting benchmarking studies.

While most people understand the importance of measurement and being able to demonstrate ROI in today’s economic environment, many people do not take the appropriate steps to ensure that they can demonstrate that ROI, so the first rule of when you should benchmark is: Benchmark your current performance, even when you know it needs to change. In order to save time and costs, many clients do not want to run a benchmarking study on a site they know needs serious revision, preferring to run the study only after a redesign or once changes have been made to correct the issues on the site. Understandably, most feel, “Why would I want to spend money on a study that will tell me that things are not working on my site? I already know that.” The problem with this idea is that the study you run after the changes are made will have less impact and will not be able to tell you the degree to which the changes made impacted the site experience. Additionally, even if you believe you know what is wrong with your site, the “before” benchmark study can confirm any hypotheses as well as potentially uncover issues you did not consider.

Once you have a baseline, what type of schedule should you consider for a benchmarking program? This question does not have a single answer that applies to all companies, however there are a few factors you need to consider:

How often does my site change? Anytime you make a major investment in the site it is important to measure the impact of the changes and make sure that you have moved the needle in the right direction. For some companies, this may mean establishing a quarterly benchmark, for others it might mean they would not test their site for a few years, if changes to the site were the only factor.

How often does my direct competition change their site? If you biggest competition completes a major redesign or implements key new functionality, you may want to understand how that impacts the experience users are having on your site and whether or not users’ expectations of your site are shifting based on the competition. However as I discussed in Part 1, your direct competitive set is not the only competition you face on the internet.

When did I last benchmark the site experience? If your site does not change frequently, nor do the sites of your direct competition, you still need to understand how changes to the internet overall have impacted your business. I told a story in Part 1 of a client who saw drops in key customer experience attributes year over year, despite not making any changes to the site – they just hadn’t responded to the evolving customer expectations based on what was happening online in general. To keep abreast of how the internet is impacting your site, an annual benchmark is key even if you have not made any changes, in order not only to measure, but to allow you to maintain the health of your site.

By establishing at least an annual benchmark, you will be able to gain insight into the impact of your changes, those of your competition, and those created as the internet evolves on the success metrics you have established for your site. By tracking these metrics over time, you will expand your understanding of the levers that influence the experience of users on your site and how to make better, more efficient changes to maintain a strong online presence. You will also be able to show the impact that site changes have had on your site performance and satisfaction.

Good luck with your benchmarking program!

Back in November, I first wrote about benchmarking, considering who you should be benchmarking your site against. Now I want to talk a little about what metrics you should be benchmarking over time and against the competition.

When thinking about the metrics you want to use to benchmark the experience on your site, there are a few key issues to keep in mind: how do you define success on your site? How can you measure the impact of changes to your site on the customer experience? How does the site experience influence other aspects of your business? There are some benchmarking methods which look at a single metric, such as NetPromoter which looks only at likelihood to recommend. While these numbers can be useful, we feel it is important to understand how your Website impacts a wider range of metrics in order to understand how changes to your site, your industry and the internet impact both the customer experience on your site and the ramifications that experience has for your business. To that end, there are a few areas we suggest you look at to come up with the metrics you will benchmark throughout time and against the competition:

1. Clearly define success metrics for your site. One of the questions we ask clients in each project kick-off meeting is how they define success for their Website. To us, it seems like a very basic question that will allow us to understand how we can help a client drive toward greater success. What seems like a simple question often draws a blank from our clients, so this is the first place that you should begin within any benchmarking project, defining success for your site. This can be very different depending on your industry or even the specific area of your Website you are focusing on. For a consumer products company, your Website may be successful if it increases brand awareness and positive feelings towards your brand, while for an e-commerce site, sales, conversion and a sales per cart may define success. So step 1 is to define these metrics and determine how you are going to measure them (questions to ask, scales to use, etc.)

2. Measure the customer experience on your Website. Typical customer experience metrics to track include success (the ability to complete the task the user came to the site to complete), overall satisfaction with site experience and likelihood to return to and recommend the site. The benefit of these customer experience metrics is that not only do they give you a picture of what is happening on your site, but you can also measure them on your competitors’ site(s) through a head-to-head evaluation. While you may not be able to determine how your competitors rank on the success metrics you have defined, customer experience metrics are measurable and can help you identify key areas in which the competition is outpacing you as well as areas in which you are excelling. These metrics also allow you to measure the impact of changes to your site quickly in ways that may not have bubbled up to the level of your success metrics yet.

3. Impact across the business. Finally, you want to make sure you are not only thinking about the Web experience, but how that experience may drive offline actions, such as visiting a brick and mortar outlet, or impact your brand, as could be seen in the increase in positive feelings toward your company. With these metrics it is likely you may measure them in brand tracking studies or other surveys that measure offline activities. The key here is to make sure you include them in your online benchmarking, asked in the same way, not only to measure the impact the Web is having on them, but also to be able to compare across the customer data you are collecting from a variety of sources.

Once you have defined these metrics, define the scale and the way the questions getting at this data will be asked and be consistent. By collecting this data in a consistent manner across your research, you will quickly build up a set of benchmarking criteria to help inform decisions as you move forward.

Stay tuned for the next post: Part 3 – how often should you be benchmarking?

Benchmarking Your Success

One of the most common questions clients ask when we present findings is “How do our ratings compare?” Benchmarks are key for providing context for the customer experience metrics we collect and can offer direction for the prioritization of recommendations and opportunities for innovation. However, determining the right benchmark to use to understand your online customer experience metrics is not as straightforward as it may seem. In a series of blogs about benchmarking, we will explore some of the questions surrounding benchmarks: Who should I benchmark myself against? What metrics should I be comparing? How do I ensure an “apples to apples” comparison? What method should I use for doing so? How often should I be looking at benchmarks?

For this first entry, I want to spend some time thinking about who is your true competition online and, therefore, what companies should you look at when you are thinking about benchmarks?

Part 1: Who are my competitors?

A few years ago, I executed a study that included year-over-year data on a number of travel booking web sites. In the second year, the top site from the first year remained top in the rankings of the competitive set, but showed a significant drop in its year-over-year satisfaction ratings. In exploring what caused the drop, we looked for changes to the site that created barriers to booking or degraded the user experience, but we could find nothing. In fact, the site was exactly the same year-over-year with no major changes made – and that, in and of itself, was enough to cause the site’s scores to fall. No action on the part of the site to stay current with what was happening around it on the internet overall caused the drop.

Looking at only the sites considered the direct competitive set wouldn’t give much insight as this site still topped them all, but the site was losing ground against the best experiences on the Web. So how do you determine who you should be benchmarking your site experience against online?

1. Your direct competition is of top importance. Understanding what is happening in your industry and how you rate against those competitors still provides the most useful context for understanding your metrics. For example, determining if features and functionality are becoming standard in your industry, identifying items missing from your site and those areas in which you excel. This can be done most easily through a head-to-head online evaluation.

2. Compare your site’s critical functionality to innovators of that functionality in other industries. Consider the functionality of your Website and look to the innovators in that specific functionality for a benchmark and best practices. For example, if you are incorporating video on your site, it is important to understand how it impacts your position among your direct competitive set. Users are not just looking at the video implementations on your competitors’ sites and forming an opinion based on those implementations, but they are also looking at YouTube.com, iTunes and Hulu.com, just to name a few, that have now become your competitors in this area. The usability and functionality of these sites are what users will be comparing you to, and therefore are what you need to be benchmarking your video implementation against. Just like in the offline world, a travel company may look to Dell as a leader in customer service, you need to determine who users will be comparing the different elements of your site to and look to those sites for benchmarks to measure yourself against and best practices to help improve your implementation. When considering new functionality, be sure you understand the benchmark set by these sites and use those benchmarks as goals for your design – measuring your design against them once it is complete.

3. Study averages across all industries. Because users’ opinions of the experience on your site are driven by all of their Web experiences, looking at customer experience metrics from a wide variety of sites in aggregate can provide important context. In the Benchmarks area of our site, we show benchmarks from hundreds of evaluations across industries and types of sites. This data is also the data we incorporate into our presentations most often because it takes into account the fact that users do not only judge a retail site by the experience they have on Target.com, Amazon.com and BestBuy.com, but also Google.com and Match.com. When it comes to usability and customer experience, users’ interactions on one Website form their expectations for another across industries, content-types and functionalities. Aggregated figures allow you to understand where your site fits among all the experiences a user may have online giving you an accurate idea for where your site is ranking.

By utilizing a combination of these three “competitive” sets for benchmarking depending on the situation and objectives of your research, you will be able to gain a complete picture of the customer experience on your site and continue to meet and exceed customer expectations by ensuring that you have a complete understanding of the playing field that is shaping those expectations.

Stay tuned for Part 2 – what metrics should you be benchmarking?