For my final post on benchmarking, I want to address when and how often you should be conducting benchmarking studies.

While most people understand the importance of measurement and being able to demonstrate ROI in today’s economic environment, many people do not take the appropriate steps to ensure that they can demonstrate that ROI, so the first rule of when you should benchmark is: Benchmark your current performance, even when you know it needs to change. In order to save time and costs, many clients do not want to run a benchmarking study on a site they know needs serious revision, preferring to run the study only after a redesign or once changes have been made to correct the issues on the site. Understandably, most feel, “Why would I want to spend money on a study that will tell me that things are not working on my site? I already know that.” The problem with this idea is that the study you run after the changes are made will have less impact and will not be able to tell you the degree to which the changes made impacted the site experience. Additionally, even if you believe you know what is wrong with your site, the “before” benchmark study can confirm any hypotheses as well as potentially uncover issues you did not consider.

Once you have a baseline, what type of schedule should you consider for a benchmarking program? This question does not have a single answer that applies to all companies, however there are a few factors you need to consider:

How often does my site change? Anytime you make a major investment in the site it is important to measure the impact of the changes and make sure that you have moved the needle in the right direction. For some companies, this may mean establishing a quarterly benchmark, for others it might mean they would not test their site for a few years, if changes to the site were the only factor.

How often does my direct competition change their site? If you biggest competition completes a major redesign or implements key new functionality, you may want to understand how that impacts the experience users are having on your site and whether or not users’ expectations of your site are shifting based on the competition. However as I discussed in Part 1, your direct competitive set is not the only competition you face on the internet.

When did I last benchmark the site experience? If your site does not change frequently, nor do the sites of your direct competition, you still need to understand how changes to the internet overall have impacted your business. I told a story in Part 1 of a client who saw drops in key customer experience attributes year over year, despite not making any changes to the site – they just hadn’t responded to the evolving customer expectations based on what was happening online in general. To keep abreast of how the internet is impacting your site, an annual benchmark is key even if you have not made any changes, in order not only to measure, but to allow you to maintain the health of your site.

By establishing at least an annual benchmark, you will be able to gain insight into the impact of your changes, those of your competition, and those created as the internet evolves on the success metrics you have established for your site. By tracking these metrics over time, you will expand your understanding of the levers that influence the experience of users on your site and how to make better, more efficient changes to maintain a strong online presence. You will also be able to show the impact that site changes have had on your site performance and satisfaction.

Good luck with your benchmarking program!

Last week, Skittles launched a radical new marketing campaign designed to take advantage of social media and harness user-generated content in a way not done before by a consumer brand.  The company replaced its Website, almost exclusively, with user-generated content.  Initially, Twitter became the site’s home page, but the company had to shift strategy after some “tweets” became profane and inappropriate.  The company has been changing the site it uses as its home page, first Facebook, then the Wikipedia entry dedicated to Skittles and, now, YouTube.  Skittles provides the information architecture for the site, but the majority of the links head to user-generated content.

This strategy took to an extreme the type of idea Aaron addressed in his blog post on the necessity of a home page, by asking, “Does a consumer products company need to generate its own content?”  While the final analysis has yet to be done on this campaign and its success, it is an interesting and in my opinion, a worthwhile experiment.

The site has certainly gotten people all over the Web talking about Skittles!  As CPG companies contemplate how to make their Websites relevant, I think the Skittles experiment can serve as a reminder to be thinking about the Web in a different way – to create brand ambassadors by incorporating user-generated content and collecting feedback in any form in which they choose to offer it.  This method, unfortunately, can backfire if the supposed brand ambassadors are not actually proponents. That said, one wonders if negative buzz is ever really negative. Even if I say that I don’t like Skittles, someone else may fervently disagree and go out and buy 20 packages!

I think Skittles has provided an interesting glimpse into the next generation of Websites, but has also demonstrated that there is still a lot of work to be done to determine how best to capture and employ the on-going online conversation about a brand in a meaningful and powerful way.

We want to know - what do you think about the campaign?  Tell us in your comments below!

Back in November, I first wrote about benchmarking, considering who you should be benchmarking your site against. Now I want to talk a little about what metrics you should be benchmarking over time and against the competition.

When thinking about the metrics you want to use to benchmark the experience on your site, there are a few key issues to keep in mind: how do you define success on your site? How can you measure the impact of changes to your site on the customer experience? How does the site experience influence other aspects of your business? There are some benchmarking methods which look at a single metric, such as NetPromoter which looks only at likelihood to recommend. While these numbers can be useful, we feel it is important to understand how your Website impacts a wider range of metrics in order to understand how changes to your site, your industry and the internet impact both the customer experience on your site and the ramifications that experience has for your business. To that end, there are a few areas we suggest you look at to come up with the metrics you will benchmark throughout time and against the competition:

1. Clearly define success metrics for your site. One of the questions we ask clients in each project kick-off meeting is how they define success for their Website. To us, it seems like a very basic question that will allow us to understand how we can help a client drive toward greater success. What seems like a simple question often draws a blank from our clients, so this is the first place that you should begin within any benchmarking project, defining success for your site. This can be very different depending on your industry or even the specific area of your Website you are focusing on. For a consumer products company, your Website may be successful if it increases brand awareness and positive feelings towards your brand, while for an e-commerce site, sales, conversion and a sales per cart may define success. So step 1 is to define these metrics and determine how you are going to measure them (questions to ask, scales to use, etc.)

2. Measure the customer experience on your Website. Typical customer experience metrics to track include success (the ability to complete the task the user came to the site to complete), overall satisfaction with site experience and likelihood to return to and recommend the site. The benefit of these customer experience metrics is that not only do they give you a picture of what is happening on your site, but you can also measure them on your competitors’ site(s) through a head-to-head evaluation. While you may not be able to determine how your competitors rank on the success metrics you have defined, customer experience metrics are measurable and can help you identify key areas in which the competition is outpacing you as well as areas in which you are excelling. These metrics also allow you to measure the impact of changes to your site quickly in ways that may not have bubbled up to the level of your success metrics yet.

3. Impact across the business. Finally, you want to make sure you are not only thinking about the Web experience, but how that experience may drive offline actions, such as visiting a brick and mortar outlet, or impact your brand, as could be seen in the increase in positive feelings toward your company. With these metrics it is likely you may measure them in brand tracking studies or other surveys that measure offline activities. The key here is to make sure you include them in your online benchmarking, asked in the same way, not only to measure the impact the Web is having on them, but also to be able to compare across the customer data you are collecting from a variety of sources.

Once you have defined these metrics, define the scale and the way the questions getting at this data will be asked and be consistent. By collecting this data in a consistent manner across your research, you will quickly build up a set of benchmarking criteria to help inform decisions as you move forward.

Stay tuned for the next post: Part 3 – how often should you be benchmarking?

Usability Labs have long been considered a speedy, effective method of uncovering fundamental usability barriers on Websites. They are especially useful before launching a site or conducting an in-depth quantitative study. Here is a quick refresher on how Usability Labs work:

Target users of a Website are invited to a research facility to participate in a usability study A moderator (from eVOC) conducts one-on-one interviews with the participants, asking them to complete tasks on the site(s) and answer questions about their experience Sessions are audio and video recorded, while clients observe the sessions from a separate room through a one-way mirror
	Usability Lab

What Sets Usability Labs Apart
Usability Labs are so effective because:

1. All variables are accounted for
• Internet connection speed, monitor resolution, and physical environment are the same for everyone
• The moderator can control exactly which pages users explore
2. Users’ facial expressions, physical reactions, and site behavior can all be observed
3. The discussion guide is flexible
• The moderator can probe on key areas on the fly
• More specific, user-driven recommendations can be collected
4. Usability interviews take 30-90 minutes, which exposes participants to more content than a survey would

It’s not surprising that many of our clients choose Usability; in fact, it’s unbeatable for evaluating a site’s information architecture, nomenclature, and overall navigability.

But what about effectiveness of navigation, ease of use, and information flow? Sure, Usability findings are great at illuminating these areas. But an advanced technology called Eye Tracking is now enabling us to uncover insights that, when combined with Usability, paint the most complete picture of a Website’s performance. We have found that Eye Tracking allows us to make discoveries about Websites that we never could have made doing Usability alone.

How Eye Tracking Works
So how does it work? Using a special Eye Tracking monitor that tracks participants’ eye movements, we can log exactly where users are looking as they browse each page of a Website. The data can then be visually quantified:

www.evocinsights.com

What Eye Tracking Tells Us
The Eye Tracking software also allows us to capture usage data, such as how long it takes users to click or complete tasks, and what percentage of users take a particular path. Here are just a few examples of the insights we can glean when we complement Usability with Eye Tracking. We can:

• Determine what areas users overlook (e.g., advertising) and complement with usability findings to explain why
• Calculate how long it takes users to complete tasks in different navigation designs and choose which one performs best
• Discover if users tend to notice certain page areas early or late, and provide recommendations on prioritizing navigation and content
• Study what paths users take as they browse a site and determine their effectiveness, as well as key differences between user segments
• Learn how users scan information and find out the most effective content layouts

We strongly encourage any company that is planning to conduct Usability Labs to consider adding Eye Tracking, particularly to test visibility of key areas, navigation, messaging, and/or ads. To learn more, go to http://www.evocinsights.com/services_labbased.html#eyetracking.

In the first week of 2009, eVOC held its annual kickoff meeting, during which we excitedly discussed our plans for the year. It was great to have everyone together and we are looking forward to a wonderful 2009!

Here is a photo from the event:

With the holiday season fast upon us, people everywhere are scrambling to make some final holiday travel plans. We thought this would be a good opportunity to take a look at a great example of how one aspect of usability can impact user experience, and even provide a competitive advantage.

Booking travel online has become the primary way to plan travel—it’s fast, convenient, and almost always provides the best price on airfare, hotel, and other costs. But there are a lot of sites competing for attention. Why use one site over another? They can all find flights, so what makes one more useful than another to a potential customer? Offering users superior control and flexibility over how they search and display travel information is a great way to stand out.

We took a look at Kayak.com as an example of a best practice in user control and feedback in a relatively complex search experience. At every step of the process, Kayak’s layout and functionality clearly anticipates users’ needs in a rather elegant way.

Let’s say we want to fly from the San Francisco bay area to New York. Both metro areas have a few major airports, so Kayak offers the ability to include nearby airports in the search. That will allow us to compare a flight fares in the same results and let us decide which airport we should arrive or leave from.

Before we even hit the friendly orange search button, Kayak is already telling us over on the right what the best fares are on all the days of the current month, offering a quick link to see the fares for that day.

Once we run the search, Kayak offers some really smart features that let you pinpoint exactly the flight you want. What if we need to arrive by 8pm, but we can’t leave until 9am? Most travel sites simply let you specify a Morning, Afternoon, or Evening set of options. With Kayak, users can use sliders to specific time ranges for landing and takeoff, for both departing and arriving. On top of that, we don’t need to run the search again—it automatically updates.

So now we’ve got a short list of reasonable flight options. If we want to get really picky, we can choose results by all sorts of common travel variables. Price range, specific airlines, layover duration, aircraft type, no red-eye, are all options that users have control over. Would you rather see that price in Euros? Kayak has you covered there too. By the time you have made all your choices you have a list of results customized just for you.

All of the controls are easily accessible on the left hand side, and all the results update automatically. The user is never required to go to another page to find what they want because they are offered complete control and feedback in a single place.

By designing a system that deeply understands what users want to know and allows them to refine on the fly based on real-time feedback, Kayak delivers a unique experience that users will remember and return to every time they need to perform the task of finding not just the best fare, but the best fare that fits their specific needs.

Benchmarking Your Success

One of the most common questions clients ask when we present findings is “How do our ratings compare?” Benchmarks are key for providing context for the customer experience metrics we collect and can offer direction for the prioritization of recommendations and opportunities for innovation. However, determining the right benchmark to use to understand your online customer experience metrics is not as straightforward as it may seem. In a series of blogs about benchmarking, we will explore some of the questions surrounding benchmarks: Who should I benchmark myself against? What metrics should I be comparing? How do I ensure an “apples to apples” comparison? What method should I use for doing so? How often should I be looking at benchmarks?

For this first entry, I want to spend some time thinking about who is your true competition online and, therefore, what companies should you look at when you are thinking about benchmarks?

Part 1: Who are my competitors?

A few years ago, I executed a study that included year-over-year data on a number of travel booking web sites. In the second year, the top site from the first year remained top in the rankings of the competitive set, but showed a significant drop in its year-over-year satisfaction ratings. In exploring what caused the drop, we looked for changes to the site that created barriers to booking or degraded the user experience, but we could find nothing. In fact, the site was exactly the same year-over-year with no major changes made – and that, in and of itself, was enough to cause the site’s scores to fall. No action on the part of the site to stay current with what was happening around it on the internet overall caused the drop.

Looking at only the sites considered the direct competitive set wouldn’t give much insight as this site still topped them all, but the site was losing ground against the best experiences on the Web. So how do you determine who you should be benchmarking your site experience against online?

1. Your direct competition is of top importance. Understanding what is happening in your industry and how you rate against those competitors still provides the most useful context for understanding your metrics. For example, determining if features and functionality are becoming standard in your industry, identifying items missing from your site and those areas in which you excel. This can be done most easily through a head-to-head online evaluation.

2. Compare your site’s critical functionality to innovators of that functionality in other industries. Consider the functionality of your Website and look to the innovators in that specific functionality for a benchmark and best practices. For example, if you are incorporating video on your site, it is important to understand how it impacts your position among your direct competitive set. Users are not just looking at the video implementations on your competitors’ sites and forming an opinion based on those implementations, but they are also looking at YouTube.com, iTunes and Hulu.com, just to name a few, that have now become your competitors in this area. The usability and functionality of these sites are what users will be comparing you to, and therefore are what you need to be benchmarking your video implementation against. Just like in the offline world, a travel company may look to Dell as a leader in customer service, you need to determine who users will be comparing the different elements of your site to and look to those sites for benchmarks to measure yourself against and best practices to help improve your implementation. When considering new functionality, be sure you understand the benchmark set by these sites and use those benchmarks as goals for your design – measuring your design against them once it is complete.

3. Study averages across all industries. Because users’ opinions of the experience on your site are driven by all of their Web experiences, looking at customer experience metrics from a wide variety of sites in aggregate can provide important context. In the Benchmarks area of our site, we show benchmarks from hundreds of evaluations across industries and types of sites. This data is also the data we incorporate into our presentations most often because it takes into account the fact that users do not only judge a retail site by the experience they have on Target.com, Amazon.com and BestBuy.com, but also Google.com and Match.com. When it comes to usability and customer experience, users’ interactions on one Website form their expectations for another across industries, content-types and functionalities. Aggregated figures allow you to understand where your site fits among all the experiences a user may have online giving you an accurate idea for where your site is ranking.

By utilizing a combination of these three “competitive” sets for benchmarking depending on the situation and objectives of your research, you will be able to gain a complete picture of the customer experience on your site and continue to meet and exceed customer expectations by ensuring that you have a complete understanding of the playing field that is shaping those expectations.

Stay tuned for Part 2 – what metrics should you be benchmarking?

Preface from Claudette Levine, Managing Partner:

We are thrilled to announce the addition of Cheryl Jordan-Aguilera to our team as Vice President of Business Development (for a brief bio, visit: http://evocinsights.com/aboutus_mngmnt.html). We feel fortunate to continue our growth during questionable economic times and are thankful for the relationships we have with our customers. The customer experience research space is so expansive that we feel we are influencing just the tip of the iceberg. With Cheryl’s expertise and vision, we hope to continue evangelizing the importance of a good customer experience and spreading our passion for research. I asked Cheryl to provide her outlook as a way to introduce herself to our readers.

Cheryl Jordan-Aguilera VP of Business Development

Cheryl says:

I’m so excited to be joining eVOC and working with Liz and Claudette again!  I remember when I was introduced to the idea of helping companies understand how to make Websites more usable and accessible to the average customer back in 2000 at Vividence (acquired by Keynote Systems in 2004).   In those early days it was exciting to be part of a trailblazing effort to build awareness in the market place. Claudette, Liz and I were part of that early effort and it is amazing to be here almost 10 years later with the need for user experience research more than ever!   I am excited to be at eVOC and back in the customer experience research space after spending some time in competitive intelligence at Hitwise and 3 years managing market research projects at MarketTools, Inc for clients like Microsoft and HP.

I think we all agree that the Web has revolutionized the way business is done.  Not only has a shift occurred for consumers but also for the way business is conducted across industries.  For many years, companies thought that Websites only mattered if there was tangible buying and selling, but through research we’ve learned the importance that Websites can play in branding, building community and connecting people.  One must only look at this election cycle to see how the Web has democratized the election process, making the candidates much more accessible to the average citizen.  Both presidential candidates have benefited from the internet for fundraising as well as getting support from the grassroots up through groups on Facebook and other social networking sites.

For many of you reading this blog, you already know the importance of the Web and user experience research.  However, there are still many companies and people unfamiliar with it or not sure about it.  My job at eVOC will be to educate marketers, researchers, product managers, and Web strategists alike.  I will talk with them about the need for user experience research and demonstrate why it is important to truly understand your customers in terms of who they are, what they know,  what they do and what they expect from your Website.

I look forward to meeting all of you, learning about your Web initiatives and how eVOC can continue to help your business and online presence grow with the Web!

When it comes to the Internet, my dear mom is not exactly a whiz. I don’t have enough fingers to count the number of phone calls I’ve received from her with Internet-related questions:

My Mom

• “Phil, I have an article that I want you to read. How can I send it to you?”
• “Phil, you know that little box where I type the Website addresses? It disappeared and I can’t get it back.”
• “Philip, my email isn’t accepting my password. I’ve typed it in at least 5 times and now I’m locked out!” (She always calls me by my full name when she’s frustrated. Turned out she had her CAPS lock on.)

So the other day when she left me a voice mail asking for help, I was ready for just about anything. I got home from work, sat down at my computer, and gave her a call. She was having trouble buying some shoes on a well-known e-commerce site.

Mom: “I went to buy them and I typed in all the shipping information. Then it asked me for my credit card. But I wasn’t sure if I had ordered the right shoes, so I clicked ‘Back’ to make sure. But then it forgot all my information!”
Me: “OK.”
Mom: “So I had to type everything again, except this time I had a typo. I saw the mistake on the last page, and I didn’t see any place where I could fix it. I’d been on the site for an hour and you know how frustrated I get, so I just gave up. Can you help me buy them on another site?”
Me: “Whoa boy.”

My unassuming mother had uncovered fundamental problems with the site’s Checkout process. And while she may sit at the lower end of the Internet savvy spectrum, she represents a significant percentage of the site’s user base. How many other shoppers, like my mom, get frustrated by poor usability and simply leave the site? How many dollars in lost revenue are those usability problems causing?

Amazon.com’s Shopping Cart

After helping my mom buy the shoes (it was thrilling, let me tell you), I decided to look into how other popular e-commerce sites handle Checkout. I was truly surprised by what I found: While there are a handful of sites that do it well (Amazon and Apple to name a few), most Websites possess key usability flaws that serve as barriers to purchase.

Using my own research findings, as well as best practices we have accumulated here at eVOC, I now present a brief guide that summarizes the key attributes of an intuitive Checkout process:

Pre-Checkout

1. Shopping Cart

• This page provides key information upfront. It offers a summary of what users are buying, which includes the name, total price, quantity, and the ability to modify items or remove them from the cart. It allows users to calculate shipping costs and timing based on their location, or provides a link to view estimated shipping info. The page also includes a prominent button or link to return to the store for those users who are not ready to purchase

Checkout

2. Enter Shipping Address

• In this step, users enter their shipping and billing address. Required fields should be clearly marked with asterisks. Guidance should be provided next to any fields which may cause confusion (e.g., Apt #, ‘Must match credit card’, etc.)

3. Select Shipping Method

• The key here is to tell users what each option is, when it will arrive, and how much it will cost

4. Review Order

• This page includes a summary of users’ information, with each step represented visually in its own section (e.g., Product Summary, Total Cost, Shipping / Billing Address, Shipping Method, etc.). A prominent ‘Edit’ link will allow users to modify their details if necessary, before entering their credit card information

5. Enter Payment Information

• The most sensitive area for users in Checkout is the entering of their credit card information. This should happen last, directly underneath the summary of their order so that they can be sure their selections were correctly recorded by the site. Provide clear instructions about the credit card’s security code and let users click ‘Submit’ to officially send in their order

Post-Checkout

6. Confirmation Screen

• This page includes a summary of the order, provides a confirmation number, and informs users that they will receive a confirmation email immediately

While each of these steps does not need to fall on its own page, Checkouts are most effective when they follow the order described above.

Some global tips to consider:

• Include a breadcrumb at the top that helps users identify where they are in the process
• Differentiate the appearance of buttons/links that take users to the next screen
• Allow users to move back and forward in the process without losing the information they have entered
• Use clear messaging that clarifies if users’ cards will be charged before they advance to the next step. If credit card entry is the last step, this is only necessary on the last page

If possible to employ, this additional functionality also helps users:

• Ability to store users’ credit card and address information on the site and make it available for selection or modification on the appropriate screen when users need it
• Contextual tips on the side of each page about shipping, delivery, gift wrapping, etc.

By following these guidelines, any company that does e-commerce can feel more confident that shoppers will see the Checkout process through to the end, and will be satisfied when they leave – even my mom!