In the past year we have seen an increasing demand for multi-device usability testing. Multi-device usability testing involves asking users to perform tasks on more than one device in a single session; devices may include desktop computers, tablets, and mobile phones. Testing multiple devices in a single session has been useful when testing Responsive Web Designs, as well as testing the fluidity of a brand’s experience from desktop websites to mobile and tablet websites, and/or mobile apps.

One benefit to conducting testing across devices in one session vs. testing designs on a single device per session is that the user is able to shed light on the overall experience and consistency across devices. How fluid is the experience? How easy is it for the user to transition from desktop to mobile and vice versa? Not only will testing multiple devices with one user provide insights into an overall experience, but it can also give direction and insight into how and why users may gravitate to one device for certain tasks vs. another.

What we have learned from conducting these tests is that preparation and flexibility play key roles in making sure this type of testing goes smoothly. Below are a few tips to help you prepare for and conduct multi-device usability testing:

Recruiting

• Recruit to reflect your real audience: If 90% of those using your mobile app are iPhone users, then testing should focus on iPhones vs. recruiting an even mix of iPhone and Android users.
• Forget the Gadget Lab: We have seen greater success in this type of testing (and mobile-only testing) by having the user bring in their own device. Not only is the user more confident navigating the device, but often there are added insights. With the users’ own device, you may get to see what apps the user has downloaded and how they have organized the information on their phone. Just be sure to clearly specify the types of mobile devices, model and version of software / operating system (e.g., FROYO, GINGERBREAD, etc.) required for testing during the screening process.
• Get the OK to Download: If users will need to download anything on their phone or tablet, it is best to ask this during the screening process to avoid any trouble down the road.

Discussion Guide

• Counterbalance Starting Device: Typically we will start an equal number of participants on each device (e.g., N=12 / 6 start on desktop, 6 start on mobile), unless the team is focused on the usability of one more than the other. We also make sure that we have an even mix of participant types (if research has segments) for each starting device. For example, if we have a total of 12 participants, with 6 Prospects and 6 Current Users, then 3 Prospects and 3 Current Users will start on desktop and 3 Prospects and 3 Current Users will start on mobile.
• Repeating Tasks is OK: It is okay to repeat tasks across devices; we have found that usability issues are still uncovered. Users will have expectations based on their initial experience, but this happens in real life, too! It is nice to add some variety to the tasks if possible, but do not stress if prototypes only allow for the same tasks to be performed on the various devices. By shuffling the starting device, the team will still be able to get initial impressions on the experience from a portion of the users. In fact, if you are looking for an apples-to-apples comparison across devices, it is important that you have the user perform the exact same task on each device.

Set Up

• Wi-Fi Ready: We typically ask users to join the Wi-Fi network on their phones or tablet prior to starting the session to save time. If you are conducting labs at a facility, it’s helpful to ask the front desk to share the Wi-Fi information to the participant upon arrival.
• Bring a Back-Up Device: Yes, we did say to forget the gadget lab; however, it does not hurt to have a back-up device for unpredictable technical difficulties. Typically, we just bring in the most popular mobile / tablet device used to access the site/app in the case that the user’s device fails to work during the test.
• Device Hot Spot: To ensure that the respondents’ actions on their mobile device are captured, it is best to designate an area for the camera to target. Typically, we tape off the area where the users should keep their device in order for it to be successfully captured on camera.
• Capture All Angles: Many of our clients view sessions remotely and it is a much more engaging experience if all angles of testing are captured, which includes the respondent’s face at all times and then the desktop screen, mobile phone, or tablet. In order to do this, we typically have multiple cameras in the room, which can be controlled by our video technician in the backroom. Our video technician is on site during the sessions to switch between cameras during testing to make sure all angles are captured at the appropriate times.

Conducting the Usability Session

• Keep it Real: One thing to avoid in user testing is forcing the participant to complete tasks on a device he or she is not familiar with or would not use in real life.
• Ask Wrap-Up Questions: One of the biggest benefits to multi-device usability testing is the ability to understand the overall experience across devices, so don’t forget to ask questions about this! Some questions you might ask are, “How does the desktop experience compare to the mobile experience?” or “Are there any tasks you prefer to do on one device vs. another?”

That sums up our tips for preparing and conducting multi-device usability testing. As each research project is unique, there are always ways to refine and adjust the methodology to ensure that the research objectives for your project will be met. If you have any questions, please do not hesitate to reach out to us at sales@evocinsights.com.

In our best practices usability guideline series, this topic has come up before which emphasizes the importance of the guideline we are addressing today - consistency. While it is a standalone guideline, it is the key to making the other guidelines work as well – to provide the best possible user experience all of the usability best practices you put in place must be done consistently across the site. Consistency allows a casual user to learn your site more quickly and a frequent user to move effortlessly to the content or task of interest. While it is often the first guideline we look at when evaluating the site, it is the last guideline I’m looking at here because it impacts each of the other areas we have looked at in detail already. To understand how, let’s take a look at the elements of this guideline in detail:

Consistency

1. Clear indication of the home page

• This applies in two situations – first, if a user reaches your home page, they should know it
• When seeing the home page, users should never have to question where they are in the site, yet, too often in testing, we see users hit the “Home” link while on the home page to double check. This should never be necessary.
• The home page should be clear, offer a value proposition and highlight the main navigation to allow users to move to content of interest quickly and easily
• Secondly, there should be consistent access to the home page throughout the site. Conventional ways of doing this include making the logo in the upper left link to the home page or providing a “Home” link in the main navigation, bread crumbing and, possibly, at the bottom of the page as well
• The key here is to be consistent in the presentation of the link. If the logo in the left is clickable, every page should have a logo in the same place, linking to the home page.

2. Possesses consistency with terminology, design, colors, etc. and - if offline entity exists – maintains consistency of brand /messaging / image

• These two guidelines go hand-in-hand to ensure you are providing a single brand look and feel to your customer, so we’ll tackle them together.
• The key here is presenting a recognizable image to a user not only across all of your touch points, but within the touch points as well. So many sites today encompass different product lines, different customer targets and vast quantities of content. These are often produced by groups within an organization that do not work together. None of this matters to a customer, so it is important to view your site through their eyes.
• A customer should be able to recognize the site as yours no matter where they are in the site by the design of the page. Logos, colors and imagery should be in harmony with the brand and the same throughout the site.
• Also vital is the language used, not just in using the same terminology throughout the site (including spelling of terms, use of acronyms and capitalization), but also in the tone and level of the communication.
• All of these elements will come together to create an overall experience of your brand with your customer, and, if implemented well and consistently, can reinforce your brand image with the consumer.

3. Page layout / organization is consistent at similar levels of hierarchy

This goes back to the concept of teaching users how your site works and facilitating easy navigation. Users should immediately be able to recognize a main navigation landing page, a category page or a product page and know how the page will work and where to find the information of interest on the page once they have seen one example of it on your site.

4. Navigation and information architecture is scalable for new content

• Web content is rarely static. Once your template is created, you will need to add new content to the site, making room in your design for this new content. In order to do this seamlessly and with a minimum of disruption to the user experience, your initial design of the navigational structure of the site as well as the information architecture you have put in place must be scalable and have room for new content to easily be slotted in.

• • While you want to try to think of all of the possible content you will want to include on your site during the design process, it is impossible to think of all of the eventualities that might cause the need for your site to contain and direct users to new information.
  • If there is no room for expansion of the subcategories / submenus of your navigation and no space in the hierarchy and design of the information on your site for additions, even of another layer of content, you will be stuck creating an inconsistent experience in order to add the necessary information or be looking at a large scale design project every time new content is available for the site. Thinking about consistency before this becomes an issue will force you to create a design that can handle not only projected additions, but any of the unexpected necessities that come with doing business in a digital age.

While these are the key consistency guidelines, there are other consistency guidelines that are important to remember as well that go basics of editing. These include:

• Uniform style for capitalization, punctuation, and correct spelling
• Text links are underlined and follow conventions (discussed in detail in our Navigation and Feedback blog)
• Consistency of speed - Main content loads within 1 second on a high-speed connection

Thanks for reading our guidelines on Consistency. For more information on improving your site, please see the rest of our series on best practices according to usability guidelines on our blog. Also, stay tuned to our blog and newsletter for more information on web best practices.

“Customer experience” has become a pretty common term over the past decade. At eVOC our focus is on the online customer experience. According to Wikipedia, a customer experience (sans the online aspect) is “a customer journey which makes the customer feel happy, satisfied, justified, with a sense of being respected, served and cared, according to his/her expectation or standard, starting from first contact and through the whole relationship.”

In simplified terms, a customer experience is how customers perceive their interactions with your company.

When looking at the online customer experience definition, the experience is focused on your website - it is the intersection of your site visitors’ expectations and how well those expectations are being met. Nowadays, the online aspect melds into the mobile realm as well. Social media (e.g., Facebook, Twitter, YouTube) is also intertwined in your brand’s overall expectations and experience.

What’s interesting about considering the online dynamic of the customer experience is how much deeper impacting a negative experience can be. Do you remember the last time you decided to ditch your favorite clothing store because they were too slow at the cash register? However, you may have decided to skip visiting an e-commerce site because it was too slow. The threshold for the less personal online interaction is much lower when it comes to service, selection and efficiency. People do not mind being rude online, or impatient. Also, they can very quickly spread negative word about or post something that gets picked up and viewed by others. It can be a nasty, downward spiral. Usually more downward than upward. Most internet users have an invisible standard that they hold for how any site should perform. If it meets or exceeds the standard, then usually little happens. However, if it falls short, the consequences can be painful.

So, no pressure(!), but just remember how important the online customer experience is for creating, defining and nurturing the relationship with your customers.

For our first Usability 101 blog entry, I will provide a basic understanding of usability testing and address the following questions:

• What is usability testing?
• How important is sample size?
• What are the primary methods for usability testing?
• What are the methodology trade-offs?
• What are the benefits of usability testing?

What is usability testing?

Usability testing is the means for measuring the quality of a user’s experience while interacting with a product or system such as a website, software application, mobile technology, or any user-operated object. Usability testing is best when conducted throughout a product development cycle to capture direct user feedback on the ease of use and satisfaction with the product in order to ensure that it meets users’ needs and business objectives.

Usability testing on websites has evolved from traditional ‘Old School’ product usability testing, which was primarily focused on basic learnability, efficiency, and error prevention, to more advanced ‘New School’ user experience testing, which covers a much broader range of factors that evaluate the quality of the user experience and the impact on calls-to-action that drive business initiatives, as follows:

‘Old School’ Usability

Learnability
Can users quickly and easily learn how to use the site?

Efficiency
How well can users complete primary, routine tasks?

Error Prevention
How frequently do users encounter errors? How can the site design be improved to eliminate common and serious errors?

‘New School’ Usability

User Control
Does the site enable exploration by letting users initiate and control actions based on their primary objectives?

Communication & Relevance
Is the site’s messaging clear and is the value proposition aligned with target users’ objectives?

Aesthetic Integrity
Does the site incorporate graphic elements and visual cues to build a connection and guide users through the site?

Consistency
Does the page layout / organization help orient users to the site experience and set expectations?

Feedback
Are users clearly informed of where they are in the site as they complete tasks? Does the site clearly indicate task progression?

Simplicity & Scannability
Is content on the site presented in a manner that is intuitive and easy to digest based on users’ intent?

Help & Forgiveness
Does the site facilitate the completion of online forms and processes by providing help and forgiveness?

User Satisfaction
Are users satisfied with their experience? How does the online experience impact their future behavior and the future behavior of their friends, relatives, and colleagues?

It is important to consider all of the above factors when conducting website usability testing, as the customer experience is not only about how easy the site is to use, but also about how well the site delivers on customer expectations and drives call-to-action.

How important is sample size?

According to usability veteran, Jakob Nielsen, you only need to test with 5 users to discover 85% of the problems on your site. This is true when the focus of your research is more tactical – focused on learnabilty, efficiency, error prevention – and if you’re testing a fairly homogeneous sample of users.

However, if your study is more strategic – focused on calls-to-action, customer experience benchmarking, making population projections, or measuring the ROI from your online initiatives – it is important to test with a larger sample of users to for strategic decision making and statistically reliable metrics.

In addition, if you’re interested in learning how distinct segments interact with the site (e.g., patients vs. physicians vs. caregivers) or if you’re comparing the needs of different user groups based on demographic or psychographic information, it is necessary to have larger samples to enable segmentation.

Most qualitative (lab-based) research includes 6-8 sessions per day, per market, and can include multiple days of testing as required. Most quantitative (web-based) studies begin at a minimum of 200 participants per study and can range up to 5000+ depending on the breadth information collected, the number of segments, and the type of survey method that is used.

What are the methods for usability testing?

There are two primary methods for usability testing:
1) Lab-based testing
2) Web-based testing

Lab-based usability testing is conducted in person, with a moderator and one or more participants, and can be performed either in-house or in a designated research facility. Typically labs are equipped with audio and video recording capabilities, including picture-in-picture recording for usability testing, and most facilities include a two way mirror to allow clients to discretely view the live interaction between the moderator and the participant(s). Eye tracking monitors can be provided at facilities to enable eye tracking as part of the usability testing.

Web-based usability testing is conducted online, using a browser or proxy-based survey technology that captures the natural behavior of participants as they complete tasks and answer questions online. Participants are invited into a study through an online intercept or email invitation, and they can complete the evaluation in their natural environment, such as their home, office or university.

What are the methodology tradeoffs?

Lab-based usability testing is best when used to uncover low-hanging-fruit problems with user-interface design and to identify clear solutions for resolving those problems. Labs are also helpful when you want to understand the consumer’s emotions, or physical interaction with the site, such as mouse movement or facial expressions. Since usability labs are conducted with a smaller sample (usually 6-8 sessions per day), they are qualitative in nature so the results cannot be projected onto the larger population.

Web-based testing is quantitative in nature and enables larger samples (200+ participants) to complete an online user experience evaluation in a short-timeframe. It also enables more advanced analytics, and statistical significance testing which results in more reliable recommendations and projections. In addition, Web-based testing is completed in the users’ natural setting, from wherever they normally access the Web, so it eliminates any moderator bias or peer influence found in a multi-participant lab or focus group.

What are the benefits of usability testing?

The benefits of Lab-based testing include:
• Smaller sample size provides more qualitative feedback
• Moderated labs enable flexibility in questioning and allow for more in-depth probing
• Direct observation helps showcase participant reactions in person
• Allows for interaction between participants in a focus group or multi-participant lab
• Ability to probe deeper into specific topics or ideas
• Quickly identifies tactical, low-hanging fruit problems and solutions

The benefits of Web-based testing include:
• Larger sample size enables statistically reliable metrics
• Sample mirrors the targeted population (and key segments), including geographic reach
• Users provide honest feedback and behave as they normally would, in a natural setting
• Provides quantitative insights that guide projections and strategic recommendations
• Enables advanced analytics across qualitative, quantitative and behavioral data
• Creates a baseline for future testing and ROI measurement