For our first Usability 101 blog entry, I will provide a basic understanding of usability testing and address the following questions:

• What is usability testing?
• How important is sample size?
• What are the primary methods for usability testing?
• What are the methodology trade-offs?
• What are the benefits of usability testing?

What is usability testing?

Usability testing is the means for measuring the quality of a user’s experience while interacting with a product or system such as a website, software application, mobile technology, or any user-operated object. Usability testing is best when conducted throughout a product development cycle to capture direct user feedback on the ease of use and satisfaction with the product in order to ensure that it meets users’ needs and business objectives.

Usability testing on websites has evolved from traditional ‘Old School’ product usability testing, which was primarily focused on basic learnability, efficiency, and error prevention, to more advanced ‘New School’ user experience testing, which covers a much broader range of factors that evaluate the quality of the user experience and the impact on calls-to-action that drive business initiatives, as follows:

‘Old School’ Usability

Learnability
Can users quickly and easily learn how to use the site?

Efficiency
How well can users complete primary, routine tasks?

Error Prevention
How frequently do users encounter errors? How can the site design be improved to eliminate common and serious errors?

‘New School’ Usability

User Control
Does the site enable exploration by letting users initiate and control actions based on their primary objectives?

Communication & Relevance
Is the site’s messaging clear and is the value proposition aligned with target users’ objectives?

Aesthetic Integrity
Does the site incorporate graphic elements and visual cues to build a connection and guide users through the site?

Consistency
Does the page layout / organization help orient users to the site experience and set expectations?

Feedback
Are users clearly informed of where they are in the site as they complete tasks? Does the site clearly indicate task progression?

Simplicity & Scannability
Is content on the site presented in a manner that is intuitive and easy to digest based on users’ intent?

Help & Forgiveness
Does the site facilitate the completion of online forms and processes by providing help and forgiveness?

User Satisfaction
Are users satisfied with their experience? How does the online experience impact their future behavior and the future behavior of their friends, relatives, and colleagues?

It is important to consider all of the above factors when conducting website usability testing, as the customer experience is not only about how easy the site is to use, but also about how well the site delivers on customer expectations and drives call-to-action.

How important is sample size?

According to usability veteran, Jakob Nielsen, you only need to test with 5 users to discover 85% of the problems on your site. This is true when the focus of your research is more tactical – focused on learnabilty, efficiency, error prevention – and if you’re testing a fairly homogeneous sample of users.

However, if your study is more strategic – focused on calls-to-action, customer experience benchmarking, making population projections, or measuring the ROI from your online initiatives – it is important to test with a larger sample of users to for strategic decision making and statistically reliable metrics.

In addition, if you’re interested in learning how distinct segments interact with the site (e.g., patients vs. physicians vs. caregivers) or if you’re comparing the needs of different user groups based on demographic or psychographic information, it is necessary to have larger samples to enable segmentation.

Most qualitative (lab-based) research includes 6-8 sessions per day, per market, and can include multiple days of testing as required. Most quantitative (web-based) studies begin at a minimum of 200 participants per study and can range up to 5000+ depending on the breadth information collected, the number of segments, and the type of survey method that is used.

What are the methods for usability testing?

There are two primary methods for usability testing:
1) Lab-based testing
2) Web-based testing

Lab-based usability testing is conducted in person, with a moderator and one or more participants, and can be performed either in-house or in a designated research facility. Typically labs are equipped with audio and video recording capabilities, including picture-in-picture recording for usability testing, and most facilities include a two way mirror to allow clients to discretely view the live interaction between the moderator and the participant(s). Eye tracking monitors can be provided at facilities to enable eye tracking as part of the usability testing.

Web-based usability testing is conducted online, using a browser or proxy-based survey technology that captures the natural behavior of participants as they complete tasks and answer questions online. Participants are invited into a study through an online intercept or email invitation, and they can complete the evaluation in their natural environment, such as their home, office or university.

What are the methodology tradeoffs?

Lab-based usability testing is best when used to uncover low-hanging-fruit problems with user-interface design and to identify clear solutions for resolving those problems. Labs are also helpful when you want to understand the consumer’s emotions, or physical interaction with the site, such as mouse movement or facial expressions. Since usability labs are conducted with a smaller sample (usually 6-8 sessions per day), they are qualitative in nature so the results cannot be projected onto the larger population.

Web-based testing is quantitative in nature and enables larger samples (200+ participants) to complete an online user experience evaluation in a short-timeframe. It also enables more advanced analytics, and statistical significance testing which results in more reliable recommendations and projections. In addition, Web-based testing is completed in the users’ natural setting, from wherever they normally access the Web, so it eliminates any moderator bias or peer influence found in a multi-participant lab or focus group.

What are the benefits of usability testing?

The benefits of Lab-based testing include:
• Smaller sample size provides more qualitative feedback
• Moderated labs enable flexibility in questioning and allow for more in-depth probing
• Direct observation helps showcase participant reactions in person
• Allows for interaction between participants in a focus group or multi-participant lab
• Ability to probe deeper into specific topics or ideas
• Quickly identifies tactical, low-hanging fruit problems and solutions

The benefits of Web-based testing include:
• Larger sample size enables statistically reliable metrics
• Sample mirrors the targeted population (and key segments), including geographic reach
• Users provide honest feedback and behave as they normally would, in a natural setting
• Provides quantitative insights that guide projections and strategic recommendations
• Enables advanced analytics across qualitative, quantitative and behavioral data
• Creates a baseline for future testing and ROI measurement

This entry was posted on Monday, February 28th, 2011 at 5:28 pm and is filed under Usability 101, Website Usability Testing. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.