Methodology Spotlight: Lab-Based vs. Web-Based Testing

March 2011

One of the questions we are often asked is how to determine whether lab-based testing or web-based testing is the right methodology for a certain situation. While there are no hard and fast rules, we want to provide some basic guidelines to help you decide which research methodology may be best for you.

We'll start with the basics: Lab-based testing is qualitative in nature and there are multiple methods of research that can be conducted in a lab setting. Web-based testing is quantitative in nature and generally includes an online survey and/or a behavioral tracking methodology.

How do I decide?

• Lab-Based Testing:

  Lab-based testing uses qualitative research methods including, but not limited to usability labs focus groups, eye tracking, in-depth interviews, card sorting, and persona research.
• 
  

• Web-based Testing:

  Web-based testing uses quantitative research methods including, but not limited to website evaluations, competitive assessments, intercept surveys, prototype testing, home page surveys, brand & concept testing, open web research, and online benchmarking.

Let's explore each testing methodology and the benefits in more detail.

---

Lab-Based Testing

The most common type of in-person lab-based testing we conduct is traditional usability labs. These labs are interactive, in-person sessions conducted between a moderator and a respondent using a computer that is connected to the Internet. The goal of a usability lab is to watch a user interact with a website and learn how easy the site is to use, how appealing it is, and how helpful the content is to the end user.

Usability labs are best when used to uncover low-hanging-fruit problems with user-interface design and to identify clear solutions for resolving those problems. They offer flexibility in the types of questions asked and the subject matter tested, such as interactive videos, paper prototypes or wireframes if a website is not fully functional. These situations are ideal for a moderated session as the moderator can provide detailed instruction and ensure that users do not get hung up on issues related to the form of the design rather than its substance. Labs are also helpful for understanding the consumer's emotions, or physical interaction with the site, such as mouse movement, visible frustrations or facial expressions. Overall, usability labs are most effective when evaluating a site prior to a redesign or while performing iterative testing of designs throughout the development cycle.

Other forms of in-person testing include: eye tracking, which enables us to track eye movements when evaluating the usability of a web page, focus groups which are ideal for brainstorming and concept development, card sorting to help examine user thoughts on information architecture, organization and nomenclature, and in-depth interviews in which a website is not examined, but users are individually interviewed to understand their thoughts, feelings and behaviors on a certain subject, which is also used for persona research.

Benefits of Lab-Based Testing:

• Direct observation helps showcase participant reactions in person
• Smaller sample size provides more qualitative feedback
• Moderated labs enable flexibility in questioning
• Ability to probe deeper into specific topics, areas or ideas
• Allows for testing of incomplete web designs: paper prototypes, wireframes, etc.
• Quickly identifies UI problems and solutions

---

Web-Based Testing

As indicated earlier, an alternative and/or effective supplement to qualitative lab-based research is quantitative web-based (online) research. The key advantage of evaluating prototypes or websites online is to obtain a more quantitative assessment in order to better project the impact of proposed changes. If prototypes are still in early stages of design, then in-person testing is usually recommended to guide participants through a flow and set expectations about the limitations of the prototypes. If the prototypes are mostly functional, then online testing is often preferred in order to obtain more statistically reliable feedback (several hundred users).

Web-based testing is conducted using a browser-based survey technology that captures the natural behavior of participants as they complete tasks and answer questions online. Participants are invited into a study through an online intercept or email invitation, and they can complete the evaluation in their natural environment, such as their home, office or university. Web-based testing is quantitative in nature and enables larger samples (200 participants or more) to complete an evaluation in a short timeframe. It also enables more advanced analytics, and statistical significance testing which results in more reliable recommendations and projections.

Web-based testing can take a variety of forms and methodologies, from intercept studies where website visitors are intercepted and answer questions about their site experience, to website evaluations, which have users conduct key tasks on a site and provide feedback, to simple home page surveys that gauge users' response to the look and feel and branding of a home page. A more detailed description of web-based testing solutions follows:

• Website Evaluation:

  Typically Website evaluations measure how well the site is performing across key metrics such as success completing tasks, ease of use, satisfaction and likelihood to use or purchase. Website evaluations can be broad across the entire site, or more focused on a specific area or tool. Website evaluations can be conducted in a usability lab or online depending on the tactical and strategic goals of the evaluation. Traffic analysis is also helpful for providing insights into site performance, however, it does not explain why users behave the way they do.
• 
  
  

• Competitive Assessment:

  The goal of a competitive assessment is to understand how well your site is performing compared to the competition and identify brand preferences. Competitive assessments can be conducted in a usability lab or online depending on the tactical and strategic goals of the evaluation. Traffic analysis is helpful for understanding competitive site performance, however, it cannot explain the why behind competitive site behavior.
• 
  
  

• Intercept Survey:

  An intercept survey helps understand who is coming to your site, what their intent is for exploring the site, how well the site delivers on their expectations and how satisfied they are when they leave. Combining survey technology with clickstream analysis helps quickly identify your target users and where the site falls short of meeting their needs.
• 
  
  

• Prototype Testing:

  Prototype testing includes any Web site or set of Web pages that demonstrates the intended functionality of a Web site, but is not currently 'live' or fully accessible. Beginning with the early stages of development, prototype testing can help inform design elements, messaging, labeling and navigation. Once a functional prototype or beta site is ready, you can test key experience flows to ensure the site navigation is intuitive and the content meets customer expectations.
• 
  
  

• Brand & Concept Testing:

  Brand & concept testing evaluates how well a specific product or idea resonates with a given audience. The three most common applications for brand & concept testing are: product development, messaging, and advertising effectiveness.
• 
  
  
• Open Web Research: Open Web Research studies explore how users make purchase decisions and how they use the online channel to help inform their decision making. This type of study enables you to understand how consumers use the Web to find information that is related to you or your brand and helps assess the current and unmet needs of your target customers.
• 
  
  

• Online Benchmarking:

  Benchmarking is critical for tracking the ROI from online initiatives. Benchmarking can be competitive, longitudinal or across a family of company Websites. The benefits of online benchmarking include:

Benefits of Web-Based Testing:

• Larger sample size enables statistically reliable metrics and strategic insights
• Sample mirrors the targeted population, including geographic reach
• Users provide honest feedback and behave as they normally would, in a natural setting
• Provides quantitative insights that guide projections and strategic recommendations
• Enables advanced analytics across qualitative, quantitative and behavioral data
• Creates a baseline for future testing and ROI measurement

---

How Do I Choose Which Methodology is Right For Me?

As we have seen, both in-person qualitative lab-based testing and online quantitative web-based testing have unique benefits and work best in different situations. So the truth is, it is best to choose the methodology that best fits your specific research needs; however, the ideal research plan incorporates both methodologies, offering the benefits of both and ensuring that you have a complete picture of customer feedback on your site. Below is an example of an online research roadmap:

Looking at this roadmap, both qualitative and quantitative studies play a role. Starting with:

• Marketing Strategy and Concept Development:

  Lab-based studies such as focus groups can play a role in this stage of site development. Having users brainstorm ideas for their needs and wants can generate new concepts and provide the basis for new features or functionality for a website. Once analysis of these ideas is completed and a solid concept is in place, a quantitative study in the form of a concept test can provide statistically significant, geographically diverse feedback as to how this concept would play out with a larger audience to ensure that you are meeting the needs of all key constituents. Quantitative testing in the form of open web research can also be used at this initial stage to understand the market that you are playing in and to identify any opportunities you can capitalize on with changes to your site.
  
  

• Website Development:

  During this stage, qualitative usability labs can play an important role in ensuring that you do not move too far down a design path that does not work with your target audience. At the early stages of design, the flexibility of in-person moderation allows you to test designs and concepts on paper, as images, wireframes or designs with very limited functionality that would be difficult for a user to navigate without some instruction. This process can be iterative, guaranteeing that your final design has avoided major usability issues and is providing a good customer experience. Once a fully functioning prototype is available, online quantitative testing can be used to expose the beta version of the site to a wider audience, providing a final statistically relevant check before designs are finalized.
  
  

• Website Optimization:

  At this point in your website development, you are likely making few major adjustments to your site and are rather tweaking the site and tracking your customer needs to determine if any changes are necessary. During this phase, web-based testing is most useful and can take a variety of forms. Intercept surveys can capture a snapshot of your visitor profile and can determine how well users' needs are being met as they complete key tasks on your site. A website evaluation can tell you how well key tasks are performing on your site and identify any areas of the site that may need attention.
  
  

• Competitive Advantage:

  A competitive assessment can track the customer experience on your site against those of your competitors, allowing you to achieve or maintain an edge in your market.
  
  

• ROI:

  An online benchmarking program can track the progress of your site and provide a baseline for the performance of key tasks to determine how future changes to the site, as identified by research conducted in the website optimization or competitive advantage phases, impact customer experience and your bottom line.

Rather than being a single straight line process, findings from each of the stages of development feed back into the life cycle of the site. For example, a few years after a site redesign, you may have been through all of the phases of development and have moved into the ROI phase; however, based on that longitudinal tracking you can see a drop-off in conversion. Through your online benchmark, you may determine that your product information may no longer be meeting user needs. This finding can send you back to the beginning of the cycle to focus only on the features, functionality and content that would improve this piece of the user experience. Once you are confident in the changes you are making to your site based on qualitative and quantitative testing, you can return to your benchmarking program to determine if your changes had the desired impact.

This combination of lab-based testing and web-based research can give you the best of both worlds: exploiting the benefits of each methodology and overcoming any deficiencies through the combination of both. By using this variety of methodologies, you can have confidence in the ability of your site to deliver on its goals and to provide a highly satisfying customer experience, which will allow you to attract, convert and retain customers for a long time to come.

Contact us with questions!

Back to Top