In the past year we have seen an increasing demand for multi-device usability testing. Multi-device usability testing involves asking users to perform tasks on more than one device in a single session; devices may include desktop computers, tablets, and mobile phones. Testing multiple devices in a single session has been useful when testing Responsive Web Designs, as well as testing the fluidity of a brand’s experience from desktop websites to mobile and tablet websites, and/or mobile apps.

One benefit to conducting testing across devices in one session vs. testing designs on a single device per session is that the user is able to shed light on the overall experience and consistency across devices. How fluid is the experience? How easy is it for the user to transition from desktop to mobile and vice versa? Not only will testing multiple devices with one user provide insights into an overall experience, but it can also give direction and insight into how and why users may gravitate to one device for certain tasks vs. another.

What we have learned from conducting these tests is that preparation and flexibility play key roles in making sure this type of testing goes smoothly. Below are a few tips to help you prepare for and conduct multi-device usability testing:

Recruiting

• Recruit to reflect your real audience: If 90% of those using your mobile app are iPhone users, then testing should focus on iPhones vs. recruiting an even mix of iPhone and Android users.
• Forget the Gadget Lab: We have seen greater success in this type of testing (and mobile-only testing) by having the user bring in their own device. Not only is the user more confident navigating the device, but often there are added insights. With the users’ own device, you may get to see what apps the user has downloaded and how they have organized the information on their phone. Just be sure to clearly specify the types of mobile devices, model and version of software / operating system (e.g., FROYO, GINGERBREAD, etc.) required for testing during the screening process.
• Get the OK to Download: If users will need to download anything on their phone or tablet, it is best to ask this during the screening process to avoid any trouble down the road.

Discussion Guide

• Counterbalance Starting Device: Typically we will start an equal number of participants on each device (e.g., N=12 / 6 start on desktop, 6 start on mobile), unless the team is focused on the usability of one more than the other. We also make sure that we have an even mix of participant types (if research has segments) for each starting device. For example, if we have a total of 12 participants, with 6 Prospects and 6 Current Users, then 3 Prospects and 3 Current Users will start on desktop and 3 Prospects and 3 Current Users will start on mobile.
• Repeating Tasks is OK: It is okay to repeat tasks across devices; we have found that usability issues are still uncovered. Users will have expectations based on their initial experience, but this happens in real life, too! It is nice to add some variety to the tasks if possible, but do not stress if prototypes only allow for the same tasks to be performed on the various devices. By shuffling the starting device, the team will still be able to get initial impressions on the experience from a portion of the users. In fact, if you are looking for an apples-to-apples comparison across devices, it is important that you have the user perform the exact same task on each device.

Set Up

• Wi-Fi Ready: We typically ask users to join the Wi-Fi network on their phones or tablet prior to starting the session to save time. If you are conducting labs at a facility, it’s helpful to ask the front desk to share the Wi-Fi information to the participant upon arrival.
• Bring a Back-Up Device: Yes, we did say to forget the gadget lab; however, it does not hurt to have a back-up device for unpredictable technical difficulties. Typically, we just bring in the most popular mobile / tablet device used to access the site/app in the case that the user’s device fails to work during the test.
• Device Hot Spot: To ensure that the respondents’ actions on their mobile device are captured, it is best to designate an area for the camera to target. Typically, we tape off the area where the users should keep their device in order for it to be successfully captured on camera.
• Capture All Angles: Many of our clients view sessions remotely and it is a much more engaging experience if all angles of testing are captured, which includes the respondent’s face at all times and then the desktop screen, mobile phone, or tablet. In order to do this, we typically have multiple cameras in the room, which can be controlled by our video technician in the backroom. Our video technician is on site during the sessions to switch between cameras during testing to make sure all angles are captured at the appropriate times.

Conducting the Usability Session

• Keep it Real: One thing to avoid in user testing is forcing the participant to complete tasks on a device he or she is not familiar with or would not use in real life.
• Ask Wrap-Up Questions: One of the biggest benefits to multi-device usability testing is the ability to understand the overall experience across devices, so don’t forget to ask questions about this! Some questions you might ask are, “How does the desktop experience compare to the mobile experience?” or “Are there any tasks you prefer to do on one device vs. another?”

That sums up our tips for preparing and conducting multi-device usability testing. As each research project is unique, there are always ways to refine and adjust the methodology to ensure that the research objectives for your project will be met. If you have any questions, please do not hesitate to reach out to us at .