16 usability testing methods you should know

Getting started on a new usability testing project—even if you’ve conducted hundreds of studies before—can be complicated.

Does the product design team need qualitative or quantitative feedback? What if you only have a wireframe or prototype? Is there a right and a wrong time to conduct a study? What if you need feedback quickly? Do you even have time to conduct a study? Who’s going to do it?

There are many different ways to conduct usability tests—all with pros, cons, and scenarios in which they work best. The key to effective research lies in planning. You're on the right track if you know what you can study, how, and when to do it. This section is for anyone looking for guidance on how to approach a usability test, when to run it, and how to interpret the results. 

Remote unmoderated usability studies

In a remote unmoderated usability test, users complete a series of pre-written tasks in their own location with their own device. Participants record their screens and speak their thoughts aloud as they complete the tasks, and the researcher reviews the recorded sessions at a later time. The researcher doesn’t communicate with the participant in real time. The researcher may include follow-up questions as part of the study, or they may follow up with the participant after viewing the session.

When to use it

Remote unmoderated usability studies can be used at any stage of the development process. They are great when you’d like to:

• Gather quick feedback: You can get results within hours since no scheduling is required.
   
• Test with larger sample sizes: Since a moderator doesn’t need to be present during each session, you can include as many participants as you like.
   
• Avoid bias: With no moderator present, participants may feel more inclined to be open and honest.

Limitations

• No guidance: Especially if you’re testing prototypes that aren’t fully functional yet, there’s a chance users might get confused or sidetracked.
   
• Fixed tasks and questions: Participants stick to the script the researcher wrote, so there’s no opportunity to dig deeper or change direction like in a moderated usability test.

Remote moderated usability studies

In a remote moderated usability test, a moderator meets virtually with participants to guide them through a pre-written test. The participant typically thinks out loud as they complete the tasks. The moderator observes the user as they work through the tasks, and the moderator may ask questions for clarification or to gather more data.

When to use it

Remote moderated usability tests can be used at any stage of development. They are great when you’d like to:

• Allow for unique, individualized follow-up questions in real-time.
   
• Ensure participants don’t get stuck while doing something technically demanding, like dealing with a complex or low-fidelity prototype.
   
• Ensure quiet participants are more vocal since the moderator can prompt them to share their thoughts.

Limitations

The drawbacks of remote moderated tests include the following:

• Less scalable: They require moderator time, which can limit the number of sessions you can reasonably conduct.
   
• Risk of bias: The presence and demeanor of even an experienced moderator can sway participants’ actions and responses.

Prototype testing

Prototype testing involves sharing prototypes or wireframes with participants to view on their desktop or mobile device. Testing your prototype with users lets product design teams iterate based on feedback before any code is written, reducing your risk of an unsuccessful launch. It can be done in person or remotely, with or without a moderator.

When to use it

You can start testing as soon as you have an idea or wireframe and then continue to test as you revise your prototype. Getting user feedback early and often will derisk your product launch and help you avoid doing costly rework.

Pro tip

Let your participants know that they’ll be interacting with a concept rather than a working product so they don’t become confused or frustrated if links don’t work and interactions aren’t fully fleshed out. Remind them to express what they expect will happen next.

Competitor studies

A competitor study provides insight into what your competitors are doing well and where your competitive opportunities are. In a competitor study, participants interact with one or more of your competitors’ products. You can also have participants complete the same tasks on two products, yours and a competitor’s, and compare the experiences. They can be done in person or remotely, with or without a moderator. 

When to use it

Run a competitor study any time you want to observe the UX of a competitor’s product, such as when they launch a new feature or make a big announcement. These studies can also be particularly helpful when developing a new product or feature. Testing during this phase will help you identify competitive opportunities and advantages before your product is developed.

You can also test the competitor experience at regular intervals to keep a finger on the pulse of the competition.

Pro tip

While comparing your design to several competitors is tempting, remember that these tests can exhaust participants. To avoid fatigue and confusion, ask each participant to use and evaluate no more than two experiences.

Check out our competitive comparison template here

Benchmark studies

Benchmark studies are a series of identical usability tests run at intervals to evaluate how your product’s UX changes over time. These studies use larger sample sizes than standard usability testing and always include a quantitative element. They also use the same core tasks and metrics each time to accurately compare one study to the next. 

Our proprietary QXscore is the best option for tracking UX performance in benchmarking studies. It’s a holistic, easily understood standard for measuring user experience that quantifies users’ attitudes and behaviors into a single score and identifies opportunities to improve. It can be used to measure user experience progress over time, relative to competitors, or across multiple lines of business, digital properties, and products.

When to use it

Benchmark studies are typically run regularly—monthly, quarterly, or yearly. They can also be helpful before and after rolling out a redesign or new feature.

Longitudinal studies

Longitudinal studies collect data from the same participants on multiple occasions over time. Studies can be run over a few weeks, months, or even years. A longitudinal study will provide valuable insight into how users behave and interact with their surroundings, an experience, or brands over time in their everyday lives. This study is an excellent tool for understanding how users interact with your product or service over time rather than just a moment in time. It can be done in person or remotely, with or without a moderator.

When to use it

Longitudinal studies can be done at any time. They can be especially helpful during the ideation phase before product development as a way to understand how users solve a problem without your product. You can also run them after development, which will give you insight into how, when, and why people naturally use your product or service.

Pro tip

Be sure to let your participants know that they’ll be conducting a series of tests, and be clear about the amount of time required to complete the study. Include a requirement in your instructions that notes participants will be expected to take a specific number of follow-up tests and that the test will last a specific amount of time. This can dramatically reduce the participant drop-off rate for this type of study.

Diary studies

A diary study is a type of longitudinal study where participants create “diary entries” as points of data. These diary entries can be audio, text, images, videos, or a combination of all and are typically combined with other studies to gauge participants’ expectations at the start and conclusion of a more extensive study. Diary studies can provide valuable insight into how users behave and interact with their surroundings or a product or service in their everyday lives. These studies are most effective when done remotely and unmoderated.

When to use it

Diary studies can be conducted at any time. They can be done before development, to gather requirements, or after a product has been launched to gather data on how, when, and why people naturally use it.

Card sorting

In a card sorting study, participants sort topics like pages, categories, or pieces of content into related groups. This can provide insight into how people group and label content and information in their minds. This is helpful when designing or assessing an information structure. 

There are two types of card sorts:

• Open card sorts: Users place items in groups and name the groups.
   
• Closed card sort: Users place items into predefined categories.

This type of study is often most effective when done remotely and unmoderated.

When to use it

An open card sort is typically done in the early stages of the design process or to evaluate an existing design, and the findings can be used to inform the new design or redesign. Closed card sorts can be done before a redesign to see if the existing or proposed structure makes sense to users or when adding new content or functionality to see if it fits within the existing structure.

Tree testing

Tree testing is a way for product design teams to evaluate the findability of topics within a website or app. Users are shown a text version of the site or app structure and are asked to indicate where they’d expect to find specific items or topics without the influence of the visual design of the navigation. Tree testing gives you insight into how easy or difficult it is for users to find essential items within a website or app. 

This type of study is also most effective when conducted remotely and unmoderated.

When to use it

A tree test is usually conducted before the design phase to evaluate a current structure that will be redesigned or during the early design phase to evaluate a proposed structure.

Preference testing

In preference testing, participants interact with and review multiple designs to determine which one they prefer and why. Participants can compare aesthetics, interactions, and content. If you have more than one design option, this type of test can provide insight into which one stands a better chance of resonating with users. It can be done in person or remotely, with or without a moderator.

When to use it

Preference testing should be conducted in the early design and prototyping phases so that you have plenty of time to iterate without having to spend much time in rework later.

A/B and multivariate testing

A/B testing presents each unique user with one of two versions of a design—the A version or the B version—and conversion rates for each version are compared to see which one performs best. A variation of A/B testing is multivariate testing, in which you test three or more design versions.

A/B and multivariate testing are most valuable when only one element differs in each version. For example, using a different call to action for each design approach while keeping everything else the same allows you to draw firm conclusions about why one design performed better.

When to use it

A/B and multivariate testing can only be conducted on a live site or product. Unlike most other methods in this guide, A/B testing is strictly quantitative. Therefore, combining A/B testing with other methods is recommended to understand why users prefer one version.

Multichannel studies

In a multichannel study, participants complete the same tasks on more than one channel or device. There are two approaches to a multichannel study:

• Groups of participants complete the same tasks on one channel (e.g., desktop, tablet, or mobile). For example, Group A will complete the study on desktop, Group B on tablet, and Group C on mobile. This option is fast, but users only provide insight on a single channel.
   
• Each participant completes the same tasks on all channels. This option takes longer to recruit for and run. You’ll need more time to allow users to set up and switch devices, but it enables participants to compare and contrast the experiences across channels.

This is different from an omnichannel study, where participants complete an activity that spans more than one channel or device, like researching a family vacation on one device and booking it on another. 

These studies can be completed remotely or in person, with or without a moderator.

When to use it

Multichannel studies can be conducted at any time, but they’re particularly valuable when conducted before any redesign effort

Omnichannel studies

In an omnichannel study, participants complete an activity that spans more than one channel or device. For example, if a customer finds a recipe on their desktop, adds the ingredients to a shopping list on their smartphone, refers to the list on their phone while at the grocery store, and follows the recipe at home on their tablet, they’ve completed an omnichannel experience.

An omnichannel study is valuable when you want to understand how a single user experiences a brand, company, and organization across touch points.

These studies can be completed remotely or in person, with or without a moderator.

When to use it

Omnichannel studies can be conducted anytime, but they’re particularly valuable before redesign efforts. By doing so, researchers can identify what’s working well and what needs to be improved in the existing omnichannel experience.

Focus groups

A focus group involves a moderator asking people about their behaviors, preferences, attitudes, and experiences on a topic or item of interest. Focus groups can be conducted in person or remotely, and they require a moderator. Focus groups allow participants to interact and converse with others while discussing a topic and gathering in-depth details about a group’s actions, thoughts, and feelings. The brainstorming leads to people building on each other’s ideas, which allows a group to dig into an idea or issue. 

When to use it

Focus groups can be conducted at any time during the life cycle of a product. However, they’re typically done before development to gather requirements.

Surveys

A survey is a series of questions answered by the target audience. Responses are open-ended or multiple choice, typically about preferences, attitudes, and experiences on a topic or item of interest. The sample size must be large enough to capture statistically significant results to claim a user group.

When to use it

A survey can be conducted at any time during the life cycle of a product.

1:1 user interviews

1:1 interviews are conducted by asking participants questions and taking note of their feedback while facilitating further discussion. Interviews can provide in-depth details about a user’s actions, thoughts, and feelings without presenting a product. Like a moderated study, 1:1 interviews are often a good idea when probing for more information is necessary to receive in-depth reasoning from the test participant.

This can be done either remotely or in person.

When to use it

User interviews can be conducted at any time during the life cycle of a product. However, they’re typically done before development to gather requirements or after the product is launched to gather feedback on how people use it.