UX testing methods

What are the different types of usability testing methods? 

You know when to do usability testing, but what kind should you do? It’s one of the most crucial questions that will shape your usability testing project: whether to conduct moderated or unmoderated usability testing. Let's look at both to understand the advantages and disadvantages of each, and how to conduct usability testing using the right method for your needs.

Unmoderated vs. moderated research: What's the difference? 

At a high level, the difference between moderated and unmoderated is whether a researcher (the moderator) oversees the test or the test participant is left to carry out the task without any real-time supervision.  

Of course, people often ask if moderated usability testing is better than unmoderated testing and vice versa. The truth is that both forms of testing are valuable in different contexts.

Moderated usability testing

In a moderated usability test, a researcher will be on hand to facilitate the test in real time. If the test is conducted remotely, the researcher will observe the participant via their testing software tool in real-time. 

During the test, the moderator will ask the participants questions, guiding them through the session to understand their behavior and discover potential usability issues. 

Typically, you'll create a test script ahead of the session as a form of structure—but the perk of being in a live setting is that you can ask questions on the fly. For example, if a user interacts with your solution unexpectedly, you can find out why there and then. 

Another benefit of moderated testing is inviting stakeholders and teammates to observe the session anonymously. When stakeholders watch a test first-hand, they often start to grasp the benefits of user research and are more likely to champion further projects. 

When should I use moderated usability testing?

Moderated studies have beneficial use cases throughout the design and product lifecycle. As a general rule, moderated testing is the ideal option whenever you need to converse with users or gain in-depth feedback. 

In particular, we advocate moderated testing for: 

• Validating high-level concepts: Whether it's an early-stage prototype or just a one-page concept, you can use moderated studies to validate your ideas with people and gauge whether they're worth pursuing further. 
• Testing complex products: Some solutions are too difficult to explain through written instructions. If you feel like you want to be "in the room" with your user, moderated testing is your best bet. 
• Formulating and refining your research hypotheses: One of the most important phases of research involves developing a hypothesis about how you expect users to interact with your product — based on historical data and personas. Moderated research allows you to refine these hypotheses granularly. In the long run, this helps you create research projects that are clearly defined and rock solid.
• Creating UX advocates: Inviting stakeholders to observe moderated sessions is a surefire way to showcase the value of usability testing. You'll be amazed at the levels of empathy your stakeholders build.  
• Testing logged-in environments: Most websites and applications require users to create a dedicated log-in account, making it challenging to test logged-in settings. Moderated studies can help you to overcome this difficulty. For example, you can have a moderator log in and then pass control over to the tester. 

Pros of moderated testing

1. In-depth user insights: Moderated testing is ideal for conducting detailed user interviews and gathering comprehensive feedback on the customer journey. This method enables a deep dive into participants' thoughts, feelings, and reactions to your solution, offering insights that are difficult to capture through purely quantitative methods.

2. Capture of non-verbal cues: Utilizing video-led discussions within moderated sessions allows moderators to observe and interpret non-verbal cues such as body language and facial expressions. These observations can unlock additional insights into user emotions and reactions, enriching the data collected.

3. Guidance through complexity: In scenarios involving intricate tasks or complex interfaces, moderators can provide real-time support and guidance. This support ensures that all participants can navigate the tasks effectively, thereby reducing the risk of user error and enhancing the quality of feedback on usability and experience.

4. Dynamic questioning and adaptability: Moderators can adapt their questioning in real-time to probe deeper based on participants' responses or behaviors observed during the session. This flexibility allows for the exploration of unexpected insights and the clarification of ambiguous points, leading to a richer understanding of the user experience.

Cons of moderated testing

1. Resource-intensive process: Moderated testing sessions are typically lengthy and require significant time investment both in preparation and execution. The need for detailed planning, participant scheduling, and data analysis makes this method time-consuming.

2. Risk of bias: There is an inherent risk of introducing bias through the moderator's influence. The phrasing of questions or the moderator's reactions can inadvertently lead participants, potentially skewing the results. Careful training and script planning are required to minimize these effects.

3. Limited scalability: Due to the hands-on nature of moderated testing, it is usually conducted with smaller sample sizes compared to unmoderated or automated methods. This limitation can reduce the generalizability of the findings, as the insights may not confidently represent broader user behaviors or preferences.

4. Higher costs: The extensive planning, skilled personnel requirements, and longer session times contribute to higher costs associated with moderated testing. These factors make it a more expensive choice compared to other testing methods, potentially impacting the overall budget for user research.

Related reading: How to avoid leading questions in UX tests

Unmoderated usability testing

With unmoderated usability testing, no researcher can observe the test in a live setting. The only person in the session is the participant, who works through a pre-defined list of tasks and questions you've assigned. 

A great benefit of unmoderated testing is that the user can perform their tasks at any time of their choosing, freeing up their time to focus on other projects as your research project effectively runs in the background. 

Because you don't have the opportunity to interact with your user, you'll need to be crystal clear in your instructions so that the script, scenarios, and tasks feel logical without any further clarification.

To get your results from an unmoderated test, you'll lean heavily on your usability testing tools, which we discussed in the last chapter. 

If you're running a qualitative study, you'll use recording software to view the session later and analyze the user's thoughts and behaviors. If you're taking the quantitative approach, your tool should automatically collect and analyze the data to generate numerically based insights. 

When should I use unmoderated usability testing?

If you want quick insights from a wide range of participants, unmoderated usability testing is the best method. Generally, we advise this form of testing in cases where you need specific answers, want feedback on your solution ASAP, or want to see a participant interacting with your product in their natural environment. 

Unmoderated is ideal for: 

• Meeting Agile demands: Agile is the name of the game today. Engineers and product managers often wait for your UX insights, so you need to deliver them stat. In this fast-paced environment, unmoderated studies facilitate the need for speed. 
• Scaling your research: Statistical significance is gained through larger sample sizes. Because unmoderated testing leaves users to their own devices (literally), you can better scale your project, potentially including hundreds of people.
• Using quantitative data to cement qualitative insights: Qualitative and quantitative research don't have to be mutually exclusive—they can work great together. For example, you could run a qualitative study and discover that a handful of users aren't satisfied with a design feature. You can then use a large-scale quantitative test to prove or disprove this insight. 
• Democratizing UX research: The demand for UX research is higher than ever. You only have 24 hours daily, so you and your team can't run every research project requested. People from other departments and roles can engage in research with unmoderated research. With the right platform and some training, you can empower your colleagues to run their own unmoderated tests at speed and scale. 
• Finding your tribe: Unmoderated research unlocks the potential for you to gain feedback from participants anywhere in the world. Whether you need a geographically dispersed sample size or need to reach a specific group of people with your product, unmoderated research has your back. 

Pros of unmoderated testing

1. Cost-effectiveness and scalability: Unmoderated testing offers a cost-effective solution for gathering user feedback, particularly when conducted at scale. Without the need for a moderator, these tests can be executed more efficiently and at a lower cost per participant, making them accessible to organizations with limited budgets or those seeking to test large sample sizes.

2. Improved statistical significance: With the ability to reach participants from diverse geographical locations, unmoderated testing allows for larger and more representative sample sizes. This increase in sample size enhances the statistical significance of the findings, providing greater confidence in the insights derived from the research.

3. Global participant access: Unmoderated testing platforms enable researchers to access participants from around the world, breaking down geographical barriers and expanding the pool of potential test participants. This global reach enhances the diversity of perspectives and experiences captured in the research, contributing to more comprehensive insights.

4. Empowerment of non-UX professionals: Unmoderated testing tools empower individuals outside of the UX field to conduct their own research projects. With user-friendly interfaces and intuitive workflows, these platforms enable non-UX professionals to design and execute user tests independently, democratizing the research process and fostering a culture of user-centricity across diverse teams.

Cons of unmoderated testing

1. Multiple rounds of testing may be necessary: Due to the absence of real-time moderation and interaction, unmoderated testing may require multiple rounds of testing to uncover the desired insights fully. Iterative testing iterations may be needed to refine the research objectives, test hypotheses, and validate findings, prolonging the overall research timeline.

2. Limited understanding of user motivations: While unmoderated testing provides valuable quantitative data on user behaviors and interactions, it lacks the depth of qualitative insights obtained through moderated sessions. Without direct interaction with participants, researchers may struggle to understand the underlying motivations, emotions, and context driving user actions, necessitating supplementary qualitative research methods for a holistic understanding.

3. Risk of misinterpretation due to unclear instructions: Clear and precise test instructions are crucial in unmoderated testing to ensure participants understand the tasks and objectives accurately. Complex or ambiguous instructions can lead to participant confusion or errors, undermining the validity and reliability of the research findings. Careful script development and pilot testing are essential to mitigate this risk and optimize the quality of data collected.

As you can see, each strategy has its use cases—so it's important to figure out your priorities and consider blending multiple methods before you begin your research project.

Related reading: Usability testing examples