The 3 most common ways to user test your product

There are three different approaches you can take when you design a test plan for your next user testing study: in-house, moderated remote, or unmoderated remote. 

Each has distinct benefits and drawbacks, and there isn't one approach appropriate for every scenario. So how do you know which approach is right for you? Well, it’s going to depend upon your situation. In this article I’m going to lay out the benefits, downsides, and scenarios that would work for each type of study so you can decide which approach makes the most sense for you.

1. In-house testing

As one of the foundational user feedback methods, testing in-house means bringing participants into a physical location. In-house studies are usually moderated by a researcher who observes the participant, asks them to complete specific tasks, and asks probing questions to get a deeper understanding of what they're doing and why.

When to use in-house testing

• If you have privacy or security constraints: By testing in-house you can maintain control of sensitive information. When Apple was testing the Apple Watch, sending prototypes to remote test participants would have exposed them to the possibility of a leak.
   
• If you’re testing a hardware prototype: When you’re testing a physical prototype and only a few of them exist, it's probably best to bring participants into the lab. That way you can keep the prototypes in your possession and moderate your test participants.
   
• If you need a controlled environment: If you want to replicate a specific environment, in-house testing is a good choice. Michele Marut built the physical embodiment of a retail store in her research lab. It was set up with a cash register so her team could observe test participants in a realistic work environment.
   
• If you’re in a waterfall development cycle: If you don’t test until after a long process of research, design, and development, then in-house testing could be a good fit for your timeline. Although we don't recommend this because you can save a lot of money on development costs by remote testing earlier in your cycle!

Pros of in-house testing

• Immediate interaction and feedback: The immediate interaction available in in-house testing allows moderators to pick up on subtle cues like body language that can be difficult to detect in remote tests. This proximity means if a participant gets frustrated or puzzled, moderators can quickly intervene to ask follow-up questions, providing clarity and deepening the understanding of the user's experience on the spot.
   
• Depth of understanding: In-house testing allows moderators to dive deeper into understanding why participants make certain choices or react in specific ways. This setup enables moderators to observe and question behaviors and decisions in real-time, giving them a clearer and more comprehensive understanding of the participant's experience.
   
• Guidance and direction: Having a moderator present can be invaluable if a test participant becomes distracted or confused. Moderators can guide the session back on track without significant disruption, ensuring the testing remains productive and focused on its objectives.
   
• Control over the testing environment: Conducting tests in a controlled environment minimizes external variables that could influence the data. This control ensures that the feedback and results are specific to the user's interaction with the product, not their response to an unfamiliar setting.

Cons of in-house testing

• Higher costs and resource requirements: In-house testing often involves significant costs, including expenses for space, equipment, and skilled personnel. These factors make it a more expensive option compared to remote testing methods, which generally require fewer resources.
   
• Logistical challenges in recruiting participants: Recruiting participants for in-house testing can be a lengthy and challenging process, restricted by geographic location and the availability of participants willing to travel. This limitation can delay the testing process and may result in a participant pool that isn't fully representative of the target user base.
   
• Potential for observer bias: The Hawthorne Effect is a well-documented phenomenon where participants alter their behavior due to the awareness of being observed. This effect can lead to data that does not accurately reflect genuine user interactions, as participants may subconsciously try to perform tasks in a manner they think is expected.

2. Moderated remote testing

Moderated remote testing is a dynamic user feedback method where researchers and participants, though located differently, engage effectively through conference calls and screen sharing.

When to use moderated remote testing

• If you’re testing a low-fidelity prototype (like a wireframe): Participants might not understand what they’re looking at or they might encounter a feature that doesn’t work. You can direct them to the specific sections you’re interested in testing, where they may otherwise wander astray in an unmoderated environment.
   
• If you’re testing an obscure or complicated product: If what you’re testing may confuse your participants (like a complex database system), moderation can help keep them on the right track.
   
• If you need to do any other hand-holding: If you're asking test participants to do something they don't normally do (or think about), they might get confused. If that happens it can be valuable to have a moderator there to get them back on the right track.

Pros of moderated remote testing

• In-depth insights: Moderating allows you to probe as you see fit, delving as deep into the user experience as you desire. The human judgement involved in moderated testing creates unparalleled depth of feedback.
   
• Immediate clarification and guidance: If participants encounter difficulties or confusion during the test, moderators are on hand to provide immediate clarification or guidance. This ensures that the testing process is not only smoother but also that the data collected is relevant and not skewed by misunderstandings or errors in task execution.
   
• Flexibility to adapt and explore: Moderators have the flexibility to adapt the testing protocol in real-time based on the participant’s reactions and responses. This dynamic adaptability allows researchers to explore unexpected issues or delve deeper into interesting findings as they arise, making each session uniquely tailored to the participant’s interactions.

Cons of moderated remote testing

• Diminished personal connection: Despite the presence of video or voice communication, moderated remote testing cannot fully replicate the nuanced interactions of face-to-face sessions. Subtle non-verbal cues such as body language and eye contact are less detectable, which may affect the moderator's ability to fully interpret participant reactions.
   
• Environmental variables beyond control: Conducting tests in a participant’s personal environment introduces uncontrollable variables that might affect their behavior or the test outcomes. Background noise, interruptions, and internet connectivity issues are just a few factors that can impact the quality of data collected during remote sessions.
   
• Influence of the moderator’s presence: Although moderated sessions allow for immediate interaction, the presence of the moderator might inadvertently influence participant responses. This can be particularly pronounced in remote settings where participants might feel a need to perform or respond favorably to the moderator’s queries, potentially leading to biased outcomes.
   
• Natural behavior may be stifled: Knowing that they are being observed and directly interacted with can make some participants behave in ways they believe are expected of them rather than how they would naturally interact with the test material. This can skew results away from genuine user experiences and lead to less reliable data.

3. Unmoderated remote testing

With unmoderated remote testing, researchers use a software program to automate their study. Participants get a list of tasks to perform independently while their screen and voice are being recorded. They complete your test in their own environment without a moderator present. When they’re done, you will receive the results as a video.

When to use unmoderated remote testing

• Getting test results quickly and affordably: If you’re in an agile development cycle, consistently testing, and making iterative changes, unmoderated remote testing is a good option. It allows you to identify and fix many issues early in your development when it's less costly and time-consuming.
   
• Testing a high-fidelity prototype: This approach is best when your prototype is a good representation of your final product and you don’t need to hold your participant’s hand.
   
• Testing a live website or mobile app: This is a good option for testing e-commerce sites, gaming apps, landing pages, onboarding processes, social media profiles, checkout workflows, or else anything that doesn't involve a complicated process.
   
• Doing exploratory research: This works well if you want to observe participant behavior in a real-world setting.

Pros of unmoderated remote testing

• Cost-effectiveness: Unmoderated testing eliminates the need for a facilitator during each session, reducing costs significantly compared to moderated or in-house testing. This makes it an economical choice for gathering user data, especially when managing a tight budget.
   
• Quick turnaround on results: Unmoderated tests can be completed and analyzed much quicker without the need to schedule and conduct live sessions. This rapid feedback loop is invaluable when critical timelines allow for faster iteration and decision-making.
   
• Scalability of testing: Unmoderated testing allows for a broad-scale deployment, where hundreds or thousands of tests can run simultaneously. This scalability ensures a robust data set, providing high-quality results that can inform broader trends and insights.
   
• Flexibility for participants: Participants can complete tests at their convenience, leading to higher participation rates and less hassle over scheduling conflicts. This flexibility often results in a more relaxed testing environment, potentially leading to more authentic user interactions and responses.
   
• Reduced participant bias: Testing in a familiar environment can help reduce the pressure participants might feel in a lab setting, encouraging more honest and natural behavior. This can lead to insights that are more reflective of real-world use.

Cons of unmoderated remote testing

• Lack of real-time interaction: Without a moderator, no one can observe subtle cues or ask follow-up questions in real-time, limiting the depth of insights gathered. Important behavioral nuances may be missed, which could be critical to understanding user interactions.
   
• No immediate clarification for participants: If a participant misunderstands a task or encounters a problem, there’s no moderator available to provide clarification or guidance. This can lead to incomplete or inaccurate data, especially if participants stray off course without realizing it.
   
• Demanding script precision: Unmoderated tests require meticulously crafted scripts with clear, concise instructions. Every potential "why?" question a moderator might ask needs to be anticipated and embedded within the script. Mistakes or ambiguities in the script can lead to confusion and compromise the quality of the data collected.
   
• Participant fatigue and motivation: Keeping unmoderated tests brief (round 15 minutes maximum) is crucial to maintaining participant engagement and preventing fatigue. Unlike moderated sessions where the presence of another person might encourage participants to complete tasks, unmoderated tests lack this interactive element, which can affect the participant's motivation and investment in the task.
   
• Uncontrolled testing environments: The absence of a controlled environment can introduce variables that affect the test’s reliability. Background noise, interruptions, and other environmental factors can influence how participants interact with the test, potentially skewing results.

Conclusion

Each of these methods have advantages and disadvantages. None of them work for every situation, and all of them have a scenario in which they’re particularly useful. To determine which method is most appropriate for your next study you’ll have to take a few variables into consideration. For example:

• Are you testing hardware or software?
• Are you worried about sensitive information?
• Are you testing a prototype? If so, is it hi-fi or lo-fi?
• Do you need to control the environment?
• Are you testing a long, complicated process?
• How long do you want to wait for test results?
• What’s your budget for this research?

Whatever method you choose, we suggest that you test early and often so you don’t end up making costly mistakes that could have been easily fixed early in your development cycle.