There are three different approaches you can take when you design a test plan for your next user testing study: in-house, moderated remote, or unmoderated remote. Each one has distinct benefits and drawbacks, and there isn't one approach that's appropriate for every scenario. So how do you know which approach is right for you? Well, it’s going to depend upon your situation. In this article I’m going to lay out the benefits, downsides, and scenarios that would work for each type of study so you can decide which approach makes the most sense for you.
1. In-House Testing
As the name implies, testing in-house means bringing participants into a physical location. In-house studies are usually moderated by a researcher who observes the participant, asks them to complete specific tasks, and asks probing questions to get a deeper understanding of what they're doing and why.
When to Use In-House Testing
- If you have privacy or security constraints --- By testing in-house you can maintain control of sensitive information. When Apple was testing the Apple Watch, sending prototypes to remote test participants would have exposed them to the possibility of a leak.
- If you’re testing a hardware prototype --- When you’re testing a physical prototype and only a few of them exist, it's probably best to bring participants into the lab. That way you can keep the prototypes in your possession and moderate your test participants.
- If you need a controlled environment --- If you want to replicate a specific environment, in-house testing is a good choice. Michele Marut built the physical embodiment of a retail store in her research lab. It was set up with a cash register so her team could observe test participants in a realistic work environment.
- If you’re in a waterfall development cycle --- If you don’t test until after a long process of research, design, and development, then in-house testing could be a good fit for your timeline. Although we don't recommend this because you can save a lot of money on development costs by remote testing earlier in your cycle!
Pros of In-House Testing
- The human element that comes with being in the same room as the test participant can be difficult to replicate in remote tests. In-house moderators can pick up on body language and other subtle cues, and address issues immediately.
- If the participant gets frustrated or doesn’t like something, moderators can immediately ask follow-up questions to find out what the tester didn’t like.
- Moderators can ask deeper questions when they want a deeper understanding of why a participant is doing what they're doing.
- Moderators can re-direct test participants if they get distracted or confused.
- You can control the environment.
Cons of In-House Testing
- In-house testing can be costly compared to remote testing. Costs include hiring trained researchers, lab space, and equipment.
- Recruiting test participants can take a lot of time and it's restricted by the location of the test lab. Plus, you can't guarantee that your participants will always show up.
- Being studied in a foreign environment can make participants change their behavior. It's a well documented bias called the Hawthorne Effect.
2. Moderated Remote Testing
In moderated remote testing, researchers are in a different location than their test participants, but both parties are on a conference call and sharing screens. Moderators can see what participants are doing, and they can both communicate with each other.
When to Use Moderated Remote Testing
- If you’re testing a low-fidelity prototype (like a wireframe) --- Participants might not understand what they’re looking at or they might encounter a feature that doesn’t work. You can direct them to the specific sections you’re interested in testing, where they may otherwise wander astray in an unmoderated environment.
- If you’re testing an obscure or complicated product --- If what you’re testing may confuse your participants (like a complex database system), moderation can help keep them on the right track.
- If you need to do any other hand-holding --- If you're asking test participants to do something they don't normally do (or think about), they might get confused. If that happens it can be valuable to have a moderator there to get them back on the right track.
Pros of Moderated Remote Testing
- Moderating allows you to probe as you see fit, delving as deep into the user experience as you desire. The human judgement involved in moderated testing creates unparalleled depth of feedback.
- Moderators can re-direct participants if they get confused, distracted or off-track.
- If your participant gets frustrated or doesn’t like something, moderators can immediately ask questions to find out what the tester didn’t like.
- Moderators can ask questions in real time when they observe something they’re curious about.
- It’s adaptable, even allowing the test plan to be changed right in the middle of a session.
Cons of Moderated Remote Testing
- Lacks the human element. Even though participants can hear your voice (and maybe even see your face) it’s not the same as being in the same room with the participant.
- You can’t control the environment.
- It's not ideal for exploratory research because a moderator interrupts how participants would naturally interact with your test.
- Participants often change their behavior when they’re being studied.
3. Unmoderated Remote Testing
With unmoderated remote testing, researchers use a software program to automate their study. Participants get a list of tasks to perform on their own while their screen and voice are being recorded. They complete your test in their own environment without a moderator present. And when they’re done, you receive the results in the form of a video.
When to Use Unmoderated Remote Testing
- When you need to get test results quickly and affordably --- If you’re in an agile development cycle and you're testing consistently and making iterative changes, unmoderated remote testing is a good option. You can identify and fix a lot of issues early in your development, when it's less costly and time-consuming.
- If you’re testing a high-fidelity prototype --- This approach is best when your prototype is a good representation of your final product and you don’t need to hold your participant’s hand.
- If you’re testing a live website or mobile app --- This is a good option for testing e-commerce sites, gaming apps, landing pages, on-boarding processes, social media profiles, checkout workflows, or else anything that doesn't involve a complicated process.
- If you’re doing exploratory research --- This works well if you want to observe participant behavior in a real-world setting.
Pros of Unmoderated Remote Testing
- You’ll save money because unmoderated testing is significantly less expensive than moderated or in-house testing.
- Turnaround time for test results is much faster than moderated tests.
- You can run a large volume of tests simultaneously and get high quality results back in a short period of time.
- Tests don't require the hassle of coordinating schedules.
- Since test participants are in their own environment, they’re more comfortable, honest, and less likely to be biased.
Cons of Unmoderated Remote Testing
- Since you’re not present with participants, unmoderated tests lack the human element.
- If participants get frustrated or don’t like something, you can’t interject and ask them a specific question. And if participants are unable to complete one of the tasks or start performing a task in the wrong way, you can’t get them back on track.
- You have to write unmoderated test scripts that are clear, concise and accurate. Every time you would ask “why?” in a moderated test, you have to mimic that in an unmoderated test script. And if a task is confusing, you can’t say “What I meant by that was…”
- You need to keep your tests short (about 15 minutes) to prevent your participants from getting fatigued. People are only motivated so much when they’re by themselves. Whereas when they're on a conference call with you (or they’re in a lab setting) they’re more likely to be invested in the test.
- You can’t control the environment.
Each of these methods have advantages and disadvantages. None of them work for every situation, and all of them have a scenario in which they’re particularly useful. To determine which method is most appropriate for your next study you’ll have to take a few variables into consideration. For example:
- Are you testing hardware or software?
- Are you worried about sensitive information?
- Are you testing a prototype? If so, is it hi-fi or lo-fi?
- Do you need to control the environment?
- Are you testing a long, complicated process?
- How long do you want to wait for test results?
- What’s your budget for this research?
Whatever method you choose, we suggest that you test early and often so you don’t end up making costly mistakes that could have been easily fixed early in your development cycle.
Insights that drive innovation
Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.