Moderated Testing 101

What is moderated testing?

Remote moderated usability testing is an adventure. During a moderated test, you're live, “on the line” with your test participants, guiding them through the tasks, answering their questions, and replying to their feedback in real time. When you moderate a test, you’re right in the thick of things with the very people who use your product.

Moderated testing is incredibly adaptable, even allowing the test plan to be changed right in the middle of a session.

We’ll often see issues on sites that were never addressed in the original set of tasks. By channeling our inner Freud and probing deeper at these points, we’re able to learn about how people use a site in a way that was previously unknown to the developers. Without the element of human judgment to recognize and explore these issues, they would likely go uninvestigated until the next round of testing, which is often too late.

Usability testing is always a fantastic way to understand how your user base interacts with your product. Moderating goes one step further to truly connect you with these users and communicate with them as a peer to glean the “how” and “why” that metrics can often fall short of illuminating.

One of the biggest differences between an unmoderated and a moderated remote usability test is convenience. An unmoderated test can be taken at the participant’s leisure, without requiring the hassle of coordinating schedules. Unmoderated tests can also be run concurrently, allowing for a much greater volume. Because of this, the turn-around time for unmoderated tests is often significantly faster than that of moderated tests.

But it’s not all doom and gloom! If you have the time available for a moderated study, the payoff can be significant.

While an unmoderated study has a set test plan, a moderated study is as flexible as the moderator. Moderating allows you to probe as you see fit, delving as deep into the user experience as you desire. The human judgment involved in moderated testing creates an unparalleled depth of feedback.

What can moderated testing do?

Speaking one-on-one with a user about your website will unveil a lot more than just having the user follow a set of steps on their own.

In an unmoderated test, sometimes a user will be unable to complete one of the tasks or will complete a task in the wrong way. When this happens, the test can become derailed: the user might end up on the wrong page for the next task, or they will not be able to answer the questions following the confusing task.

If the test is moderated, however, the moderator can lead the tester back in the right direction and get the test on track again. While the moderator should never interfere with the tester’s instincts, having a moderator available can salvage a test that would otherwise have flopped.

Our research team ran one moderated study for an education company that helped teachers select textbooks for their classes. The third participant in the study struggled heavily to browse and select a textbook from the search results. She had some fascinating insights regarding what information would be most useful to her as she narrowed down her choices. The researcher was able to build off of that session and probe the following participants on some of the things that the third participant said.

Had the sessions not been moderated, we probably would have only scratched the surface of what was really important to users as they tried to accomplish their goals.

Being able to dig deeper’ allows for both test participant and moderator to find things they may not have sought out in an unmoderated test.

When should you use it?

While moderated testing can uncover important user insights that unmoderated testing might have missed, it’s not perfect for every situation.

Our research team has found that moderated tests are best used early on in the development process. In the prototyping phase, you can use moderated testing for exploratory research or where you might have little to "show" participants, but want to gather their opinions and reactions to ideas. By prompting participants to brainstorm their thoughts on your prototype, you can gain invaluable information that can save you from spending a lot of design and development time on a product that users don’t understand.

With prototypes, users may often run in to features that simply don’t work. In an unmoderated session, this often results in confusion, frustration, and biased task performance. A moderator can guide the tester right past these missing or broken features and refocus the session on the features of interest.

Moderating can also be useful if you need to test an obscure or very complicated product, such as an advanced database tool. While the average tester might not understand the applications of your product, you can walk them through the context and the controls. You can direct them to the specific sections you’re interested in testing, where they may otherwise wander astray in an unmoderated environment. There is a delicate balance here of keeping the participant on task while not being too overbearing and contaminating the user’s natural experience. When this balance is struck properly, even the most convoluted of tasks can provide rewarding feedback.

Lastly, products with strict security concerns are another example of where moderated testing brightly outshines unmoderated tests, as the moderator can maintain complete control over how the site or app is accessed.

But don’t think that you have to only use either unmoderated or moderated testing. You can run a moderated test to find the nitty-gritty issues of your prototype, then run an unmoderated test when checking to see how the post-prototype site is going. The methods are meant to complement, not compete.

Five misconceptions about moderated testing

1. Moderating is hard and requires specialized training.

With a few easy guidelines to follow, just about anybody can moderate their own usability test. At its core, moderated testing is simply a tool to put yourself in your users’ shoes. The experience will help enable better feedback extraction, but the learning curve is easier than checkers. Steve Krug’s Rocket Surgery Made Easy is a fantastic resource for anyone looking to moderate their own studies.

2. A moderator should assist the user.

A very common mistake for moderators to make is to hold participants’ hand through the tasks.  This contaminates any feedback you may get and defeats the purpose of observing users’ natural behavior. There is a very fine line between guiding the user and helping the user, so it often helps to remember that a struggling user will likely provide some of your most valuable feedback.

3. Moderating is expensive and time-consuming

Gone are the days when a moderated usability test required renting out expensive usability labs with one-way mirrors and all sorts of recording gadgetry. Remote testing allows you to connect with participants with an unprecedented level of ease. Setup is no harder than connecting to a teleconference. Depending on your test plan, the entire process can be carried out in the time it would take to draft an email to your boss explaining why your usability research is behind schedule. Additionally, the cost for remote participants tends to be much lower than their on-location counterparts.

4. Moderated testing requires a large sample size

Depending on your development strategy, moderated testing can be phenomenally effective when carried out often and in small doses. Observing three to five users at a given stage in development will often provide more than enough feedback to incorporate into the next iteration of the product. In high-volume studies, researchers often learn nothing new after the first handful of participants. For best results, stick to small batches of participants and high frequency of studies.

5. Moderated testing is always more useful than unmoderated testing

Ultimately, the best format of usability testing depends entirely on what your product is, what you’re looking to learn about it, and how much time you have available. As discussed earlier, there are a number of situations where moderated excels over unmoderated testing. However, there are just as many instances where the opposite is true. It is up to you to consider which format will best suit your needs in order to gather the most valuable feedback on the user experience of your product.

How can UserTesting help you with moderated testing?

    With UserTesting, you get access to a team of research experts who will help moderate sessions - either for you or with you! The research team can also help plan your tests, if you are an Enterprise client. You not only get someone to do the moderating, but you get someone with you from start to finish, really grasping what you want to get out of the research. Plus, the research team can help determine when moderated sessions are needed and when they are not.


    Insights that drive innovation

    Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.

    About this author:

    With UserTesting’s on-demand platform, you uncover ‘the why’ behind customer interactions. In just a few hours, you can capture the critical human insights you need to confidently deliver what your customers want and expect.