Debunking the Top 8 Myths about Remote UX Research

As we approach our 8th year at UserTesting, we still run into common misconceptions people have about conducting their UX research online. I took the time this week to debunk some of the top remote research myths I frequently hear.

Myth #1: Online = unmoderated

When doing research online, instead of in the lab, a natural misconception is that it will be the exact opposite of in-person research.

However, just because you’re not in the same physical location as your participants, doesn’t mean you can't communicate with them directly. Even here at UserTesting, we will do moderated studies depending on the research objectives and interface we’re testing.

For example, remote moderated testing is ideal when you want to test the usability of an early prototype.

Myth #2: Online = quantitative

The benefit of running studies unmoderated, outside of the lab, is that you can standardize the instructions and remove facilitator bias. This can be ideal for conducting quantitative studies, such as for a usability benchmark.

It’s also much easier to reach 30+ participants remotely, than bringing them into the lab.

But using an online tool doesn't prevent you from obtaining qualitative insights. If the tool you’re using doesn’t record the session (and therefore the participant's voice), you can follow up with written questions to get some input on why they did what they did.

Myth #3: You need to see the participant’s face to understand what they’re thinking

In person, you can see the participants’ non-verbal cues while they use your product or talk about their experience. There’s a lot of value in that, but it’s not all lost without seeing their face. You can gather a lot about people’s emotions just from the tone of their voice.

That being said, putting a face to the participant can lend credibility to your study. You can always decide to run a moderated remote study through GoToMeeting (or a similar online meeting software) and ask the participant to turn on the webcam. Make sure you’ve asked them about this during recruitment so they know what to expect.

Myth #4: If you never meet the participants, you don’t know if they are your target audience

Screening participants for studies is an art of its own and something we spend a lot of time on here at UserTesting. We want to make sure we’re recruiting the right participants for studies so it’s the most natural situation possible.

But of course, once in a while, participants will answer a question wrong. Besides making sure your screener questions are well-balanced and not leading, we like to include a question at the beginning of the study that asks the participant to elaborate on their experience or skills so that we can make sure they meet the requirement.

For example, if you want to test the usability of your website that sells shoes with people who are online shoe shoppers, one of your screening requirements could be “Must have bought shoes online in the past 6 months.” In addition to that, the first question in the study could be: “Tell me about the last time you bought shoes online? What website did you use? What shoes did you get?” This will help you confirm that you’re actually testing with your target customer.

Myth #5: You can’t test prototypes

I actually love hearing this concern, because it’s so far from the truth! Remote, unmoderated usability testing is actually perfect for testing early in the design process. Because studies can be run and analyzed in the same day, it’s the easiest way to get quick insights and iterate on your design.

As long as you can host the prototype (even just an image) online, you can test it. I’ve even linked to images in Dropbox or Google Drive and directed participants there.

Alternatively, you can use tools like InVision that make it easy to create clickable prototypes and host them for you.

Myth #6: Participants are too distracted if they’re home or at work

Guess what? Your participants are distracted when they’re using your website or app on their own, whether or not they’re participating in a usability study. Mimicking the real world as much as possible will give you a much more accurate representation of what causes issues (and what doesn’t).

In general, participants try harder to complete tasks during usability testing, so it’s important to make it clear to them to stop when they would normally stop if they were doing this on their own time. This is much easier to convey when the participant is actually in their natural environment; i.e. at home or at work conducting a study remotely.

Myth #7: You can’t test sensitive information

An important distinction to make is if you’re protecting the participants’ sensitive information or your own.

If you have a new design of an app you don’t want anyone to get wind of, you can ask participants to sign an NDA. (If it's extremely sensitive, you’re probably better off testing it in person to make sure nobody takes a screenshot.)

Protecting participants’ information is equally important, but I’ve found users to be very willing to comply if you alleviate their fears. For example, you can promise to blur the screen in the recording or not record the screen at all for that portion of the test.

Myth #8: You can use the same research plan as for in-person studies

Unfortunately, the tasks you write for in-person studies will likely have to be adapted for remote studies, especially when unmoderated.

Because you can’t redirect participants if they misunderstand the instructions, you have to be very specific, without leading them to the right answer.

In unmoderated studies, you also can't stop participants if they get sidetracked or take too long to complete a task.

For example, in person, you might ask participants to “Buy a gift for your friend’s housewarming party this weekend.” You can always clarify or encourage the participant to keep working. In a remote study, it would be best to phrase that task more specifically, like, “Buy a gift for under $20 that has high customer ratings for your friend’s housewarming party this weekend.”

What are some other myths you've heard?

Have any misconceptions about remote user testing to add to the list? Let us know in the comments below.