User Testing Mobile Apps and Websites: You've Got Questions, We've Got Answers

Mobile user testing plays a critical role in creating winning experiences. We’ve compiled a series of answers to the top questions we hear time and time again when helping our clients run their UserTesting mobile studies.

Expert Q+A: Mobile Website User Testing

(Jump to the Mobile App Q+A)

Q. We have a successful computer-based website. How much should we worry about mobile?

A. It depends on your customer base. If you’re sure that none of them ever use mobile devices, you probably don’t have anything to worry about. But a majority of Americans now own smartphones, 40% have tablets, and about half of Americans age 18-29 say they primarily access the Internet on their smartphones. Ask yourself, if you’re not active on mobile, are you leaving yourself vulnerable to competitors who are?

Most companies should at a minimum test their websites on mobile to make sure they work properly and meet user expectations. And you should seriously consider either designing a mobile site from the bottom up, or modifying your current one for smartphones and tablets. That involves rethinking not just how the site works, but what tasks users will want to do on mobile.

Q. Which mobile platforms should we test our website on?

A. The leading mobile web platforms in the U.S. are iPhone, Android phones, and iPad. So you should definitely test on at least those three platforms. We’ve also found a lot of incompatibilities on Android tablets, so we recommend testing on them as well. Amazon’s Kindle Fire is the most popular Android tablet in the US.

Q. When should we test on mobile?

A. You should do user tests throughout the development process, so you can fix problems before they get too deeply embedded in your site. You can start testing as soon as you have anything to show to users, even if it’s just conceptual sketches.

It’s best to run frequent tests of a few testers each than to save up your tests and do them all at once at the end of development

Q. What sorts of tests should we do on mobile?

A. When people hear “user testing,” they tend to assume that means only usability testing. That is, of course, one of the things you should do with user testing: Have users go through the main functions of your site, make sure they’re intuitive, and identify questions or hesitations users might have (this is especially important if you have a purchasing process on your site).

But it’s also important to test for emotional engagement. In other words, how do people feel about your site? Can they quickly accomplish what they want to do? Do they feel rewarded by using it? Mobile users are notoriously impatient and easily distracted. Even if your site is easy to use, people may not stick around unless they feel engaged.

You should also plan different tests for smartphones versus tablets. Smartphones are used most often for quick access to info while people are on the go. For example, in shopping sites, people frequently use smartphones to do product and pricing research, even though they may not be as likely to make the final purchase during their visit. You should test to make sure it’s easy for people to find product information, pictures, and price information on your site.

In contrast, tablets are much more likely to be used for long browsing sessions. Many e-commerce companies are finding that people are much more likely to make purchases when using a tablet, and some report that their most lucrative customers are likely to visit them through a tablet. So you should make sure that the tablet shopping experience is rich, engaging, and easy to use.

Q. I understand the importance of designing specifically for mobile, but I have trouble convincing the people I work with. How do I educate them about the specific needs of mobile?

A. If you’re struggling with a specific design issue, user tests can be a terrific way to end the argument quickly. You’ve heard the old saying that “a picture is worth a thousand words”? In our experience, a user video is worth a thousand hours of debate. If your team is having an argument about a feature, you can use UserTesting to get quick video of some real users reacting to the proposed solutions. We find that those videos can be far more persuasive than a roomful of opinions.

Q. Should we focus on a mobile app or a mobile website?

A. No single answer is right for every company.

We’re finding that many commerce companies choose to do both. The website is aimed at casual visitors, while the app is aimed at their most loyal customers (the people who are most likely to go to the trouble of downloading an app). So you use the website for prospecting and the app for deepening the relationship.

On the other hand, many major brands are using mobile apps as a marketing tool to help spread awareness and affinity. For example, Oreos distributed a popular game in order to freshen its brand image among young people. In this case, the target of the app is not current users, but new ones.

The most important thing is to understand what your mobile strategy is. How does your presence on smartphones and tablets fit with all of the other ways you engage with customers, and what are you hoping to accomplish in mobile? We’re long past the days when you could create something on mobile and expect users to respond to it just because it’s trendy.

Expert Q+A: Mobile App User Testing

Q. We already have some beta testers who use our app. Do we also need to do user testing?

A. Any testing is good, but as beta testers get to know your app, it gets harder for them to spot usability problems because they no longer have fresh eyes. Also, friends and family (the usual source of beta testers) are not necessarily a good proxy for typical users because they are too emotionally invested in your product or aren’t representative of your customer base.

UserTesting gives you feedback from typical users who don’t already know your app and have no emotional investment in it. User tests help you understand the needs and reactions of normal users, the people who will be writing your reviews in the app store.

Q. Should we test before or after app release?

A. We strongly recommend testing apps prior to release, during development. It’s much easier to fix a problem in the early stages of development than to do it after the product is finished (not to mention after you’ve received a slug of bad reviews in the app store).

Q. Testing an iOS app before release is very difficult because of Apple’s security. How do you handle that?

A. We have special processes to simplify the testing of iOS apps. We take care of the UDID management, and we don’t deplete your allocation of UDIDs. Just fill out the test form online, give us a link to your app, and we’ll take care of the rest. (Note: due to the extra logistics involved, iOS unreleased tests take an average of three business days to complete after we receive your .ipa file.)

Q. When in the development process should we do user tests?

A. Whenever you have something you can show to users, it’s a good idea to get feedback on it right away. You can even test prototypes and preliminary wireframes (anything you can display in a browser, you can show in a user test). Generally, the sooner you identify problems, the easier it is to fix them. We’ve seen tragic examples of companies that tested at the end of development, identified problems, but released anyway because it was too late in the process to make changes.

Q. What sorts of tests should we do on mobile apps?

A. When people hear “user testing,” they tend to assume that means only usability testing. That is, of course, one of the things you should do with user testing: Have users go through the main functions of your app, make sure they’re intuitive, and identify questions or hesitations users might have.

Many apps include icons and controls that the developer custom-created for the app. Although artistic creativity is great, one of the most common causes of confusion in user tests is buttons and controls that users can’t easily understand. It’s essential to test these features.

But there are also three other important tasks for user testing. The first is to look for emotional engagement. In other words, how do people feel about your app? Can they quickly accomplish what they want to do? Do they feel rewarded by using it? Mobile users are notoriously impatient and easily distracted. If your app doesn’t make people feel engaged, they may move on to something else, and never come back.

The second task is to get a deeper understanding of how your customers think. The better you understand them, the better you can make decisions on their behalf. User tests are like mini-focus groups, but you can organize them on a day’s notice and with far less expense and hassle.

The third use is settling internal arguments. Because mobile devices are highly personal, you may find that people on your team can be extremely passionate about a dispute over a feature or UI element. Rather than having a knock-down argument on the subject, it’s often easier and faster to let the actual users give you a ruling. User tests make it easy to bring in that voice of the customer.

Q. Should we test on both tablets and smartphones?

A. If your app is designed to run on both tablets and smartphones, you should test on both of them. A screen layout that looks good on a smartphone can look uninviting on a tablet, and vice-versa. And user behaviors are subtly different on tablets and smartphones.

Q. We’re testing an Android app. Which brands of smartphone should we test on?

A. Samsung devices account for about 50% of Android web traffic in the U.S., so that’s the best place to start if you’re short on time or money. LG, Motorola, and HTC are also major Android vendors in the U.S., and each has its own unique skin of Android that may behave differently than you expect. It’s best to test on a mix of brands. (Source for brand share: Statcounter GS, 2013).

Q. Can you test Windows Phone and BlackBerry?

A. Yes! Since these are less common platforms, you’ll need to contact us to place the order.

Download these Q&A sets as PDFs:

Related reading: App store optimization for better UX: how to test your listing