How to test mobile apps and websites

Summary

The rules of software success in mobile computing are different from the desktop. Mobile devices are used in different ways than computers, in much more varied locations, with inconsistent wireless coverage, higher levels of distraction, and different customer expectations. Mobile apps and websites need to be tested in real-world conditions to produce a great customer experience. This guide describes what needs to be tested and how to do it.

Why test mobile experiences, and what to test

Here are the key aspects of mobile that you need to test, and the ways they can improve your mobile experience.

1. Test in the wild. This is the most important priority: Test your mobile experience on mobile devices that are being used in the real world, in as realistic a setting as possible. Many experiences that work great in a lab or in a desktop prototype fail badly when they get into the real world. The usual culprit is distraction: people use mobile apps while they are doing other things, and they will routinely overlook notifications and details that they would spot on a desktop. You won’t always hear about these problems from customers, because they’ll blame themselves for failing to notice something. But a bad experience still reduces your customer loyalty, and makes it much easier for a competitor to steal your customer.
2. Test icons and labels for comprehension. Because a mobile screen is so much smaller than a desktop, it’s tempting to use tappable icons heavily in the interface, often without any explanatory text under them. This is risky; if a user can’t understand the purpose of an icon, there’s no popup tooltip function they can use to get more information. Sometimes the meaning of a particular icon will be different in Android versus iOS, which can lead to even more confusion. You need to test user comprehension of any icons or other labels you’re using.
3. Test the full omnichannel journey. Many mobile apps and websites are part of a broader experience. For example, an online shopping app for a brick-and-mortar store needs to sync with the real-world pickup function. You need to test that entire journey, including the real-world pickup process. That’s possible if your insight solution has the right capabilities.
4. Test your ability to operate outside of wireless coverage. Many mobile developers assume that the user will always have wireless coverage. But are you absolutely sure of that? Even if your users stay in an urban area, what happens when they go into a parking structure or store where wireless coverage is limited? A great mobile experience will cache information so the user isn’t stopped by a glitch in coverage. You can run user tests to make sure that works well.
5. If you’re testing a mobile app, test your first-time user experience (FTUE) and app store assets. Launching a mobile app is like launching a movie—if you don’t make an immediate great impression, you’ll get bad word of mouth, and the customers may never come back. It’s critical to test your first-time user experience thoroughly to make sure it’s satisfying and entices return visits. You should also test the assets in your app store listing, including screenshots, app icon, app name, and written description. Make sure all of them resonate with your target customers.
6. Use the backside camera to get real-world context. A good human insight system will let you turn on the backside camera to record the user’s surroundings. Where are they using your mobile website or app? What else is going on around them? Are they able to navigate any transitions from online to the real world? If you understand your context, you can do a much better job of optimizing your experience.
7. Understand the law. In the US, you can record most experiences in which a reasonable person would not expect privacy. That’s not necessarily true in other countries. Before launching a test that requires use of the camera, be sure you understand the recording laws for your location and give appropriate instructions to the participant. 

Alternate ways to test mobile are limited

Some human insight solutions test mobile on a desktop or laptop computer, by displaying an image of a smartphone screen and asking people to interact with it using the mouse and keyboard. Others allow the participant to use a mobile device, but it has to be plugged into a computer to record the test. Another approach to mobile testing is to bring people into a research facility and use video cameras to record the test session.

All of these approaches are unnatural and don’t allow you to fully understand how the mobile experience works. In particular, the unnatural setting is almost always quieter and less distracting than the real world, making it very difficult to understand whether the experience will perform adequately when the user is out and about.

Some companies use quality assurance testers to also run mobile user experience tests at the same time. This approach is risky because once testers have been trained to look for bugs, they no longer think and behave like regular users. QA testing is great for bugs, but to optimize the experience you need a different kind of tester and different processes.

Some companies outsource their mobile app development to a third party. No matter how much you trust your supplier, it’s important to test any app that will bear your brand. If the experience isn’t good it will hurt your brand image, and you may not even realize it’s happening.

The fast way to test in the real world

A good human insight solution will have dedicated features for mobile testing. The capabilities to look for include:

• Test in the wild. Mobile tests can be recorded anywhere wireless coverage is available—at home, at work, and on the road. This lets you see how the app or website performs in real conditions, with variable lighting, noise, and numerous distractions.
• The supplier should automatically install the SDK. For an app test, you should be able to just upload your build and the human insight solution will provision the software to participants’ devices.
• It should be easy to recruit the right people. The larger and more active your vendor’s contributor network, it’s easier to find the right people for your test. The participants should be accustomed to recording things with their smartphones, so it’ll generally easy to get good results. And if you need special recruiting help, the vendor’s audiences team should help you recruit the right people.
• Camera recording is built in. The insight vendor’s mobile recorder should automatically record the backside camera, capturing any experience you can record with a smartphone. 

You should be able to get help with special logistics. Your vendor’s pro services team can help you if you need people to test in a particular location, to do an unusual task, or to unpack or use a device that you provide to them.