The company has a practice of running an annual desktop benchmark study to see how they stack up against the competition. However, this time around they wanted to run a large mobile benchmark study to see how their mobile site experience compares to their competition.

The goal of the study was also to assess the impact of mobile site experience on brand perception and to establish a baseline to compare against the following year.

Challenges

The company wanted to understand not only where their mobile site stands relative to competitors but also whether the difference is statistically significant.

After establishing the baseline, they wanted to run the same study again next year to see whether the difference is higher or lower and what design changes are impacting the experience.

The company knew that running a study with 20 or 30 people would not allow them to detect anything but the largest differences. So running a study with a large sample size was very important to them.

Running a large-scale mobile study in a lab was not feasible as well as it didn’t allow them to have participants in their natural context.

Solution

The company knew that running a study with 20 or 30 people would not allow them to detect anything but the largest differences. So running a study with a large sample size was very important to them.

Running a large-scale mobile study in a lab was not feasible as well as it didn’t allow them to have participants in their natural context. Participants were asked to complete three different tasks related to finding a specific product and adding an item to the cart.

They were also asked to answer a series of questions about the brand and about their experience. The study included participants with both iOS and Android devices.

Results

With UserZoom, the company has gained a deep understanding of how the user experience on their mobile site compares to their competition. Participants rated the company second-lowest on several key metrics including ease of use, credibility, and likelihood to recommend the site to others.

By obtaining such important metrics as effectiveness and efficiency ratios, and collecting user behavior, the company was able to uncover the areas that needed the most improvement on the mobile site.

The study also showed that participants’ brand perception improved from pre to post mobile site evaluation for all of the sites tested in the study. However, the company’s increase in brand perception was significantly lower compared to some of its competitors.