CarMax + UserTesting

Challenge

CarMax revolutionized the car-buying experience when it was founded in 1993. While they were a successful company with an extremely popular online presence, they knew they couldn’t become complacent. The online world was changing and with it the shopping trends of online consumers.

That’s when the organization doubled down on its digital product development efforts. They began incorporating lean/agile practices. They sought ways to improve the customer experience and connect with car buyers in innovative ways. They also hired product and UX experts, including Archie Miller and Chip Trout, to help the product teams leverage Lean UX methods to build better, more effective products, faster.

Before the product teams were formed, the majority of customer research was conducted by web analysts. While the analysts’ reports were useful, the teams who actually built the products didn’t always have the opportunity to obtain hands-on experience of running the tests, watching the videos, and building empathy for the user. The company also did in-person testing in usability labs, which, while insightful, could be expensive and time-consuming, and it was a constant struggle to recruit first-time users.

These research methods helped the company make UX improvements, but didn’t spark the kind of digital innovation CarMax needed. They needed to empower their teams to get valuable user insights more frequently—and firsthand—so they could design solutions that moved the needle for the organization.

Solution

Having the product manager, the product designer, and the product engineer all involved in the user testing process was key. Not only did the product team members set up the UserTesting studies themselves, but they also watched and discussed the videos together, allowing them to find new ideas and solutions. Miller said, “This gives the team a sense of shared ownership and shared understanding of how customers use the product.”

The product teams at CarMax now user test everything, including their website, prototypes, landing pages, mobile apps, and even TV commercials. They do research throughout the development cycle of every product, both to generate new ideas to solve problems and to evaluate the solutions they’ve developed. They get feedback through UserTesting within minutes by keeping their demographic requirements fairly open. As Miller shared, “Everybody’s looking to buy a car at some point,” so the products need to be usable for everybody. For special projects, however, the team can also quickly recruit test contributors at a certain stage of the car-buying funnel.

The team members and stakeholders get together to watch the UserTesting video clips and unpack how the customers interact with the product. Any disagreements about the design or product direction can be discussed and resolved then and there, rather than relying on notes and additional meetings. The team uses insights from the videos to make empathy maps that include customers’ feelings, actions, behaviors, and quotes. They apply these pieces of information to prioritize new features and improvements that will have a high impact on the customer experience and the business.

Most importantly, CarMax can quickly evaluate, validate, and iterate on product ideas that will impact revenue and conversions. The product teams often come up with hypotheses during in-person user interviews and then run a series of user tests to refine the idea and use KPIs as a way to measure their impact

Results

CarMax has seen results from their research in multiple products and campaigns. One recent win was the “badge” feature. During discovery, the team noticed customers spending time reviewing photos of the tires. They ran several UserTesting experiments with different solutions for calling out how many new tires a car has and watched and listened as customers noticed (or didn’t notice) that information on the page.

Through rapid iteration, they developed a solution that made customers take note. That has spawned a whole other discovery project: what other valuable and unique content could they badge? In addition to uncovering new ways to increase sales and conversions, the product teams at CarMax use UserTesting results to influence company decisions, effectively giving their customer a seat at the table with executives and other stakeholders.

Trout explained, “I let the users tell their own stories. It’s one thing for a Product Designer to make a recommendation—it’s another, more powerful thing, to combine that recommendation with video from real users. These clips are shown at bi-weekly Open Houses where the company can see firsthand what users think of a new idea or feature. The UserTesting clips have definitely helped us educate stakeholders and influence product decisions.”