Win More A/B Tests by Answering These 3 Questions

Posted on December 1, 2015
3 min read


When it comes to gauging the effectiveness of your A/B testing efforts, there’s one KPI that beats them all: win rate. Sadly, 85-90% of all A/B tests are doomed to fail. It’s not only disappointing to lose an A/B test; it’s also costly. Your team has to go back to the drawing board each time, hoping the newest B version will be the one that works. We’ve mentioned before that some of your most successful A/B test ideas are going to come straight from your users. By observing users’ behavior and seeing where they get confused, you’ll identify the optimization opportunities that actually move the needle, rather than trusting your gut. That’s why it’s always a good idea to run a few quick user tests before you embark on a new A/B test.

Here are three key optimization questions to ask yourself as you plan and analyze your user tests.

1. Where are users getting stuck?

As you review your user test videos, keep an eye out for any significant usability problems your users encounter as they go through the flow of your site. We’re talking about things that prevent your users from doing what they need to do, such as confusing navigation menus, buggy forms, and popups that cover up important information. Fixing these problems can be your key to a successful B version. Your job is to find the low-hanging fruit: the usability issues that are easy to fix and make a big impact on your conversions. Make note of any usability problems that occur, and then organize them by the frequency and severity of the problem. Next, estimate how easy it will be to implement a fix for each one. This will help you plan out your optimization roadmap.

2. What are people missing?

The things people don’t do on your site can be as interesting as the things they do. As you observe users interacting with your site, pay attention to what they’re not doing or noticing. People notoriously skim online rather than thoroughly inspecting every element of your site. That means they’re likely to miss a lot of information on their first visit. You need to identify whether your users are skipping over any crucial info that would lead them to convert—if only they had seen it. For example, maybe there’s a call-to-action that seems really obvious to you and your team, but when you watch your test videos, you find that users are missing it completely because they get distracted by something else further down the page. This will give you the opportunity to make those elements more prominent in your next B version.

3. Does your target market understand your value proposition?

The headline on your homepage or landing page is usually the first thing visitors read, but it doesn’t always hit home. If users spend a few minutes on your site but don’t really understand what you’re offering (or why they should care), then it’s time to try out some new copy. But before writing a clever new line of copy for your next A/B test, run a few user tests to find out whether people understand the value of your product from your existing headline. One way to do this is to have users spend a few moments looking around the site, and then ask them to explain what the company or product does, using their own words. This will help you figure out whether your copy adequately conveys your value proposition. Plus, you’ll get some interesting new ideas from hearing the words people in your target market use to describe what you do.

Take your optimization efforts to the next level

To learn more about combining A/B testing and user testing to optimize your site, don’t miss Optimizely’s webinar, The Hidden Gold in Your Optimization Program, on December 9. Sign up here.

In this Article

    Related Blog Posts

    • A/B test your mobile apps and websites for quick UX wins

      Every product designer or developer needs A/B testing in their toolkit, including those who...

    • Website checklist: Test and optimize your website before launch

      Pre-launch tests enable you to evaluate whether your website can withstand real-world usage scenarios...

    • Researcher's guide to customer personas and creating detailed buyers

      How well do you know your customers? If you’re like most of us, probably...