While the topic of qualitative vs. quantitative research may sound intimidating, they’re easy concepts to understand, and they represent things that you’re probably doing already. No matter your industry, most business professionals want to get customer feedback and know their audience, whether you call it research or something else.
First things first, suggesting that qualitative and quantitative research are at odds with one another is misleading. While quantitative research is the method that most people are familiar with (and the one that gets all the credit), the two complement each other fundamentally. Together, they can give a more holistic view of a problem or situation. Having one without the other means that you’re only getting half the story. Both play a valuable role in measuring your customer experience.
For example, after a sizable European car rental company spent time and money developing a car rental subscription membership on a hunch, they were stumped by its poor performance. First, analytics showed that users saw the ads but didn’t sign up. Then, after getting feedback from a handful of users, it became painfully clear why no one wanted to belong to the exclusive club.
Watch Kate Margolis, UX/UI Design Lead at Thirdfort, tell the story.
Thanks to both qualitative and quantitative research, the team changed the membership value proposition to boost its desirability for potential users—leading to a 65% conversion of membership after only being asked once.
While there are significant differences between qualitative and quantitative research methods, it’s essential to understand the benefits and blind spots. So let’s start with quantitative research.
What is quantitative research?
Quantitative research is the process of collecting and analyzing numerical data. It aims to find patterns and averages, make predictions, test causal relationships, and generalize results to broader populations by representing data expressed as numbers.
Quantitative research is unlike qualitative research in one critical aspect—it’s numerical. This is because the output of quantitative research is numbers and statistics.
How UserTesting can help you conduct quantitative research methods
The following features, which are popular ways of conducting quantitative research, are all available on the UserTesting platform.
1. Surveys (ratings, ranking, scales, and closed-ended questions)
Not all surveys have to be taken silently. Our customers can customize surveys, hosted on third-party tools like Survey Monkey, for example, and ask contributors to speak aloud and offer reasoning as they answer questions. Not sure when to use them? At UserTesting, they’re commonly done before development, during the design process, or after a product launch.
2. A/B testing
Suitable for product managers and marketers alike, this method tests two options to see which comes out on top. To ensure quality results and prevent do-overs, get a feel for customers’ preferences and even get qualitative feedback before deciding on what to test. When creating the test, it’s always best to over-communicate any questions to better understand why contributors prefer one design over another.
A major pediatric health system came to UserTesting to transform its website and apps, aiming for a more customer-focused experience. Using A/B testing, the team showed contributors multiple homepage image options and dove into why some were preferred over others. Contributors’ feedback showed that there was a strong preference for photos with real patients over stock images, and the team’s digital marketers were able to share this confidently to stakeholders.
You may know all too well that a website or app is never fully done. As the market and customers’ needs evolve, so must our work. Whether your product launched for the first time, or for the 20th, benchmark studies are a way to assess how your product’s experience has changed over months or years. The results historically document how redesigns and changes are perceived by contributors and how competitors stack up. Consider launching these tests before committing to any serious redesign efforts, and keep it focused, whether you’re targeting the overall user experience or a specific feature.
4. Web analytics
UserTesting offers a variety of analytics tools that display contributors’ screen interactions, customer behaviors, keyword maps, positive or negative sentiments, and more. These tools will help track where users tend to click on a screen, offer a glimpse of where you should dig deeper, highlight common behaviors, and showcase any pain points among others. This helps cut down on any manual effort you might have spent pinpointing and organizing data.
Advantages and disadvantages of quantitative data
What’s excellent about quantitative data is that you can easily replicate it. Quantitative data collection is relatively easy to do, and so is analysis. Since you’re dealing with numbers, it’s typically easier to interpret quantitative data and present your findings to others in a less subjective way.
|Larger sample sizes||Less flexibility|
|Quicker||Can’t follow up|
|Easier||May not reflect actual feelings|
|Less expensive||Lacks context|
|Can uncover patterns and correlations|
|Traditionally easier to automate|
|Offers continuous information|
|Data interpretation is more straightforward|
It’s human nature to trust numbers. We tend to believe they’re concrete. Quantitative methods get more attention because it’s easier to tie quantitative measurements to performance metrics and ROI. But unfortunately, there are many ways numbers can be unreliable.
While numerical data can tell you that there’s a problem, it seldom tells you why. Plus, by focusing on numbers only, there’s a risk of missing a factor or common denominator.
Here are some examples where quantitative data isn’t enough information to make an informed decision:
- An e-commerce agency notices that her client’s shoppers are dropping off on one of their biggest channels before reaching checkout, but they don’t know why.
- A product manager is getting survey data showing that new customers are not satisfied with the onboarding process. While she has an idea of what it could be, she’s not sure where to start.
- A marketing team spent weeks developing and rolling out a campaign that flopped. While the team believes they’re on the right track, the President of the company, who never liked the idea, tells them to abandon it altogether.
As you can see from the examples, quantitative data only gives you half of what you need before you can move forward. Lastly, a significant drawback to quantitative research is that numbers don’t convey stories well. So while it’s easy to share a table of data points with an audience, it’s harder to get them to absorb the information and remember it later.
What is qualitative research?
Qualitative research is a behavioral research method that relies on non-numerical data derived from observations and recordings that approximate and characterize phenomena. It’s collecting, analyzing, and interpreting non-numerical data, such as language. It sometimes seeks to understand how an individual subjectively perceives and gives meaning to their social reality.
Instead of numbers, qualitative data comes from studying subjects in their natural environment and focusing on understanding the why and how of human behavior in a given situation. It’s especially effective in obtaining information about people's values, opinions, and behaviors. At UserTesting, qualitative data is collected through contributor observation and interviews.
How UserTesting can help you conduct qualitative research
Researchers, marketers, product managers, and more conduct qualitative research daily using moderated or unmoderated testing with UserTesting. With our platform, business professionals can have access to qualitative data at the speed of quantitative analysis. These are three common qualitative research methods:
1. Contributor observation
Observing and listening is the core of UserTesting’s platform. Whether you’re conducting a moderated or unmoderated test, observing our contributors to obtain their feedback is vital to improving or redesigning your product. We recommend keeping your questions neutral to prevent swaying any feedback, over-communicating your ask (without bias), and letting the contributor take the lead as they complete the tasks.
For a use case idea, consider optimizing in-person experiences by watching your customers record themselves in a new store or at home—known as an ethnography study. This type of study can highlight how contributors, using rear-facing cameras, have something set up or stored in their home and how they live day-to-day and make decisions. For example, REI relied on UserTesting to better understand various types of buyers and supporters. Using light ethnography studies, REI asked contributors to show how they stored their REI equipment and discuss how long they’ve had them. As for the result? REI became more efficient in testing the right demographic for future studies.
2. In-depth or unstructured interviews
In-depth interviews are optimal for collecting data on individuals’ personal histories, perspectives, and experiences, mainly when exploring sensitive topics or follow-up questions are likely necessary. When asking open questions, the interviewer can get a real sense of the person’s understanding of a situation. For example, they might say one thing, but their body language says something else. You may decide on this method to create better solutions and experiences by exploring your users' attitudes, preferences, and opinions as they test out designs and prototypes.
3. Focus groups
Focus groups are what you may consider a “group brainstorm”—when a moderator effectively gathers information from multiple subjects at once and generates broad overviews of issues or concerns related to the demographics represented. These offer a lot of flexibility, and can be done in person or online. Here at UserTesting, focus groups can consist of up to three or five contributors and while they may be less thorough compared to our 1-on-1 Live Conversations, you’ll get a real time look about multiple people’s thoughts and actions—and how they differ.
Though these can be conducted at any time during a product’s life cycle, we recommend setting these up before design development, before you have limited time and budgets on the line.
Advantages and disadvantages of qualitative data
The most significant advantage to qualitative data is that it’s easy to present your data as a story to your audience. In this way, qualitative data has both staying power and the ability to persuade others. People remember stories and how they make them feel. While charts and numbers can convince others to change, they won’t always translate into action or follow-through. One of qualitative data’s biggest advantages is that it offers rich, in-depth insights that allow you to explore new contexts and deeper understandings.
|Allows for context||Traditionally time-consuming and expensive|
|Empathy||Impossible to replicate|
|Ambiguities and contradictions||Challenging to interpret raw data|
|Deeper insights||Analysis is subjective|
The drawbacks of qualitative research are that it’s often not a statistically representative form of data collection, and it can require multiple data sessions, which can lead to varying analyses and a lot of time and resources spent
Examples of qualitative vs. quantitative research questions
When planning research, you want to be strategic with your test questions. Here are some examples of qualitative vs. quantitative questions to give you a better idea of what to ask.
Quantitative research questions
Quantitative research questions are typically set up so that the answer is numerical or statistical or so that the answer is objective. Typically, this process is automated and answers can’t be followed by more questions.
- How long have you been a customer of our organization?
- On a scale of 1-5, how likely are you to purchase our products again?
- How often do you drink coffee at home?
- Do you prefer to watch movies at home or in the theater ?
Qualitative research questions
Qualitative research questions are open-ended. The interviewer can react to answers and probe for more detail.
- What does the app need to do to improve your experience?
- Do you have any comments, questions, or concerns about our website?
- What do you like most about your favorite coffee shop?
- What did you like about this movie?
Why you need both qualitative and quantitative research
Most importantly, the intersection of quantitative and qualitative data methodologies is where human insights come to life. Both methods can be helpful, but combined, they allow you to see things you may have missed.
According to Justin Wei, Former Head of Digital Marketing at Royal Wins, while quantitative data is the black and white picture of a problem or opportunity, qualitative data can color your understanding.
Quantitative data is 'the what' and qualitative data is 'the why'
Commonly, quantitative data will surface trends that you can use as a springboard for qualitative research. However, it’s important to use qualitative research to drive innovation. Organizations that fall into the habit of only using qualitative research to react to quantitative data run the risk of reducing team efficiency and restricting their ability to optimize.
In general, here are some common reasons to use qualitative research or quantitative research:
- Validate hypotheses: quantitative research will get you the key performance indicators (KPIs) you need when you need objective information to confirm or disprove your theory.
- Find answers: It’s typically easier and less expensive to have people fill out a survey than participate in a focus group. In this way, quantitative methods can help answer questions like: were you satisfied with your experience? Would you recommend us to a friend? On the other hand, qualitative research enables you to respond to open-ended questions like: why were you satisfied with your experience? Why would you recommend us to a friend?
- Uncover emotion: qualitative research is especially good at uncovering the emotions behind data. This can be verbal, body language, or facial expressions caught on video. It helps to hear and see your customers describe wants, needs, concerns, frustrations, etc. Qualitative data will get you that.
Watch Jonathan Greenblatt, User Research and Design Leader, explain how WarnerMedia uses quantitative and qualitative research to flesh out its user personas.
Want to learn more?
Grab a copy of User Tested: How the World's Top Companies Use Human Insight to Create Great Experiences, co-authored by UserTesting’s CIO Janelle Estes and CEO Andy MacMillan.