On a scale of 1 to 5, how unimportant (1) or important (5) is it to get awesome, reliable data alongside your qualitative UX research findings?
Hopefully, you think it’s important, and if you did, then read on! In this article, you’ll learn how to use Rating Scale questions to supplement your qualitative research with some good, hard data.
In a previous post, Streamline your UX research with customer experience analytics, I discussed the overall benefits and guidelines that UserTesting’s Research Team keeps in mind as we incorporate survey-style questions into our studies. This post will dive specifically into the creation and use of Rating Scales to support your qualitative findings.
Rating Scale questions are the most utilized of the three types of metrics questions, probably because they are the easiest indicator of user highs and lows. The responses provide terrific data for the creation of charts and graphs, which can help you convey to others how your team’s product is performing.
How difficult (1) or easy (5) was it to log in to the app using Facebook? (1=Very difficult, 5=Very easy)
How unlikely (1) or likely (5) would you be to recommend this game to a friend? (1=Not at all likely, 5=Extremely likely)
How cluttered (1) or organized (5) do you consider the home page? (1=Cluttered, 5=Organized)
How untrustworthy (1) or trustworthy (5) do you consider the site? Please explain your answer aloud. (1=Not at all trustworthy, 5=Entirely trustworthy)
How disinterested (1) or interested (5) are you in purchasing boosters for this game? (1=Not at all interested, 5=Extremely interested)
Please rate your agreement with the following statement: "I understand what I can do here and who this site is for." (1=Strongly disagree, 5=Strongly agree)
Giving clear labels for each endpoint of your Rating Scale will make your question easy to understand.
Rating scales are highly adaptable, and they can be used to measure many different things, but only if the people taking your test understand what you’re asking them to measure!
Make sure that your question involves two ends of a spectrum (and that you’re effectively communicating those endpoints).
To prevent bias, make sure you include both endpoints (the good and the bad) in your question.
People are naturally eager to please, so if you ask users how easy it was to accomplish something, their inclination is to say it was easy---even if they struggled.
To avoid introducing bias into the response, it’s best to mention both ends of the spectrum as you ask your question (in this example, how difficult or easy). Including the both sides of the scale in the body of your question also reinforces the endpoints you’ve defined and lowers the chance that a user will misread your question.
We tend to consider low numbers to be bad, and high numbers to be good. Take a look at the chart below, for the example.
Which of these tasks seems like it was hardest to accomplish? Which one was easiest?
Even without knowing the question, you probably assume that finding an item didn’t go very well, while adding an item to the cart went just about perfectly. We automatically associate higher numbers with good news and lower numbers with bad news.
But what if I told you that in this question, 1 meant "Very Easy" and 5 meant "Very Difficult?" It drastically changes our understanding of the results.
Because we tend to think of the highest number as the positive point in a rating scale, we can easily overlook a scale that has been labeled the opposite way:
This question uses endpoints that are the opposite of what we would naturally expect.
The UserTesting Research Team has repeatedly observed that if confronted with a question like this, users will often provide a positive verbal response (i.e. "It was great!"), but click the “5” radio button, despite the label saying that 5 = Low Quality.
So do yourself---and your users---a favor, and always label 1 as the “pain” point, and 5 as the “positive” point.
Pro tip: If you set up a rating scale question and realize that the two endpoints are neutral and/or the positive point is right in the middle, consider turning it into a multiple-choice question instead.
For example, asking users to rate something on a scale of “Too small” (1) to “Too big” (5) is a sure-fire way to confuse your users, and it’s more difficult to tally up the results of ratings than it is to determine how many users answered “Too big”, “Too small,” or “Just right” via multiple choice.
Imagine you’re testing out a new application. You found it in the App Store and downloaded it easily, but spent 7 extremely frustrating minutes trying to find the option to create a new account. Then you’re confronted with the following question:
This question is actually asking two separate things (downloading the app and setting up the account).
Do you rate the first part (downloading the app) as Very Easy? Do you rate the last part (account setup) as Very Difficult? Or do you split the difference, and average out the two ratings by answering 3?
Our Research Team has watched users do all three options, which makes for some very messy quantitative data; so try not to measure too much with your Rating Scale questions.
Instead, write individual rating scale questions for each task that your users perform. If you’re worried about users mistaking the second question as a duplicate, you can always capitalize the task you are asking users to rate, for additional emphasis:
Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.