In this three part series Jessica DuVerneay, Los Angeles based information architect and user researcher at The Understanding Group, will share testing tips on When to Test, What to Test, and How to Test.
As discussed in the last part of this series When to Test, unmoderated testing can add value at any stage of the product design cycle. Knowing when to test is the first step - knowing what to test is the second.
This is the area of inquiry where most companies or IA/UX professionals will begin. Unmoderated tests can identify high-level issues, or can drill down into a specific flow or conversion point depending on how you structure your tasks & scenarios.
There are some basic measurable indicators to consider while usability testing. (Note: The indicators listed below heavily reference Measuring the User Experience by Tullis and Alberts)
It is worth mentioning that what constitutes success for one product may not necessarily be indicative of optimal state for another product.
Furthermore, while aggregating qualitative data is valuable, do not make the mistake of ignoring the insights extracted from qualitative information (opinions, ratings, exclamations, and comments), as they can guide some of the most effective product changes.
One of the lesser-known benefits of unmoderated testing is the ability to show a test subject a competitor’s product without the risk of moderator bias. Learning from a competitor - which flows, content, and design patterns are successful and which should be avoided - can be particularly useful in the startup space where lean UX teams may have to make complex high-stakes product decisions on limited research.
Questions to consider during competitive testing might include:
These tests are relatively easy to set up yourself, but UserTesting also offers assistance with competitive testing through their Agency and Enterprise subscription packages.
I’ve seen it, my colleagues have seen it, and you’ve probably witnessed it as well - a product is ready to launch and some inane battle about “The Blue Version vs. The Orange Version” threatens to delay launch of the product by weeks.
Preference testing can assist teams in determining which approach to run within a lean and cost-effective fashion, independent of internal personality conflicts or petty political battles.
While the statistically meaningful solution is to launch both versions and validate success via A/B Testing, oftentimes companies do not have the bandwidth or time to create and support multiple versions of a site. Akin to competitive testing your product against itself, Preference Testing with remote users can help you get a few user opinions to back up which product to launch.
Preference testing may provide direct user insights around:
As UserTesting.com allows test subjects to access any content that can be hosted on a URL, showing a live, ready to launch product is not necessary. Linking to multiple versions of a visual comp or an HTML wireframe can provide great insight from users.
I saw this tweet a few weeks ago:
This is a sentiment I wholeheartedly agree with. If you test nothing else, or at no other point - please do a Validation Fit and Finish test of your product prior to launch.
Fit and finish is a “measure twice - cut once” approach that will prevent unknowingly faulty products from being released. It will also build confidence in leadership who may have been removed from the design process to see a handful of users positively interact with a product just prior to launch.
Validation of fit and finish should address:
The main question for a Validation of Fit & Finish should be “Are there any show stoppers that should delay launch?” While other usability or preference issues that appear can certainly inform future usability testing efforts, they should never confuse the main goal: to feel confident about launching and identify any insurmountable fatal flaws prior to launch.
While the information you can gather from unmoderated user testing about Usability Issues, Competitive Analysis, Preference and Fit and Finish should be ample motivation to jump in and add user testing to your design cycle immediately, there are additional ways to use testing to improve your product. Some examples:
Don’t limit your testing strategy or use of unmoderated testing to tried and true approaches that you are comfortable with. Think about the nature of what you need to know to make your product better, and try new approaches as your schedule allows. Your product, your team, and hopefully your users, will thank you for it.
This diagram is an example of how the different types of testing outlined in the above article might work in a singular design cycle - showing you now not just when to test, but also now what to test.
When creating your test, it is of note to understand that you may write a test that approaches multiple topics. For example, I’ve written a battery of tests for a product that address specific IA & UX issues, Persona Info, and Competitive Analysis at the same time quite effectively.
In the final installment of this series, I will share some thoughts on How to Test. I’ll address a few common mistakes to avoid when creating, analyzing and presenting the findings of your unmoderated test. And, in case you missed it, you should check out part 1 of this series about When to Test.
Until then, feel free to comment below with questions or your thoughts or email me at email@example.com. Thanks for reading!
Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.
About the author:
Jessica DuVerneay is Los Angeles based information architect and user researcher at The Understanding Group. DuVerneay has been happily using UserTesting for the past several years, and teaches workshops for both practitioners and clients on Lean Unmoderated User Testing and other IA topics.