1. Make your scenarios open-ended, but specific
Sometimes asking a user to search for a particular phrase is helpful, but usually you want to have a scenario that is open-ended enough that users will use their own mental model and linguistic phrases, not used the supplied language or steps that set your site up for success. You want to find problems, so let users do their own thing.Good Test Question – Open-ended, but specific.
Using search, try to find a pair of shoes in a color, style and size the appeals to you.Bad Test Question – Feeding users specific color and brand that your site might work fine for.
Search for “brown dockers.”Bad Test Question – Open-ended, but not specific.
Search for shoes.
2. Test a wide array of visitor demographics
Your site may perform great for someone searching for women’s shoes, but not men’s shoes. Depending on gender, how price conscious a customer is, their age or computer savviness, a user may use vastly different search keywords and filtering behavior to find the same product. UserTesting.com allows you to choose the demographics of your panel to test all of these different possibilities.
We asked a man to find brown dress shoes
. He didn’t think to type "men’s dress shoes." His first three result pages were filled exclusively with women’s shoes, and there was no readily apparent, above-the-fold option to filter for men’s shoes.Watch Video
3. Test ambigious terms or terms that could have multiple meanings
A term like coffee table book
has one meaning; someone wants to buy a book to put on their coffee table.
Amazon understands the difference between the search "coffee table" and "coffee table book."
But these same searches on walmart.com yield puzzling results.
Not only does the search on Walmart for “coffee table books” yield no relevant results (even though they do sell the product), but indescribably, increasing the specificity of the query returned 1,000 more
results. Walmart clearly knows this is a problem and is investing significant resources in improving the situation. They recently released a new version of their search engine built over 9 months by 10 to 15 developers. (Source: Reuters
The results of bad search can signifantly affect brand positioning. In the video below, the user, after failing to find a book on walmart.com says, “I wouldn’t go to walmart.com to shop for books.
I am just going to look through the next couple of pages, I really don’t expect that [books] will come back.”
A user is looking for gardening coffee table books (of which Wal-Mart sells many). He never finds themWatch video
4. Test non-product keyword phrases
A search for "returns" on macys.com leads to the following, “permission denied illegal parameter” message.
A search for "returns" on kohls.com leads to the following ad for “Superman Returns.”
A search for "returns" on Forever 21 leads to the ominous "0 RESULTS."
The auto-suggested phrase, "returns policy" on sears.com oddly enough redirects visitors to the watches page. Are they subtly trying to say that the watches are likely to be returned?
When the visitor searches for the auto-suggested phrase, "returns with a receipt," he gets a not so subtle
hint, “Buy a wallet to help you keep track of your receipts!”
Target hits a bullseye with a search for "returns" bringing the guest directly to the return's page.
5. Test sequential searches across multiple departments or categories
Our tests show that people don’t usually choose a department before searching. However, if they do choose a category, one of the biggest problems they have is getting trapped in the category and not being able to get good results, especially if category/department names are ambigious.
A 55-year-old woman is looking for a gift for her mother. She chooses the “Home and Gifts” category then searches for book related topics, but there are no book results.Watch video
6. Test Product / Brand Not Offered / Discontinued
A search on Macy’s for the comfy shoe brand Keens just shows me ads for other companies
that sell Keens. I suppose sending me to another retailer is one way to monetize my visit, but why not suggest I check out the competing brand Ecco that you do
Drugstore.com gets it right. A search for "bumble and bumble" lets me know it’s no longer available and
shows me other relevant options.
7. Carefully test words your search engine may stem
Accounting is very different than Accounts, but it is the same as accountancy. Finding something as simple as the accounting degree on college website should be easy, but our users struggled on many college sites.
While Boston University has a Management School that offers a BS in Business Administration with a concentration in accounting, you won’t find it through their search engine.
Searches for "accounting degree" or "accounting program" are stemming to the keyword "accounts," which leads to IT related pages on how to create a user account, reset your account password, etc. None of the results are remotely relevant.
Sometimes stemming is too liberal, but oftentimes it is not liberal enough.
Searching for "accounting major" on Notre Dame leads to a list of irrelevant faculty pages, old PDFs, and news articles.
This search doesn’t work because Notre Dame lists their accounting program under the term "Accountancy."
Amusingly enough, a search for "accountancy major" brings up the correct result in the top position, but leads to an auto-correct suggestion that will lead you astray.
8. Test universal search results
Accounting Degree is not someone’s name. You should know that since nobody is named that in your directory. If a universal result type returns 0 results, are you hiding those search results?
A prospective student searches for the accounting degree at the University of Texas. They are alarmed because the first place their eyes go to is the “Search Returned 0 Results”. It takes them several seconds to realize that web results are below people results and that there are
web results available.
9. Test results frequently
Algorithms change, new products are introduced, special events and promos come and go, and users' expectations shift.
10. Test queries with that have a high exit rate or a high refinement rate
In Google Analytics you can view exit rate and refinement rate by keyword. Almost every keyword with an exit rate above your average site exit rate is a query that can be optimized.
Conclusion – Tame the beast
In Search Patterns
, Peter Morville and Jeffery Callender remark:
Search is an elephant that hides in sight because executives lack the right radar. Many in management don’t realize the role search plays in defining the user experience. They fixate on the home page, they fuss about look and feel, and they care about the content. They may even fume about findability, but they are easily distracted or misled because they really don’t understand search.
Let’s face it: search is a wicked problem with no definitive formulation, considerable uncertainty, and complex interdependencies. Stakeholders have divergent goals and radically different world views. Requirements are incomplete, contradictory, and ever changing. Search is both a project and a process. It’s a problem never solved.
Search is integral not only to site conversion goals, but also to brand positioning goals. But search quality is not a problem to be solved, it is a never-ending pursuit. Regular and rigorous usability testing of search can help you meet and exceed your customers’ expectations, raising value per visit in the short term and greatly increasing the potential revenue per customer in the long term.