Sometimes asking a user to search for a particular phrase is helpful, but usually, you want to have a scenario that is open-ended enough that users will use their own mental model and linguistic phrases, not used the supplied language or steps that set your site up for success. You want to find problems, so let users do their own thing.
Good Test Question – Open-ended, but specific.
Using search, try to find a pair of shoes in a color, style, and size the appeals to you.
Bad Test Question – Feeding users specific color and brand that your site might work fine for.
Search for “brown dockers.”
Bad Test Question – Open-ended, but not specific.
Search for shoes.
Your site may perform great for someone searching for women’s shoes, but not men’s shoes. Depending on gender, how price-conscious a customer is, their age, or computer savviness, a user may use vastly different search keywords and filtering behavior to find the same product. UserTesting.com allows you to choose the demographics of your panel to test all of these different possibilities.
We asked a man to find brown dress shoes. He didn’t think to type "men’s dress shoes." His first three result pages were filled exclusively with women’s shoes, and there was no readily apparent, above-the-fold option to filter for men’s shoes.
A term like coffee table book has one meaning; someone wants to buy a book to put on their coffee table.
Amazon understands the difference between the search "coffee table" and "coffee table book."
But these same searches on walmart.com yield puzzling results.
Not only does the search on Walmart for “coffee table books” yield no relevant results (even though they do sell the product), but indescribably, increasing the specificity of the query returned 1,000 more results. Walmart clearly knows this is a problem and is investing significant resources in improving the situation. They recently released a new version of their search engine built over 9 months by 10 to 15 developers. (Source: Reuters).
The results of a bad search can significantly affect brand positioning. In the video below, the user, after failing to find a book on walmart.com says, “I wouldn’t go to walmart.com to shop for books. I am just going to look through the next couple of pages, I really don’t expect that [books] will come back.”
A user is looking for gardening coffee table books (of which Wal-Mart sells many). He never finds them.
A search for "returns" on macys.com leads to the following, “permission denied illegal parameter” message.
A search for "returns" on kohls.com leads to the following ad for “Superman Returns.”
A search for "returns" on Forever 21 leads to the ominous "0 RESULTS."
The auto-suggested phrase, "returns policy" on sears.com oddly enough redirects visitors to the watches page. Are they subtly trying to say that the watches are likely to be returned?
When the visitor searches for the auto-suggested phrase, "returns with a receipt," he gets a not-so-subtle hint, “Buy a wallet to help you keep track of your receipts!”
Target hits a bullseye with a search for "returns" bringing the guest directly to the return's page.
Our tests show that people don’t usually choose a department before searching. However, if they do choose a category, one of the biggest problems they have is getting trapped in the category and not being able to get good results, especially if category/department names are ambiguous.
A 55-year-old woman is looking for a gift for her mother. She chooses the “Home and Gifts” category then searches for book-related topics, but there are no book results.
A search on Macy’s for the comfy shoe brand Keens just shows me ads for other companies that sell Keens. I suppose sending me to another retailer is one way to monetize my visit, but why not suggest I check out the competing brand Ecco that you do carry?
Drugstore.com gets it right. A search for "bumble and bumble" lets me know it’s no longer available and shows me other relevant options.
Accounting is very different than Accounts, but it is the same as accountancy. Finding something as simple as the accounting degree on a college website should be easy, but our users struggled on many college sites.
While Boston University has a Management School that offers a BS in Business Administration with a concentration in accounting, you won’t find it through their search engine.
Searches for "accounting degree" or "accounting program" are stemming to the keyword "accounts," which leads to IT related pages on how to create a user account, reset your account password, etc. None of the results are remotely relevant.
Sometimes stemming is too liberal, but oftentimes it is not liberal enough.
Searching for "accounting major" on Notre Dame leads to a list of irrelevant faculty pages, old PDFs, and news articles.
This search doesn’t work because Notre Dame lists their accounting program under the term "Accountancy."
Amusingly enough, a search for "accountancy major" brings up the correct result in the top position, but leads to an auto-correct suggestion that will lead you astray.
Accounting Degree is not someone’s name. You should know that since nobody is named that in your directory. If a universal result type returns 0 results, are you hiding those search results?
A prospective student searches for the accounting degree at the University of Texas. They are alarmed because the first place their eyes go to is the “Search Returned 0 Results”. It takes them several seconds to realize that web results are below people results and that there are web results available.
Algorithms change, new products are introduced, special events and promos come and go, and users' expectations shift.
In Google Analytics you can view exit rate and refinement rate by keyword. Almost every keyword with an exit rate above your average site exit rate is a query that can be optimized.
In Search Patterns, Peter Morville and Jeffery Callender remark:
Search is integral not only to site conversion goals, but also to brand positioning goals. But search quality is not a problem to be solved, it is a never-ending pursuit. Regular and rigorous usability testing of search can help you meet and exceed your customers’ expectations, raising value per visit in the short term and greatly increasing the potential revenue per customer in the long term.
Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.