Usability Testing Siri, The iPhone 4S Virtual Assistant

The new iPhone 4S arrived at UserTesting headquarters. The iPhone 4S internal hardware is better; its HD camera is sharper and it now speeds apps with a dual processor. However, the external sheath remains unchanged. Perhaps this is why it was not called the “5.”

For those who like to show off their new toys, there is no way for people on the street to tell they have the 4S. Unless, however, it talks back.

We decided to put Siri, its famed virtual assistant, to the task to see how well it really worked. (iPhone 4 users who upgrade to iOS 5, still cannot use Siri.) Our team is highly impressed with how far Siri has come, but the technology has noticeable limitations. The commercials said that Siri learns from us, but at the end of the test, I felt like Siri made me her robot slave, or perhaps a disgruntled master.

What Siri Can Do:
1. Call your Mom
2. Schedule a reminder in your calender
3. Tell you the weather
4. Tell you the stock prices
5. Find bars if you want a drink
6. Find the nearest Taco Bell if you say you want “Mexican”
7. Send a taxi if you say “I’m drunk”
8. Shuffle your music
9. Find articles about the latest news
And last…
10. Answer most “Why” and “Who” questions with search results or glib answers. In case you were curious, it will answer “Who’s your daddy?“, but not “Who let the dogs out?”

What Siri Can’t Do
1. Get you driving directions from Point A to Point B, if you are not currently at Point A
2. Add an App
3. Do anything in your browser after the initial Google search
4. Connect device to Bluetooth or WiFi
5. Set Phone to Airplane Mode
6. Find songs or music videos that are not in your playlist
7. Search flights
8. Answer most “How to” questions
9. Answer complex queries
10. Make qualitative decisions (It can only sort Yelp reviews by top rated)

Siri Usability Testing Results

The Good
1. Siri knows facts

In this video, Siri correctly answers “Why is the sky blue?” It’s because of the effect of Rayleigh scattering, of course. She also knows that Al Gore did not invent the Internet, but rather it stemmed from a DARPA project. Siri is integrated with WolframAlpha, a computational knowledge engine which provides access to world facts, data, and peforms calculations.

2. She searches for restaurants if you are hungry
Siri recognizes the term “hungry” and works with your internal GPS to find nearest restaurants. The virtual assistant will find you the closet Taco Bell or Chipotle if you tell her you’re hungry for “Mexican.” Siri can sort by closet proximity and even Yelp ratings.

While it took several attempts for us, the assistant Siri will even point you to the bars. Note: She’ll even responsibly call a cab if you tell her you’re “drunk.”

3. She finds the latest news
Siri was able to quickly pull up news on Occupy Wall Street, partially in thanks to Google bringing the latest news to the forefront of search results.

It felt somewhat unnecessary, though, to have to press the recording button to say to Siri: “Search the Web,” rather than it automatically recording our response right after it asked for confirmation.

The Bad
1. Siri cannot perform complex queries or understand “ideas”
I told Siri I wanted birthday present ideas for my boyfriend. She eventually pulled up directions to a gift stores and flower shops.

Siri does not understand complicated, long queries. This is understandable, but it could better recognize that a common request to “find ideas” for something, whether it’s for a present or travel destination, should arguably produce a search query.

At this point, I began to speak louder and more robotic, in a vain attempt to be understood by the woman in the machine.

2. Siri is biased towards Yelp over Google Search

I asked Siri to find ingredients I would need to bake a chocolate cake. The iPhone pulled up pastry shops. Perhaps the software did not recognize “ingredients” or “bake” and just registered “find cake.” I assumed she knew the phrase “find recipe” so I tried that. Instead, she found me close by restaurants wherein “chocolate cake” is mentioned in their Yelp reviews.

And this was when I started to get angry — frustrated — realizing that I could not control her. I could only do what she let me, how she wanted me to say it.

And Then It Got Ugly
1. Nearly impossible to find video content or search for music beyond your playlist
Siri will perform a Google search for music video results as a last resort. Just saying the term “YouTube” triggered a negative response. This is likely not a conspiracy, but because Siri does not search within the YouTube platform itself. However, it should not be difficult to run a search query that will likely yield YouTube results.

In the video below, it goes from bad to worse, as Siri picks up on all the wrong cues in a failed attempt to find the Party Rock Anthem: shuffling music at random, playing Limbo Rock, and even googling ‘Google’!

2. Won’t turn off the music
Siri’s hyper-sensitivity to music trigger words like “play” and “shuffle” is a one way streak. Once it starts, it doesn’t stop.

Music should not automatically play until Siri first confirms “Do you want to listen to X song?” Later, when I asked it where Footloose (the movie) was playing, it started *playing* music at random!

***

While Siri is too quick to play music, it was annoying how I had to press the recording button to verbally confirm I wanted to search the web, rather then it automatically recording my affirmative or negative response.

Furthemore, Siri could better pick up on cues such as “how to” and “learn” which signify a Google search should be performed. Moreover, it could better recognize words in context. For instance, just the word “to” in “Find ingredients to bake a cake” should differentiate it from “find [food item],” which in our tests demonstrated to trigger Yelp results.

The purpose of Siri is to enable us to be more hands free and get information faster. In the future, Siri will hopefully have better integration with YouTube and Safari.

In my experience, it was unclear if Siri knew when she did not do what I wanted, and left me feeling somewhat helpless, as if she was the boss. Nevertheless, we are pretty impressed by the beta offering, and looking forward to see the technology progress as it learns from interactions from the millions of consumers (they’re selling like hotcakes).

Tell us your experience with Apple’s Siri and any interesting discoveries about its power or limitations.

***

UserTesting now offers mobile usability testing to test your mobile websites or apps on iPhones, iPads, and Droid. Watch videos of real people interacting with your app today.

Join over 24,000 subscribers who receive our email newsletter each month to receive the best content from our blog, and invitations to webinars with top UX and usability experts.