Skip to main content
Close PromoBar
Access now Access now Access now The executive's guide to empathy-driven ROI   Access now    The executive's guide to empathy-driven ROI Access now Access now Access now

Optimize your AI experience with human insights

Asha Vergis  |  May 01, 2018

It wasn’t that long ago that artificial intelligence (AI) was a futuristic concept. Now, computer systems can complete tasks that historically required human intelligence. Smart home systems like Amazon Echo and intelligent personal assistants like Siri can understand your commands and complete tasks for you. You don't even need a partner to play games anymore—computer systems, like DeepMind’s AlphaMind Zero for the game Go, can be your counterpart in multi-player games. While these advancements are exciting, they also introduce new considerations when trying to deliver the best customer experience. Before the use of AI was widespread in technology, experiences on digital platforms were rigid. The customer’s input was dictated by the platform, and therefore the output was always pre-determined. AI revolutionized this model—now the customer dictates the input, and the computer system uses this input to provide a relevant output. In addition to these big changes, AI interfaces aren’t always tangible, which introduces a new interface with its own challenges for customers.

Use human insights to uncover areas of opportunity

AI would not be possible without copious amounts of big data to inform its responses. However, it is critical to also incorporate qualitative, human insights into AI development as emotional understanding and emulation are key to making AI-driven experiences successful. We recently ran a study on smart home systems to evaluate the experience of using these systems to accomplish some common tasks. Smart home customers were asked to create a shopping list, play music, and plan a movie outing. Here's how non-tangible AI systems can provide an exceptional experience:

Make the parameters of the verbal command or action obvious to the customer

With tangible interfaces, the input or action is clear. But with AI systems, verbal commands are not obvious and need to be defined or risk providing a poor customer experience. For example, the success of adding multiple items to a list using a smart home system depended on how the customer gave the command:

  • If the customer paused between each item, the device would accurately add each of the items to the list
  • Any “ums” or rambling between list items would cause the smart home system to incorrectly add an item or get confused

To mitigate this confusion, instructions on how customers should construct their commands to smart home systems would help them accomplish their task faster.

Allow an easily accessible “undo” feature

Most AI systems have “stop” and “go” commands that allow customers to start and end interactions with the system. But, we found that undoing actions was challenging because it couldn’t be done using the same verbal command “language.” Adding items to a shopping list on a smart home device was easily done through voice commands; however, when attempting to remove items from the list using voice commands, customers were frustrated to learn that they couldn’t. They needed to use the smart home system’s app to remove items from their list. This hurdle was frustrating because customers expected to edit their shopping list the same way they created it—using verbal commands instead of an entirely different device.

Provide the means to complete an activity or tasks

Just like in human-to-human conversations, back-and-forth interaction between the AI system and customer is required to complete some tasks. Take, for example, planning a trip to the movies. There are many factors that go into seeing a movie in theaters: selecting the showtime, theater, movie, etc. Some smart home systems listed out the movies that were playing at a theater close by but were unable to take it a step further by letting the customer know specific times that movies were showing. This information was essential for customers to make plans. Since the smart home system did not provide a means for customers to complete their planning, they needed to use another device to complete the activity independently.

Enable seamless omnichannel experiences

If your smart home system directs customers to a mobile app, the transition to another device to complete the activity should be seamless. Our study found that experiences became fragmented when customers needed to use another device to complete their task:

  • Though smart home systems were able to successfully communicate to customers that they needed to use the app to remove items from their shopping list, customers were not able to find the shopping list in the app.
  • While trying to play music from their smart home system, many customers were instructed to link their music streaming account to the smart home device through the smart home system’s app. However, once customers got to the app, they were unsure how to link their account or choose which streaming service they wanted to make their primary service.

How to effectively gather human insights to inform your AI system

Testing and validating your AI system isn’t always as simple as sharing a URL and providing some sample tasks; and only referencing your analytics will provide you with what is happening but won’t provide you with the full picture of why it's happening. Here are three techniques to keep in mind when running remote, unmoderated studies on your AI system.

1. Set the stage with the right introduction

Communicating the right expectations ahead of tasks and questions will enable customers to test your AI experience effectively. Sometimes, you will need to provide a more thorough introduction versus just making sure customers are in the right mindset. For example, if you are testing a prototype of your AI experience, make sure you specify the commands that customers need to provide. If you set parameters that are too broad, the AI system may not respond correctly, and the feedback you receive won’t be actionable. Or, if you are testing how well the AI system responds to certain emotions, you should make it clear to the customers that they should be “feeling” that way.

2. Validate the perception of success

Something that might be deemed “successful” in your analytics might not be perceived as such by customers. For example, the AI system is successful when it produces the correct answer to a specific command, or when it provides the customer the tools to complete an activity. However, if to receive the right response, the customer needed to provide more commands than they expected, then the interaction might not be viewed as a success by the customer. Understanding and aligning with your customers’ perception of success via human insights will enable your AI system to create a better customer experience.

3. Use this opportunity to gather emotional feedback

With AI systems, success can also be perceived through the tone of the customer response—characteristics that can’t be easily measured through metrics. To work effectively, AI systems need to also understand and empathize with your human customers. Use qualitative studies to understand if the AI experience is responding correctly to emotional cues. Does the system react appropriately to your customers’ frustration, sadness, or other negative feelings? Identifying points of misalignment between the AI system and your customers allows you to make informed optimizations, whether it is to change the response or to redirect to a human operator.

Want to learn more?

If you’d like to learn more about how UserTesting can help you understand your customers through on-demand human insights, contact us here.

Insights that drive innovation

Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.

Thank you!
Get ready for some great content coming to your inbox from the team at UserTesting!

About the author:

Asha is a Senior UX researcher at UserTesting, hoping to create a more user friendly world. She enjoys beautiful designs, cat gifs, and all forms of dance.