It wasn’t that long ago that artificial intelligence (AI) was a futuristic concept. Now, computer systems can complete tasks that historically required human intelligence. Smart home systems like Amazon Echo and intelligent personal assistants like Siri can understand your commands and complete tasks for you. You don't even need a partner to play games anymore—computer systems, like DeepMind’s AlphaMind Zero for the game Go, can be your counterpart in multi-player games. While these advancements are exciting, they also introduce new considerations when trying to deliver the best customer experience. Before the use of AI was widespread in technology, experiences on digital platforms were rigid. The customer’s input was dictated by the platform, and therefore the output was always pre-determined. AI revolutionized this model—now the customer dictates the input, and the computer system uses this input to provide a relevant output. In addition to these big changes, AI interfaces aren’t always tangible, which introduces a new interface with its own challenges for customers.
AI would not be possible without copious amounts of big data to inform its responses. However, it is critical to also incorporate qualitative, human insights into AI development as emotional understanding and emulation are key to making AI-driven experiences successful. We recently ran a study on smart home systems to evaluate the experience of using these systems to accomplish some common tasks. Smart home customers were asked to create a shopping list, play music, and plan a movie outing. Here's how non-tangible AI systems can provide an exceptional experience:
With tangible interfaces, the input or action is clear. But with AI systems, verbal commands are not obvious and need to be defined or risk providing a poor customer experience. For example, the success of adding multiple items to a list using a smart home system depended on how the customer gave the command:
To mitigate this confusion, instructions on how customers should construct their commands to smart home systems would help them accomplish their task faster.
Most AI systems have “stop” and “go” commands that allow customers to start and end interactions with the system. But, we found that undoing actions was challenging because it couldn’t be done using the same verbal command “language.” Adding items to a shopping list on a smart home device was easily done through voice commands; however, when attempting to remove items from the list using voice commands, customers were frustrated to learn that they couldn’t. They needed to use the smart home system’s app to remove items from their list. This hurdle was frustrating because customers expected to edit their shopping list the same way they created it—using verbal commands instead of an entirely different device.
Just like in human-to-human conversations, back-and-forth interaction between the AI system and customer is required to complete some tasks. Take, for example, planning a trip to the movies. There are many factors that go into seeing a movie in theaters: selecting the showtime, theater, movie, etc. Some smart home systems listed out the movies that were playing at a theater close by but were unable to take it a step further by letting the customer know specific times that movies were showing. This information was essential for customers to make plans. Since the smart home system did not provide a means for customers to complete their planning, they needed to use another device to complete the activity independently.
If your smart home system directs customers to a mobile app, the transition to another device to complete the activity should be seamless. Our study found that experiences became fragmented when customers needed to use another device to complete their task:
Testing and validating your AI system isn’t always as simple as sharing a URL and providing some sample tasks; and only referencing your analytics will provide you with what is happening but won’t provide you with the full picture of why it's happening. Here are three techniques to keep in mind when running remote, unmoderated studies on your AI system.
Communicating the right expectations ahead of tasks and questions will enable customers to test your AI experience effectively. Sometimes, you will need to provide a more thorough introduction versus just making sure customers are in the right mindset. For example, if you are testing a prototype of your AI experience, make sure you specify the commands that customers need to provide. If you set parameters that are too broad, the AI system may not respond correctly, and the feedback you receive won’t be actionable. Or, if you are testing how well the AI system responds to certain emotions, you should make it clear to the customers that they should be “feeling” that way.
Something that might be deemed “successful” in your analytics might not be perceived as such by customers. For example, the AI system is successful when it produces the correct answer to a specific command, or when it provides the customer the tools to complete an activity. However, if to receive the right response, the customer needed to provide more commands than they expected, then the interaction might not be viewed as a success by the customer. Understanding and aligning with your customers’ perception of success via human insights will enable your AI system to create a better customer experience.
With AI systems, success can also be perceived through the tone of the customer response—characteristics that can’t be easily measured through metrics. To work effectively, AI systems need to also understand and empathize with your human customers. Use qualitative studies to understand if the AI experience is responding correctly to emotional cues. Does the system react appropriately to your customers’ frustration, sadness, or other negative feelings? Identifying points of misalignment between the AI system and your customers allows you to make informed optimizations, whether it is to change the response or to redirect to a human operator.
If you’d like to learn more about how UserTesting can help you understand your customers through on-demand human insights, contact us here.
Get our best human insight resources delivered right to your inbox every month. As a bonus, we'll send you our latest industry report: When business is human, insights drive innovation.