Collecting customer and user feedback has become a cornerstone for building exceptional products and experiences. However, it’s not without its challenges. Methods like usability testing can generate tons of quality insights, but they’re not always easy to parse out. At UserTesting, we believe that all teams should have access to actionable insights that lead to people-centric solutions. Through smart tags, we’ve made it easier to do just that.
In a nutshell, smart tags are a great way to quickly review large quantities of feedback. With smart tags, you can see suggestions, expectations, and expressions of sentiment, like confusion or delight, directly within the video player, transcript, or written response.
To power this new feature we developed a purpose-built machine learning model to handle our natural language processing (NLP) needs. The goal was to create a model which evaluates feedback in a very human way, based specifically on the types of feedback being collected on our platform.
In this case, we wanted to make sure that our natural language processing approach identified moments that matter specifically for researchers, designers, product managers, or other users of our platform. For this, we used an extremely sophisticated approach called Bidirectional Encoder Representations from Transformers (BERT) which, unlike other NLP techniques, takes into account the context for each occurrence of a given word.
This was crucial for developing the smart tags. If we used models designed to analyze social media posts, earnings calls, transcripts from call centers, or something of the like, the output would likely not provide tags frequently associated with customer experience feedback.
In order to generate smart tags that truly matter to our users, our model was developed by focusing on common observations made by users on our platform. As such, we focused on smart tags that were most often highly correlated to positive, negative, and neutral sentiments:
These labels were selected based on analyzing notes, clips, and highlight reels generated within our platform. We looked for artifacts, evidence, and notes of where our users defined specific areas of interest within videos. We then took those artifacts and clustered them into groups. Some of this was done using machine learning and algorithmic approaches, some were tuned by inferences of internal experts, and all of it was a reflection of the types of tags our customers and internal pros are using already.
Once the labels were selected, we had a data set of over ten thousand examples to feed into our machine learning models. We passed in sections of transcripts that customers or internal pros had labeled with smart tags. We were then able to provide all of these historical examples of smart tags to a machine learning model. That model was able to unpack these sentences and understand what makes a sentence indicate like versus dislike, confusion versus trust, or simply identify when someone is discussing price. The more data available, the more nuanced speech patterns the machine learning can pick up.
After the machine learning models are trained, we can pass in any transcript or written response from a study, and the model is able to predict which smart tag, combination of smart tags, or lack of a smart tag is most appropriate. This is real human-level empathy and synthesis passed through a machine learning model. By training the machine learning model in such a way, it’s able to accurately predict which labels a human would likely apply themselves.
Developing smart tags was a collaborative effort between our marketing, product, research, artificial intelligence, and data engineering teams. The awesome thing about working in artificial intelligence, machine learning, or data engineering at UserTesting is that we have treasure troves of anonymized historical data to learn from.
If you’d like to join the teams working on the next level of machine learning and engineering, check out our careers page.