The Future of Customer Insight for the AI-Native Enterprise

Speaker 1

Thank you, Eric. Let's give him another round of applause. He's my boss. Please, round of applause. Yeah. Thank you.

As Eric said, my name is Baran Urkel. I'm our chief strategy here at User Testing. I'm so excited to be here. Thank you for coming, spending a day or two days with us.

I want to talk a little bit about why we do what we do. Eric talked about what we've heard, what we've done, and where we're headed. My colleagues that are going to come up in about ten minutes are going to talk about what we've built and what we're building, and show you, more importantly. I want to take a few minutes, maybe ten, to talk about why we do what we do, why I'm so excited, and why we need your help.

First, a story.

It's a late evening in June twenty fifteen at a luxury resort in Napa Valley, California.

Two men are sitting around one of these outdoor fire pits. These are two friends, and they're talking about artificial intelligence. But this isn't just any normal Silicon Valley intellectual chat about technology that's so common. These two men run some of the largest technology companies on the planet.

And as the night gets later, the debate between these two friends grows heated.

And one of them shares that he thinks that super intelligent artificial intelligence is going to surpass humanity, supersede even humans as a species.

The other one is horrified by this.

He's horrified that somebody could think that would be a good outcome, and he's horrified mostly that his friend, his friend's company controls the most powerful AI on the planet at the time.

This is a true story.

This conversation led to three things.

One, the end of a friendship.

Two, the founding of OpenAI and the launch of ChatGPT.

And three, to this AI revolution in the current moment we're all living in.

Now, I'm not here to talk about AI, existential risk, the future of humanity. I don't feel qualified to, and happy hour is still like four hours away. That's that's pretty heavy.

But I want to talk about what this means for us.

For this community. For our practice.

Because there's a similar debate, a similar battle going on right now.

And I think the future is going to be decided in the next year or two which way we go.

Is the future of products and experiences going to be human centered? And I think all of us in this room want that future. Or is it going to be machine led?

If you read Twitter, if you listen to the pods, if you read all the AI thought leaders, there's a lot of momentum behind machine led right now.

A future dominated by algorithms, by data, by compute, by token spend, by full automation.

But I think there's a better way.

We're at a crossroads, a fork in this journey, where we can go the way things might might go if we don't fight.

Which is full automation.

Which is full synthetic feedback, full synthetic research.

Or we can impact the future to be more human centered. To be based on human understanding, to be to put humans at the center of products, experiences and content.

And I think that's the future I wanna see. I think that's the future we wanna see.

And I actually think that's the best future for our businesses. That is the winning future as well.

But we don't have tons of time to make this happen.

Things are moving super fast.

The technology is developing at incredibly fast rates. I just saw this morning, there's another model version released.

Tomorrow there will be another one.

All of us right now are rethinking our product development processes. We're rethinking the future of our product experiences, our digital experiences. This is the time to make the future more human. This is our time. This is the moment. But we have to make it so. Now I want to share with you three things that make me believe this, and that make me actually believe that the opportunity for human understanding and human centered design and experiences is actually going to be bigger than ever.

I'll start with the most obvious and then go to maybe what I think is the most interesting and rich as a source of opportunity.

Number one, we'll make more decisions in the next three to five years than we have in the last twenty years.

What does AI allow us to do? Create more. More apps, More experiences. More content. More features.

More more more more more. You probably sound like your CEO. Sounds like my CEO. Every day he's like, more features.

Faster.

We get it, boss. We get it.

But we are going to be be able to build so many things and create so many things, which is amazing. But what does that mean?

Well what does it not mean first of all? What does AI not do? It doesn't allow our customers, our users, to be able to absorb and digest all the things.

Just because we can build ten x or one hundred x more doesn't mean our customers have the attention, the time, the hours in the day to consume and digest ten x more or a hundred x more.

And do they need ten x more or a hundred x more? No. So what happens with this equation of maximal production but limited ability to absorb?

We have to make a lot more decisions.

So many decisions. And the great thing is for us, we, you all are great at making decisions about products and experiences. That's what we do. That's what this practice is. That's how we impact our businesses and move them forward. So point of optimism number one. We are going to have to make a lot more decisions in the future, and that's going to require the need for a lot more human understanding and human insight.

Point number two, Eric made this. You're going to hear this as a theme throughout. My colleagues are going to talk about this as well. When anyone can build anything, the winners will be the ones who know what to build.

Deciding what to build is going to be even more important than it ever was because anybody will be able to copy you. They'll be able to copy your features.

They'll be able to replicate things on your website.

But deciding what to build and building it so that your users, your customers love it and can't let go is going to be the differentiator.

Quick story from this morning actually at six AM. I had a phone call with the CEO of a software company. Founder CEO of a software company in the research space.

Growing very fast. Very excited. And I asked him, why did you get in this space?

And he said, I had a start up before. It was a software company in the data and analytics space.

And he said, and this was before AI, so we spent a lot of time building this product. I mean tens of thousands of hours. I told my engineering team, let's work our butts off, let's build this product, let's go take this hill.

He said, and then the product flopped. Nobody cared.

And I could see the pain on his face. He said, do you know how painful it was to motivate my team for months, for years, and then nobody cared?

I said, yeah.

I was an engineering manager once a long time ago. I don't know what that feels like. It's very painful. And it's very costly. And now, these strategic decisions that we've all helped with and made, helped make in the past, they're going to become even more strategic and even more important because they are going to be the main differentiator. Not can you build, it's what you build.

And my third point of optimism, and I think the biggest, richest area of opportunity for all of us, is that we are going to build things we can't imagine.

The technologies which we're all using, these magic superpowers, they are changing and developing so fast that we know they're going to be different, wildly different, but we have no idea how different. And to me that's super exciting because that means opportunity for what we do. We are about to design and build things that none of us have ever designed and built before. I think this revolution in products and experiences is going to be as big or bigger than the graphical user interface. In nineteen eighty four, the Mac put a GUI and a mouse in front of ordinary humans and let ordinary people use computers. And that changed the world.

We got lots more people using computers. We got lots more applications. We got we got this industry for the last thirty years designing these products, making them usable, making them attractive.

And that was great.

And now, we're talking about a brave new future of intelligent, conversational, hybrid, personalized, dynamic. I don't know. There's going be words that we're going to be using in two years that I'm not saying right now.

They are going to be so different and so personalized that I think we're going to be so, so busy. But we have to tell the world and we have to make it so. Now, you don't have to just believe me on this because I know there are some people, we had a good debate at dinner last night with some customers, there are some people that think, oh, the end is known for the product experience. It's intent based. It's conversational. It's chat. Easy, done, check, chat.

Other people vehemently disagree, definitely don't want that.

And I think none of us know, and I actually think it's going to be much more rich and complicated than that. But you don't have to just believe me.

A few experts in our space, really interesting thought leaders. Victor Yoko, UX researcher at ServiceNow, and he's also an author. He's coming out with a book soon, Designing Agenetic AI Experiences. He says, we are not just designing interfaces, we are architecting relationships.

Think about that.

Architecting relationships. Have we talked about relationships in the past? Yes. But this is at a different level. A relationship with an agentic product is something entirely new. We've done research on this at user testing, and you'll see, we'll put out more and more of this.

But users of these products and experiences now, they feel like they are in a relationship. They feel like they are talking to something with a personality. They imputing personality onto these agents, these chatbots.

Fascinating stuff.

Catherine Wu, the head of product for Claude, Code and CoWork at Anthropic.

She says, the personality is what makes Claude so good at so many things. It's so core to the success of Claude. Personality. I I don't know that we've used that word a ton when we talk about at least b to b SaaS applications. I know that.

Personality. So how yeah. Like how do we design for that? I What a rich vein for us to mine. And we The people in this room, how many we have Raise your hand if you have a PhD in this room.

Okay. I think I see a good number. We have people that understand people at a deep deep level, and we're the best at that in our companies and our organizations. So who better to architect the future than those of us who understand people deeply.

And lastly, Brett Taylor, the cofounder of Sierra AI, a hot AI startup in the customer service agent space. He's also the chairman of OpenAI. He was the co CEO of Salesforce way back He's a legend in Silicon Valley. His resume is too long to recite. He he was one of the creators of Google Maps in like twenty years ago.

He says, we believe that AI agents should be brand ambassadors. So we spend a lot of time training our agents not just on standard operating procedures of your company, but what does it mean to represent your brand.

Brand ambassadors. I haven't thought of our products and software and digital experiences as brand ambassadors before, but now they will They are and they will be. So how do we design that?

So this is the moment that human understanding becomes the most valuable thing. I truly believe that. But we have to make it so.

It's not going to happen by default. But if we make it so, the future is gonna be super interesting. Now how do we make it so?

I think I used the word architect before. Last night at dinner actually, I was sitting next to Angel, who's somewhere hopefully here in the crowd. She was an architect before becoming a UX researcher, which I thought was really serendipitous and very cool.

But we've been amazing advocates. User Advocates for the user. Advocates for the customer in those meetings where product decisions are being made. We should continue to do that. That's so important. But we should also now step up and be architects.

Be architects for those new product experiences that nobody knows what they should be like and how they should feel.

Be architects for the process. Be in the room when the processes are being redesigned on how we build these things. Those are happening right now, those conversations. You should be in those rooms, in those discussions, architecting how human understanding and human insight is integral, not an afterthought, integral to how we build.

And our commitment as user testing is to be your partner on this journey. We're going to use our megaphone and our marketing dollars to get the word out, and we're going to build with you and for you to make this a reality and make this a possibility so that human insight, human understanding is a part of the future, can move at the pace of AI.

And so let's do this together. Thank you.

Now I would love to welcome my colleague, Jennifer Ardebain,

Speaker 2

our head.

Speaker 3

Thanks, Baran.

And thank you all for joining us this afternoon. What Baran painted as a picture for us is the reality I think all of us are living in. And even in our own organization and user testing, I lead our product design and research teams and we are also feeling and seeing the same trends and the same problems all of you are thinking and facing. We're also thinking about how do we make sure we're building the right solutions for our users just like you are and for our customers.

And wanting to make sure that we bring that true voice of the human customer into our process. But before we go into the future and start thinking about all of those things in the future, I wanna give us a quick glimpse into the past. Let's go back for the last year or so, and let me show you and talk about a few of the things that we've delivered over the last year. Eric mentioned a few of these earlier, but I really wanna dive in and and hopefully you are as excited about them as I am.

First, we'll start with our Figma plug in. This has been the first place where we are bringing the embedded experience into the work and the lives of all of you.

We're thinking about how do we make sure that getting that human insight and that human voice fits into your workflow. It's not something you have to leave what you're doing to go outside and have that interview discussion. But instead, for those of you that are designers in the Figma product on a day in, day out, day out basis, just go ahead and launch your test. Create your test. Use the AI test generation, which we'll demo in a few minutes, to build that out. And then also have those insights brought back right into the Figma Canvas so that you and your team can have that information right there and bring all of that new and wonderful information into your designs. Make those iterations, continue to test, and bring that forward to your customers.

But for those of you who are researchers who just had a mini heart attack thinking about every single one of your designers running research and asking biased questions, Don't worry, we have you. We also made sure that we improved some of our administrative controls because that's also important to make sure that the right people are asking the right questions, that we're going through the the right checkpoints.

And so we've improved some of our flexible roles and administrative oversight. Having making sure that we fit into your enterprise organization's ecosystem is another really important piece for us Because governance of the data that you're capturing and making sure that it's being treated appropriately is really important to us. So we are continuing to make sure that our tools have all of those great security and scalability features.

But we also have improved a lot of the interactions and and components that you have as cap excuse me, as capabilities to get that great feedback. Making sure that we have the best and most robust feedback so that we can then incorporate that into our designs or into our products is what is critical for you. Having all of the right ways of asking and getting that feedback is important. Making sure that you are getting it quickly and in again that process of what you're working on is incredibly important. So we're introducing more and more of those capabilities for you. We released our balance comparison functionality with some enhancements. So now you can have four different choices to balance, which again starts make sure you're eliminating that bias in all of your designs and interactions.

We've also improved our skip logic or skipping capabilities, as well as image assets directly in the questions. So while these are not things that you normally would use as a headline in a keynote, they're all those really important features that we know each of you needs in order to make sure you're getting the best feedback.

But that's not all. We've got more coming. We are excited that we are continuing to work on ensuring that we're bringing these embedded experiences, these extremely important capabilities to you. And over the next few months, you'll continue to see more of the functionality such as manual recruitment, better visualization of your results so you have those incredibly dynamic Sankey diagrams to look through and and investigate.

Intercepting your customers on your website has been something we've talked about a lot. And of course, a lot of improvements to our live conversation, scheduling functionality, expanding it to other platforms so that you have all of those capabilities at your fingertips as well. So stay tuned for those things. But before I move off of the future, and we're gonna stay here in the past, I wanna highlight for you one of the most exciting things that we've released in the last few months. Eric mentioned that last week, we formally introduced our MCP or model context protocol, capabilities into the market. We have a few of our customers in an alpha testing that with us now and starting to incorporate it to their various tools.

In this capability, you're able to access user testing from within whatever LLM or chat feature you're using. Whether that's, in this case, an example, Claude or Copilot. You can ask and create your tests right here. Get your audiences set up.

Execute and launch those tests. And then have that test result come back directly to you. So again, embedding it directly in the tools you're already using and in the workflows where you are working today.

But we didn't just launch one MCP. I also wanna mention we launched a second one which was for our user interviews functionality as well. I'm very excited that we brought user interviews into our product family earlier this year. And we also released a user interviews MCP which allows you to create selected audiences for those of you who are using our user interviews product.

Over time, you can imagine that these will come together so that you have all of that functionality in one place as well as even more capabilities from within any of your LLM or chat interfaces.

But none of this matters without the right participants.

So let's just take a moment and think about the participants. Again, I mentioned user interviews. We were absolutely delighted to bring that team into our company and bring those participants into our participant network. They have built some of the best fraud detection capabilities and the best panel that we can imagine.

And we wanna make sure that you have access to those as well. So we are extremely excited to bring that again together with user testing. And one of the things that is launching, as as Eric mentioned, in the next few days is the capability to access that panel from within the user testing application. Again, we'll be continuing to incorporate these over time.

But these panel participants are really robust capabilities of of people. They're different individuals who maybe have more history of expertise, not necessarily just your average general population audience.

Folks that maybe if you're looking for cardiologists, for example, you can access and find so that for your healthcare software, you're really truly getting the right perspective.

And behind every single one of those panelists is a real person who wants to be part of this process with you. Who wants to extend their voice into your feedback so that you can have that in your in your decision making.

But I don't wanna just talk about the panelists and talk about this. I wanna show it you. So I wanna invite Mike McDowell up to the stage to help me show you some of these great features that we've recently released. Mike.

Speaker 4

Alright. Hello everybody. This has been a lot of talk about the panel and getting the right people and trusting who it is that we're hearing from and I'm gonna take you through that right now, show you the integration of user testing and user interviews, really. So right here, we all know this is where you start defining our participants. We go and say, yep, we're gonna actually define a new audience.

We just come down here

Speaker 3

you do that, Mike, how would you have done this before?

Before we had user interviews in here and some of these premium audiences.

Speaker 4

That's actually a really great question. We did it with screener questions. Everything was done with screener questions. And Screener.

You'd write five, six, seven, ten. I've done fourteen for a customer before. And then you launch it. Yeah, exactly.

You launch it out to the general panel and then you hope for the best. Know, we we will always fill it, hopefully, but it can take a long time.

And with these new advanced targeting criteria, we're actually sending the invitations directly to the specific people that you need so that only the right ones are going into the screener questions. And hopefully, that is going to speed up dramatically the time to fill hard to reach niche audiences.

Speaker 3

Awesome. That sounds great. Let's let's get to it.

Speaker 4

Let's do it. So I go in here. I'm gonna select United States here. Pick a country.

Just keep it easy. Come down. We've got our basic criteria right here. We're gonna say add new criteria.

I'm gonna click right into advanced. And let's just say, for the sake of argument, I'm a luxury car manufacturer, and I'm trying to figure out how high powered finance people make car purchases. So I'm gonna come in here and I'm gonna say professional. I'm gonna go to industry.

I'll pick banking. Then I'm gonna come over here and I'm gonna say, you know what? I said high powered. Right?

So I heard a lot of acronyms for Eric over there. So we're gonna say VP and C suite, chief ex officer. And then we're gonna come down and say, you know what? We actually want to know about people who buy cars that maybe they're not in the big cities.

So we want people who work remotely. They're working out. They're out somewhere else. So we'll pick remote and then just for good measure, I'm gonna click into another new option here, products and services, and look at all these options that you can actually start targeting on streaming services.

What bank do you have? What insurance do you have? We're gonna come down here and pick the vehicles that you currently own or lease, and I'm just gonna say BMW or the Mercedes.

Speaker 3

And and I own an Audi, so let's go out of that one in too.

Speaker 4

Jennifer, you are not your customer, so make sure that you're only testing your actual customers. It's a good lesson for all of us, and one hard learned often.

So I'm gonna say apply criteria and we've got our audience defined here on the screen.

So the question now is how do we get these people to actually take the test? Right? We're gonna launch our test.

Speaker 3

Well they, I mean they're high powered individuals. They drive really nice cars. They are not gonna take a test for like ten dollars. Right, Mike?

Like, you're gonna have to pay them at least fifty

Speaker 4

dollars That's a question that comes up a lot.

We can actually come down here now and use your testing, and you actually have the ability to set a custom incentive. So I'm gonna put it to one hundred. But even if I didn't know how much to actually incentivize people with, there's a handy little incentive calculator right over on the side.

Speaker 3

That's awesome. Makes sense. Helps me figure out how much I have to pay them in order to make sure that they're gonna take my test.

Speaker 4

And that's it. That's all there is to it. With the new integration with user interviews and user testing, you will be able to find these niche hard to reach audiences in no time and make sure they're getting compensation that aligns with who they are and their desire to take a test.

Speaker 3

Cool. Well, we've sped up the finding the right audience, getting the right people that you want, but there's so much more to this process that takes time. And as we all just heard Barron talk about and Eric talk about, speed is everything. Making sure that we make all of these decisions fast and that we get process moving as quickly as possible. Let's show them some of the things we've done around test creation.

Speaker 4

This is this is one of my favorite things that we have ever done in my five years that I've worked here. AI test creation. You can see on the list now create test, you your unmoderated, your survey, your live conversations, and then create with AI. So we can go ahead and just press that and we're gonna be able to start creating AI.

Speaker 3

But it's only gonna be like one type of test, right?

Speaker 4

No. Actually it's not. That's what I thought too when we first started rolling this out. And then I actually saw this You always think you start small.

We just went for it all in one shot. So you have Think Out Loud, Interaction Test, or Survey on any device you want. So you basically just select. I will pick Think Out Loud today and we'll go ahead and we'll just jump ahead to a very simple prompt.

So all you have to do is actually just explain what it is you're trying to learn from the test, what your goals are, and user testing will do the rest. If I click into the box, you're gonna notice we actually offer some help options. Like if you need a little guidance around a test goal or some context, you can click those and they'll put in a little framework for you. But I'm just gonna click in here and see that today we're just comparing multiple travel booking sites.

I think a lot of people who know me know I worked in travel for twenty years. So we're comparing different sites. We want to compare other websites, our competitors, to our own. Right?

Competitive analysis, one of the most hidden gems of user testing that people don't take advantage of. And then we give it some questions. How easily can users search for and compare options?

Where do they struggle? Which experiences feel more intuitive or efficient? Once we've got that, we're happy with our prompt. It can be simple or complex, whatever you prefer. Actually, should also have mentioned, if you're interested, you can import a pre written test as well. I know a lot of researchers, you guys like to write your own studies and that's been asked for for a while.

That's That helps a

Speaker 3

lot because I know in my case, I don't know that I'm ready to trust AI completely.

I don't want the robots taking over my life. But if I can import my own test, that would be fantastic. Or but what okay. Maybe I'm willing to try it a little bit. Like if I try it and I still don't like it, what do I do?

Speaker 4

Well let's do this. We're gonna generate it. We're gonna add a URL, we'll generate it, then we'll see what comes back. Okay.

And we'll decide if there's any further action that we need to do. So I'm just gonna add a URL and then I'm gonna say generate my test. And you can see it just takes a few seconds generating the test, and we have our test plan here. Now if the test plan, once we review it, we say you know what, I actually don't like the way this was written or I want to change it, I can just update the prompt and make changes.

Okay. Now the very eagle eyed of you out there may have noticed, not only do we create a test plan, it created a balanced comparison test. You can see group A, group B. So it's not just creating some simple little test.

It's actually creating the test that you need based on what you actually told it. So we're happy with it. We go ahead. We say we're going create our draft.

It moves us into the User Testing Editor. You've got everything here.

If now we decide, you know what, there was one thing that I wanted to add, I want to massage, I can make some edits here or I can just go and add an additional question type. And this is for anyone who has not used the new Think Out Loud test framework. You're going to notice there are a lot of new question types here. QX score, matrix questions, rank order questions.

If you want to figure out, hey, I've just done this balanced comparison, what do people actually pick? Do a rank order question. You can have them figure that out. So very, very easy.

And once you're done with all that, you launch your test.

There's AI insights as well and AI overview of the entire study.

Speaker 3

Awesome. So what we've brought to you today, including this AI summary and the AI test creation, which is also embedded in our Figma plug is available to everyone today.

Our MCP and our premium audiences that we were showing just a minute ago are coming soon or available within the next few weeks. So stay tuned for those. So I'm really excited for what the team has done in the last year, really excited for what we're planning in the next year. But before we leave today, I wanna bring up my colleague, Bobby. Bobby Meixner is gonna talk a little bit about how this all comes together for product and design in a little bit of a of a view. So Bobby?

Speaker 2

Alright.

So there's a lot of talk about how AI driven development has outpaced customer understanding. And in some situations, that might be true up until now. So let's take a look at, in this case, a conceptual conceptual fictional case study about how people just like you, researchers, designers, product owners, builders and creators, work together leveraging human insights and agentic tools to build the next generation of applications and experiences. So to do that, I'm going to introduce you to our fictional organization, ThreadLine.

You can think of them like a restoration hardware. They're a home furnishing, home furniture company, and we're going to pick up with two characters. In this case, we're going to be working with Jordan. Jordan is a UX researcher, but she's not just any old UX researcher.

She's a UX researcher that actually has a lot of responsibility. She is in charge of answering the tough questions for the organization, providing critical insights that help them make decisions on where they need to go strategically, and she works very, very closely with her product team. In this case, it's Maya. Maya is a product owner, she may be a designer.

She falls into that builder creator category that everybody is talking about. So let's take a look at how the two work together.

Alright. So to pick up, we're going to start off as Jordan, the UX researcher. And what do UX researchers love? They love data, right? They love insights, they like lots of it, and they get it from multiple sources. And what we're working with here is essentially a custom application that has been built on top of an LLM that's pulling in data from a variety of different MCP servers and sources, including user testings, to help her answer critical questions around what's going on, not only within their own customer experiences, but within the customer experiences of their competitors, as well as the industry in general. And this is all being fed in from an insight subscription that comes directly from user testing that is being pulled in from agentic insight agents.

So when Jordan comes in here, she can see, again she has information on her channels, she can see what her competitors are doing, and she has quick access to click in and watch video footage of highlight reels with key insights that may help her make decisions. But we know that researchers always have questions, and that's what we love about LLMs, they help answer them. So she comes in and she types, you know, what's changed in our luxury home shoppers behavior over the last two quarters, because she wants to spot some emerging trends before they start showing up in their support ticket volume.

Well, the research insight studio comes back and it comes back some with some really valuable insights that are produced by spanning out across a number of different sources. So it goes through and it looks at insights from user testing. It's looking at web analytics from Amplitude. It's scanning in and looking at service tickets from Zendesk, funnel metrics from Shopify. There's a number of sources and it spots a very very interesting pattern here. So in this case, we can see that mid funnel abandonment is actually on the rise. Now, what's the question that everybody asks when they look at digital analytics?

Why? Why is this happening? And user testing helps uniquely answer that question right here. We can see right in the dashboard that according to historical tests, there's been twenty four mentions of people wondering, will it fit in my room? And we can see that there have been actually eighty nine mentions of scale anxiety, or questions about scale and size within support tickets.

So in this case, this is where Jordan has a judgment call to make. Right? The AI is presenting the data, she uses her judgment to determine how to solve the problem. Is this a design problem or is this a content problem?

It's her call. And she says, you know what, based upon my instinct, I think this is a content or is a design problem. So what she's going to do is she's going to work with Maya, her design partner, to go through and build a prototype, build some concepts to help solve this. So Insight Studio goes through and gathers up all of this information and prepares a nice brief for Maya to use as she's going through and building the prototype.

There's the research question that they want answered, the hypothesis, and any historical insights that they want to leverage in building this new experience, which is essentially an experience where people can upload a photo of their room, see the furniture in the room, and have a better idea of whether it's going to fit in the space.

So now we switch over here as Maya, who is the designer. She's the product owner. And what tool is she working in? Well here, she's working in what we call Design Studio.

In Design Studio, you can think of in this case as an anonymized or a generic version of a Claude design, a Figma make, something of that nature. So when she logs in and she comes into Design Studio, here she can see the details in the brief from Jordan, and she comes in with her prompt. She wants to design that visual for shopping assistant to help solve this problem. She can take a look at all of the data coming over from the brief that Jordan put together, and Design Studio says, hold on a second.

We're not about to go through and start cranking out some AI slop. So let's make sure that we are leveraging all of the insights that we have historical access to. So this once again goes and fans out across the stack to pull design specific insights that should be incorporated into the first pass of this design. So we're producing better outputs, we're producing better outputs faster, we're reducing our token usage, and we're eliminating all of that rework that needs to happen when we get AI slop served on our plates, and we've we've all been there.

So this is very interesting to Maya because she has an opportunity here to dig in a little bit deeper and try to understand what customers want in the design, right? So there's some themes that have come up here. So in this case, scale anxiety has been dominating the journey, customers are abandoning mid flow because they're not sure if it's gonna fit in their room. They have luxury shoppers that describe rooms in certain adjectives.

Over recommending starts to feel pushy to this audience, so we want to consider that, and the maker, the origin, the materials, these are details that are things that can only be discovered through very detailed, think out loud qualitative feedback that's collected from user testing, and now incorporated, potentially incorporated in this case into the design. And if she wants to drill into any highlight reels, video feedback to just get a better sense of what folks are talking about, she can do this. But this is really great. She thinks this is gonna work for her.

So she comes over and says, I got it. Let's design around all four of these key themes. We wanna make sure that we're reducing scale anxiety, enabling shopping by mood, and a few others. So Design Studio gets to work and starts cranking out prototypes.

So we have iteration one, and this is looking pretty good. We've got a spot to upload a photo. There's four items that are being presented here. But Maya is like, hey, you know, before we put this in front of real people, or even test with real participants, let's leverage our historical insight, as well as information that we have coming in from our digital analytics solution to provide some synthetic, quick, fast feedback in order to gut check whether or not we think this is going to work based upon what we already know about our customers.

So here we can see that the predicted task success is up, but scale anxiety is still only partially resolved. And there's a few other friction areas that we're a little bit worried about, and I can drill in again and look at highlight reels from historical sessions to bring that feedback to life, and to better understand what is ultimately driving the synthetic feedback that has come back so quickly.

Okay. Well let's keep going here. So iteration two, let's lead with mood. We have some new filters on the top, and we've scaled down some of the recommendations.

Now we're only getting three, and Maya's like, okay yeah, let's let's take a look at synthetic again. We've we can see some of the metrics have improved. This is looking good, but actually you know what? We've uncovered a new issue.

In this case, three recommendations kinda reads as a menu to these folks. They may not like that, so we're going to need to make another adjustment. So in this case, we're gonna go through and produce yet one other iteration, and Maya looks at it and she's like, you know this feels ready. Let's actually go out and get feedback from real luxury shoppers.

Now remember Maya is a designer. She's a product owner. She's not necessarily a trained researcher. She may know generally what she's looking for and who she wants feedback from, but she may not necessarily know everything about designing a test and conducting research.

She doesn't necessarily have to in this case. You can see in the prompt, she can describe who she's looking for. She's looking for people who actually spend on heirloom pieces in their homes, not just aspirational browsers. Right? And she's she's talking about what she wants to uncover.

So in this case, Design Studio is interacting with the user testing m c p. It's going through and defining an audience, creating some screener questions, going through and establishing a list of tasks that need to be performed. But this is a niche audience. Right? They have special needs. We need to make sure that we're compensating them correctly.

So before we just launch this test, let's send it back over, in this case to Jordan who's working back in Insight Studio here to take a look and make sure that the feasibility is holding. The incentive is going to work. All of that looks like it's checking out. Jordan has the ability to take a look at how long it's going to take to actually get results. In this case, we're gonna get results in a few hours. If she wants to manually go through and accept people into the test, she has the ability to do that. But this is looking good, and she's gonna go ahead and approve it and launch it, and Insight Studio goes through and launches this test.

Okay. So the test is launching. What's happening now? We're back over here in Design Studio as Maya, and she's working away, right? But she wants to understand what's happening.

What what feedback does she need to understand so she knows what she needs to adjust. So in this case, she's back over in Design Studio, she can see the feedback is coming in, she can drop in and take a look at some of the unmoderated sessions that are happening. She can drill into some of the ones that are complete. She can see what has yet to be scheduled and occur.

All of this is available here, and Design Studio is interacting with user testing to synthesize and surface these insights as they come in. Now, once they come in, we have four concrete changes that should be made to this design based upon the feedback. Things are looking good, we have a pretty good q x score, but there's some recommendations that should be included here, and again access to a highlight reel to better understand why these recommendations are being made. So in this case, fifteen out of twenty participants were pausing at the recommendation because they couldn't still necessarily visualize the furniture in their room.

Some folks are asking, who made this?

There's a few other tweaks we need to make. Maya reviews those, again using her judgment and taste as a designer to go through and say, accept all four, ship this out, let's go ahead and make these changes. Design Studio makes these changes. She can see those here, and everything's looking good.

Alright. So at this point, the the the hard work has been done. But Jordan is like, you know, we actually got some really good feedback from these people, and these are probably people that we want to talk to again and perhaps include in a longitudinal study. So in this case, she's working with Design Studio to capture these highly valuable participants and incorporate them into a custom panel that she can leverage for future research. And in this case, it's also going through and suggesting additional tests that can be run-in a longitudinal fashion to continue improving and iterating this experience.

If it were only this easy though, if it was only up to just these two folks in terms of the experience that needs to happen. Right?

We know that's not the case. Everybody has stakeholders that they need buy in from.

We're helping with that too. So in this case, Jordan prompts Design Studio to go through, or Insight Studio in this case, to go through and prepare a stakeholder report, because they have a review meeting with marketing, and product, and development to go through and make sure that everybody knows what decision they made. They're shipping that single piece room photo recommendation. There are highlight reels with very detailed evidence that really help win stakeholders over when they understand the emotional responses that people have to these experiences. And then they've also included three things that they're watching out for post ship, so that they're continuing to constantly improve this experience.

So everything moves forward, it launches, things are good for a while, and now three weeks later they're like, you know what, we gotta make some more changes. We need to up the ante a little bit. Let's make these recommendations even better. Perhaps the original recommendations were built on an off the shelf recommendations engine, and ThreadLine wants to truly differentiate their experience and provide much more detailed recommendations, and in order to do that, they need to develop their own AI model that is very specific to their business and their customer base, and they're going to launch a new portion of their app, ThreadLine Atelier.

How does this happen?

Well, we pick back up with Jordan, the researcher. She is fresh out of a meeting with the product team and learns this information that they need to up the ante with their AI design room feature.

The product team owns that model, but they know that they need to train the model based upon insights and feedback from a very specific audience. In this case, she wants to actually pull residential interior designers that have at least five years of experience in published portfolios to help train this model. Wants to be able to capture the intellectual property and AI training consent when they sign up, and she wants to persist that specific cohort of people within Research Hub in a custom panel, so these interior designers can be used over the course of the twelve week project that is going to kick off to train this model.

And Insight Studio says, I'm on it. Off we go, standing up the panel right now. Let's get to it.

So it goes through based upon the query, runs a filter, finds the right audiences. In this case, there are nearly nine thousand folks that match this criteria in the panel. They've vetted two hundred and forty seven of them, and a hundred ninety three have opted in to participate in this. We have a summary of the criteria, we can see a preview of some of the members here, and these folks are going to be participating in a variety of feedback sessions to train the model.

And that's what kicks off next. So Jordan comes in and says, go ahead and configure the training loop, where we want these people to do preference comparisons, taking a look at the outputs of these models on a weekly basis. We want them to also go through and provide narrative critiques around the output, and there are certain scenarios where we actually want the designers to demonstrate to the AI model what they would build in the situation for a particular customer. And all of that is going to be managed by not only Jordan, but the team of testing agents and insight agents that she's standing up with the help of user testing to make this all happen.

So this becomes activated, this is good to go, the test starts running, and now six weeks have passed. She wants to check-in. Where are we? Well, they've captured nearly fifty thousand evaluations from this group.

The quality of the outputs based upon the q x score we can see is rising steadily. Everything is looking good, and time goes on. They continue going through and doing these tests, and at some point Jordan says, let's get Maya back in the loop. We have our model, let's integrate it into the app.

All is good, and what we end up with is a beautiful looking app that's producing much more immersive, much more relevant, much more personalized recommendations, very very specific to the thread line audience based upon a custom model that's been trained by a team of experts leveraging the user interviews panel.

Alright. So wrapping up, right? All of this is enabled by, we can go back to the demo for one second, by three things. The three things that we've been talking about today.

None of this matters if you're not talking to the right people. Right? It's a trusted panel of real vetted high quality participants to provide the right level of feedback, and to do that quickly. Number two, it's the user testing platform. The biggest, baddest platform out there to allow you to answer the toughest questions in a way, in in ways that meet the needs of your business.

And number three, to be able to scale that leveraging agentic agents to go out and help you collect the insights and determine value from those insights, all within areas and applications that you work in on a regular basis.

So, going to the slide now.

This is a change, right? This is a change in how people work. It's a change in the applications that they work in, and we're here to guide you. So I want to draw your attention to a newly released guide around the changing product development lifecycle.

There's a QR code. We'll happily send you this as a follow-up, But this is a huge topic of conversation within audiences like this, right? How is the product development lifecycle changing? How are the roles evolving?

There's lots of blurred lines. This is a great guide to give you some of that guidance. So ultimately, when you are working with your research partners, your design partners, you have the ability to know that when AI allows you to build anything, knowing what to build is everything, and you have the right tools to do it. Thank you very much.