Designing with Confidence: Customer Insight for the AI-Native Enterprise

Speaker 1

My teammate Bobby now is going to take us back to some of those key moments throughout history where we've seen similar things, and how we can apply them to the AI era. Please welcome Bobby.

Speaker 2

Hi, Alright. Thank you. Thanks a lot.

Alright. How's everybody doing? Good afternoon. Okay.

So what we're gonna do is I'm going to start us off with a bit of an extreme throwback Thursday. Let's go back, all the way back to 06/27/1967, North London. There's a small crowd gathering outside of a Barclays Bank because something a little bit strange was about to happen.

A man by the name of Reg Varney, he's a sitcom actor, he's wearing a cardigan, he's got a golf cap on, he walks up to a wall, slides a piece of paper into a drawer, enters a four digit pin, out comes 10 pounds. What am I talking about? Right?

The first ATM. Right? There was no teller, no signature, no person on the other side making sure he is who he says he is.

Really just a voucher, a piece of paper, which was, by the way, mildly radioactive so the machine could read it.

When you think about this moment, somebody had to go through and actually design this moment.

And before they went through and did that, guess what they had to do? Somebody had to go through and actually defend that moment. Somebody had to sit in a room full of likely fairly conservative bankers and say, you know what? People will walk up to a wall, put a piece of radioactive paper into a slot, and trust that it's going to give them their money, and here is exactly how I know.

2007.

Somebody had to convince a company, and essentially the world, that people would carry the entire internet around in their pockets on a piece of glass the size of a deck of playing cards.

Let's go a little bit further. 2010, somebody had to defend the idea that people would get into each other's cars willingly, actually do it on purpose based upon a rating and a name.

And somebody had to argue that people would decide who to love or at least who to meet up with based upon a swipe of their thumb. Right? When we think about this, what do all of these moments have in common?

They didn't start with technology. Right? They started with somebody who understood people deeply enough to imagine what they did not yet know they needed, and they had the ability to defend that vision until it actually existed. That person was a designer. That person was a researcher. That person is somebody in a room just like this one.

When you think about it though, the job is actually the same every time. And it's really to understand how people think, how people feel, what they truly need, and to defend that to the people who haven't necessarily seen it yet.

That's the craft. Right? It's older than any of the tools that we use, and it's really what brought those first time moments into the world.

But we know that everything is changing. Right?

These moments that I just walked you through, they took decades to build. Decades of research, of arguments, of rewrites, of networks, of teams. There's a lot that goes into all of that, and that's just not necessarily the timeline that everybody can operate under anymore.

Because we have tools like these. Right? Tools like these, dozens of, and other tools like them, mean that a single person can build just about anything. Right? A founder with no engineers. A PM with no designer, a designer with no dev team, and really for the first time in history, a single person can build just about anything.

Which, when you think about it, sounds pretty amazing until you reflect upon what were all of those people doing together before. Right? They would pressure test each other. They would challenge each other's assumptions. They would wait and ask, is anybody actually going to use this? Right?

And all of that is happening really really fast right now. So, we know that the question isn't, can we build faster? We know that we can, and we are, and we're all doing that. The question is really the same one that the designer of the ATM had to answer, that the Uber designer had to answer.

Right? And that really is, can I trust this? Can I defend this? Do I feel confident in this decision?

So what we thought we would do is do one of the things that we do best. We thought we would do some research. So a few months ago, we released the Defensible Design in the Age of AI report. And we went out to 183 designers across The UK, France, Germany, and The US, and we wanted to know how AI is really changing their work, and more importantly, how is it actually going through and changing the confidence that they have in their work.

So, let's take a look at what we found.

So this first statistic, probably not necessarily surprising, ninety one percent of people felt that they were moving faster. But when you ask them about confidence, only fifteen percent said that they feel much more confident in the quality of their outcomes. When you look at that, that's a six to one gap between how fast we're moving and how confident we are around where we're going. When you think about the designer of the ATM back in 1967, they were slow. They were painfully slow, but when they stood up in front of their board to present their decisions, they knew that they had done the work. And when we look at this data, it seems like we may have flipped it. We're moving fast, but we're not necessarily sure that we're going in the right direction.

When we take a deeper look at confidence and we break it down by stage, the story starts to get a little sharper. When you look at the earlier stages around discovery and ideation, confidence is higher.

4.1 out of five. We know that AI is a great brainstorming partner, great synthesizer, it solved the blank page problem. But when we start to get further down the cycle, right, when we start to get closer to the final call where we're actually spending the money, people are actually going to use this, we start to see confidence drop to a 3.6,

3.7. So that's essentially the reversibility factor. Confidence holds early on when things are easy and cheaper to undo, and the further we get down the cycle, they start to collapse when we're not. And when you think about it, right, I mean, getting into a stranger's car, yeah, that's pretty irreversible.

Right? Somebody had to be really, really sure.

Let's take a look at one of the more uncomfortable statistics in the report. Sixty five percent of designers say that they can't confidently say that their outcomes are better because of AI. When we break that down even further, thirty percent say the results are maybe about the same, and another 19% say that the results are sometimes even a little worse. So, we're moving faster than before, but when we think about the products and the outcomes by our own reporting, sometimes it's best unchanged, or unchanged at best.

So then, if things aren't changing, if things aren't going in the right direction, the question is, who's ultimately responsible? This was actually another very interesting spread here. So, we know that speed isn't necessarily delivering the outcomes, and when it comes to who's responsible for these decisions, it's really fairly split. 27% say design leaders, 20% product managers, 17% goes to leadership, 16% individual designers, and 7% are like, we're not even really sure.

So then it leads you to ask the question, is this risky? Are we going in the right direction? And you would expect people to think so, but designers were actually evenly split on this.

And so you've got 37% saying that risk has increased, or 38% say risk has increased, 38% think it has actually decreased. What's more interesting is what separates those two groups. When you look at the group that says risk increased, what are they doing or what are they not doing? Right?

They validate less. They're getting their information based upon peer discussions and critique, or they're not validating at all. Those who feel like risk actually decreased, well, they validate more. They cross check with their own research.

They're doing primary research. They're doing peer reviews. They're taking a look at their analytics and their qualitative data. So, they're using the same tools, it's the same AI, but they're getting different outcomes and they see it differently.

So, the question is, what do we do next? According to the design leaders that we spoke to, there's numerous things here. I'm not going to go into each one individually, but I'll cover the last one. And that really is to protect human judgment. Right? Keeping designers doing the things that they're good at. Their talent, their craft.

And how do we hone that? How do we hone our craft? We stay informed with what our customers think, feel, and need. And in order to really leverage that, that feedback, those insights, they need to be fast, they need to be insights that you can trust, and they need to be embedded exactly where you're already doing all of your work.

So to tell you how we can help you accomplish that, I'd like to introduce our vice president of product management, Jennifer Arderbain.

Jennifer?

Speaker 3

Thank you all. So I've been thinking recently. I woke up not too long ago and realized it's September. Does anyone else besides me feel like 2026 has just gone by in a blur? I remember back in like January and February, everyone was talking about AI and it was still kind of what is going on and what are we doing and which tools do we have access to. And we're all trying to figure out what are we going do with our teams and all the rest of it. And here we are, nine months later or two weeks in my brain.

And all of a sudden, this is all we're talking about all the time and we're still all trying to figure out what does it mean. We're all still trying to figure out how does it work, how do I fit it into my teams, how does it change my role, how does it change my job.

And our team, my product team and the product team within user testing, which includes of course our design, our writing, our research, and our engineering organization are transitioning just like all of your companies are, just like all of your teams are, trying to figure out how to make it work in a way that works for us. We wanna do the same things all of you wanna understand our customers so we know what we're building.

So we've been working very hard over the last year to try and make sure that we can bring you a lot of great innovation, a lot of wonderful things that will hopefully resonate with you and your teams as you're going through this transition.

So, let me tell you a lot of really cool stuff and hopefully you're as excited about this as I am.

So, we're going to go through, to start with, some things that are available now. Hopefully, you've had a chance to try some of these things yourself, work with your teams. If you haven't, surprise, there's some new stuff in the application. Give it a try.

We'd love to help you out. And I'm going to do this in sort of the same way that Bobby was talking about. First, talking about fast. Fast, trusted, and embedded.

Again, this AI era that's upon us is making everything feel like it's at warp speed, whether that's January to September or the beginning of the day to the end of the day.

And you're trying to make sure that you're doing everything that you can, both as quickly as you can, but as thoughtfully as you can.

What we've introduced in a number of different places is AI within the workflow in user testing to help streamline some of the tasks that you were doing day in day out, or are still doing day in day out. And, you know, again, speed that up for you and your teams. We've introduced things like our AI test creation, where you can simply use a prompt to give us some general information of what you're trying to accomplish in your test, and we'll help you generate that study, and then help you, of course, get it launched to whomever it is that you're looking to get feedback from.

In addition, we're helping you on this on the other side, once the results come in, to understand what you're hearing and what's being said. This doesn't mean you can't sit down with the raw results and comb through it and read all of it and watch all the videos. Some people love that and I love that too. But when you're trying to get a quick understanding of, did I get the information I'm looking for? Where should I jump to within some of these transcripts to understand what maybe I need to dig into a little bit more? We've introduced a lot of places where you can now have more AI summaries on your results, both at the task level as well as the study level.

Both of these innovations are just here to help you. Again, you always have the capability to go in and get in underneath the covers and do it all yourself, and kind of adjust it as needed. But where it helps you speed things up and bring other team members on board, we want to help you do that as well.

Some other places where we've improved the ability to capture the information you want is within the study itself. So, we've brought a bunch of capabilities into the test building functionality across all of our different tests, including things like embedded images that allow you to make sure that you're asking the right questions and getting information.

We've also updated our results experience, so that you have more information about what's actually happening as people are going through your test. These experiences that you see on the screen here are available right now for our Figma task, as well as, it will be expanded in the next few months for navigation tasks.

So, really excited about that.

In addition, we want to help make sure that what you're doing with the product, excuse me, with the product and with your customers is trusted. We want to be a trusted partner of yours and you want to be a trusted partner of your customers.

And we have the foundation of the 7,000,000 participants globally that Eric talked about, to help you find the right answers to the questions that you're trying to ask. So, that you can be sure that the that the problems you're solving are correct and where you're going is correct. We of course are super happy that we've brought the user interviews organization into our families.

We when we acquired them earlier this year, we've been integrating their panel together with our capabilities over the past year to bring you more functionality and more reach.

Including things like making it easier for you to find some of those hard to reach individuals, or individuals that maybe have a particular piece of information or a particular part of experience that you need to tap into. We can help make sure that you're able to find them.

And specifically, we've introduced, and this make sure it goes there. This video here is explaining how we've introduced our new advanced targeting capability. So, this is tapping into that broad group of higher hard to reach audiences or b to b audiences, giving you the ability to choose exactly whom it is that you're looking to find. In this case, we're looking for senior individuals within the financial services industry, who drive a certain type of car, because that's maybe a particular persona that we're trying to go after to better understand.

We also want to make sure they're working remotely.

In any case, all of this new functionality is available for you without having to write a ton of screener questions within our advanced targeting module.

A note on that, of course, what you're seeing here on the screen does reference factors within The United States. It will be extending to our international panelists in the next few weeks. So, we're really excited to also bring that to you here in our international audiences.

I'll let this finish up for just a second, as we continue to find exactly who we want. And once we've applied these criteria, we also have the ability to decide if what we're going to incentivize them with is correct, or is the right amount. Now, I've looked, I'm looking for these particularly senior individuals in financial services. It might not be that they want to only receive $25 to answer a survey. I might need to incentivize them a little bit higher in order to make that happen. And again, with advanced targeting, you have the ability to increase that incentive and ensure that you're going to get the responses that you're looking for.

We've also added things like our flexible roles, or the ability within your administration to choose who within your organization has access to which functionality. So, that as your team is expanding, as you are bringing this functionality and this capability to others within your product organization, you can ensure that your product managers aren't going to go off and build a whole bunch of tests without maybe research oversight or somewhere in the middle. So, ensuring that everyone has the right level of capability.

We've also ensured that you have access to the people that you want to talk to, that maybe you've talked to before. So, you have some of those participants that are favorites of yours, or that you've had complete surveys or studies for you in the past, and you want to bring them back.

And you have the ability to include both favorite audiences, which is a criteria or a set of criteria, going back to like my individuals with certain sports cars, but also certain people that maybe I want to say, you know, I really want to hear from Jennifer again.

Can I make sure that she's in the group that we're going to target?

Some other functionality that we're adding to make sure that you have the ability to really get the information you need, is around our balanced comparison and skip logic, and more functionality like this is coming over the next few months.

Lastly, I want to end on embedded. And again, everything I'm telling you about right now is in the application today, is already there for you to use, and with advanced targeting, again, coming from international in the next few weeks. But we also have been embedding user testing elsewhere, because we know it's not always gonna make sense for your teams to always be in user testing. You have designers who are working in Figma, who are looking to create their tests directly from the Figma canvas, launch it directly from Figma, and get their results back in Figma, so they don't ever have to leave that interface, and instead, are able to, again, still get that information that they need, tap into their customer, and better understand that.

And we've launched an MCP. We've actually launched two MCP's, both our user interviews MCP and our user testing MCP, that allows you to interact with the user testing application from your favorite LLM client, whatever it might be, Claude, ChatGPT, and the like. And so, we're very excited about all of the innovation that's been delivered over the last year. It has felt like a whirlwind for us. I hope that it's also been something that that you've been able to experience.

But before I let you go, I've got more to tell you. So, we've got a whole September release that's launching this month. As Eric mentioned, we moved from quarterly releases to monthly releases. And quite frankly, what that really means is we're just releasing code and releasing functionality as we have it available, so that you're getting the benefit of that. So, the things that I'm going to talk about here, some of these may have already gone out the door and some of them are still coming over the next fifteen days, But you'll see a bunch of things that we're excited about as well.

We are closer than ever to helping you automate the entire workflow behind the scenes of your your insights process.

It's possible today with our MCP and our AI moderation capability that I'm gonna show you here in a few minutes, to build the capabilities for everything that you might want from your own workflows.

You can have your agents talking to our MCP to help set up those tests. Then the test is launched to the audiences that you need and want. The responses are captured and again, results are brought back to your system, wherever that might be, or to your agent, to then help feed that and continue that loop forward. There's This helps increase the streamlinedness of the workflow, it does not take the human out of the loop.

This absolutely is not intended for there to not still be someone checking or being there along the way. We expect that. I expect that of my team. Don't want to know that there's just a whole bunch of tests going on forever, but at the same time, it does make it a whole lot easier for my team to be continually getting the information they need and bringing that into our product development process.

So, instead of hearing my voice for the next few minutes, I'd love to have you guys listen to a few videos. And I'm going to start with one about AI moderation. In this case, what's going to be releasing later this month is the ability for an AI moderator to probe further on questions and get hopefully deeper answers to some of your problems. So let's listen to Andrew, our product manager, and in this case, stand in, participant.

Speaker 4

What specific detail in the cart showed you that the sofa was added successfully?

Speaker 5

I can see here that it's got the total, cost of the sofa. I can see that the shopping cart clearly shows that the ThreadLine modular sofa has been added, and I'm able to remove it if I want to, and I've got a very clear kind of checkout button. So to me, it feels very, very clear that the sofa has been added. And one thing that's maybe not clear, though, is that I can't see if there are any additional costs beyond shipping, such as taxes.

Speaker 4

How does not seeing those additional tax costs affect your overall confidence in checking out?

Speaker 5

It makes me a little bit less certain that the cost that I'm being shown here is the actual cost. And, unfortunately, I've had some scenarios where companies hit those additional costs before, I got to the part where I actually completed the checkout and got an invoice. And so that just makes me a little bit less certain. The number that I'm seeing here, this $1,890, is actually what it will cost me.

Speaker 4

Where in the checkout process would you ideally expect to see those additional costs clarified?

Speaker 5

I think it needs to be present on the, like, maybe, like, a customer information or support page that exists that just clarifies where those costs would be. I also think, ideally, it would be on this shopping cart. Right? There should be an additional line here underneath shipping, which is tax and just shows, like, if there are additional taxes applied and any service fees as well. I also know that some ecommerce websites, they like to allow you to, add as many, sorry, and ensure that someone can come and help you build it, and that's not included in the cost here either.

Speaker 4

That's all for this tech.

Speaker 3

What's amazing about that besides the fact that I think Andrew is a fantastic Think Out Loud participant, you probably all wanna add him to your favorite groups, is that the AI moderator was able to respond to some of the things he was saying and make sure that it probed deeper. You'll have the capability within the user testing test setup to choose how deep or how much you want it to to probe and ask additional questions. So, if it's not something you want, or you wanted to ask specific types of questions, you can let it know that as well. We're very excited about that, again, to help you ensure that you're getting the information you need.

I also mentioned our MCP, which we launched earlier this year, both for user interviews and user testing. And one of the pieces of feedback that we received, and we've been continuing to add to both MCP's, is that we needed access to more of the raw information and raw results that's available. So, in this example here, we are going through the typical process of talking to Claude to create our test, and now I'm looking back at those results to better understand what I received and what's possible. And of course, directly to those user testing tests and clips, so that I can go right to it if I need to find that information.

In addition, it can help you think further about your test. Again, maybe you want to launch a follow-up test.

So, that process continues to improve and streamline, but there's still your your team, your product managers, your designers, your researchers, whomever within your builder organization who's interacting with the the system and with user testing through whatever method they want.

And speaking of people interacting in whatever method they want, I think one of the most interesting things that's going to launch later this month is a new and updated extension list browser experience for participants.

We know this is a pain point for many folks who are particularly trying to work with particular sets of restricted audiences that maybe don't have the ability to download user testing

Speaker 6

user testing session.

For most participants, that's a speed bump. However, for someone on a lockdown machine, this is a block and the session ends it even had the chance to begin. So we built this, a brand new setting in the test builder called testing without a browser extension. It's off by default.

However, when you turn it on, there is nothing for your participant to install or download to get stuck into the session. Here's what your participant sees. They open the session, allow the webcam and microphone to record, and share their screen, and then the test begins. From there, the experience is identical to what they're already doing today.

Written questions, rating scales, it's all the same behavior, so nothing about how you write the test has to change. The part that does change is how the Figma navigation task look, but it changes for the better. Today, a participant is taking part in a user testing classic think out loud test and they're on a navigation task, they have to work between two tabs, the question in one and the website on the other. They read the question, switch tabs, do the thing, and then come back to answer.

That switching is a genuine friction point, and it's really easy to lose your place on what you're thinking. With the new experience, those tasks move into a floating taskbar that stays on top of the page that they're viewing. Therefore, we've removed the need to jump between tabs to remember what you've been asked. So when should you use it?

Well, if you're testing with participants on lockdown devices or who use Edge or Chrome, then this is the setting for you. Or if your test doesn't include a navigation task, we suggest turning on, as there's no trade off. It's simply one less step between your participant and the session. The one thing that the extension still does that this doesn't is track page views and button clicks on a website.

So if your test depends on that, stick with the extension. As a small bonus, if you can't install the extension either, you can enable this setting to preview the test before you send it to participants. Testing without browser extension for think out loud and interaction tests on desktop, Edge, and Chrome.

Speaker 3

I think we should all hire Calum to be our next our next marketing director.

And and, you know, but he's he's so excited and we all are so excited about what this can open up for so many folks.

But I'm not done yet, guys. This is still September, and these all are coming out this month, as is a new live conversation experience. So, really excited about this capability as well. We're improving the live conversation capability within user testing, making it easier to start an instant live conversation or send a particular link to folks, as well as including those AI summaries that we were talking about earlier.

All of that same functionality and ease of use that we have earlier in our conversation to make sure that your workflows are improved comes back here with live conversation as well.

Okay. That's a lot.

We're still working on more. The team is working at the at a high rate of of speed these days, trying to make sure that we're covering all of the things that we know are important to you and your organizations to best reach your customers. Some highlights of things that are coming in the next few months include manual recruitment, which is of course really important for anyone who needs that two step recruitment or being able to confirm whom you're going to talk to.

The ability to intercept on your own website and start a study.

That improved results visualization that I showed before will extend to both our Figma task as well as our navigation task. And we'll be extending our language support so that we can ensure that you have the ability to launch tests in any language and receive the responses in those languages as well.

So, we've been working on a lot of things, we've been talking about a lot of stuff, but it might be really hard to sort of think about how this all comes together. So, I'm gonna do a quick demo of what this could look like over the next few months maybe even, as we think about how all of these different enhancements come together in your organization, as well as maybe a few nods to some things we're thinking about for the future.

So, in this example, I'm going to use three individuals, a team together that are working within their organization to build better products and understand their customers. Priya is our product manager, Devin is our designer, and Marcus is our researcher.

As Eric said, when AI can build anything, knowing what to build is everything. And so, this team needs to ensure that they get all of the information they possibly can to ensure that they can prioritize what they're gonna focus on.

In particular, in our scenario today, they are all working for a financial services company Meridian. Meridian recently launched a new chat bot interface within their digital experience for their customers to help with financial advice. To give in in suggestions on what they might want to do. And Priya, Devin and Marcus are going to dig into this, each from their own perspective, as they've been also seeing and wanting to improve this.

So, as we showed just a minute ago, Priya is going to start here in her favorite LLM, looks a bit like Claude, and go ahead and ask it to help her better understand what she's seeing. She's noticing that her customers are getting a recommendation in the chat bot, but then they're not doing anything about it. What could possibly be going on? Well, as happens in most LLMs and with many NCPs, hopefully you're all using many of these yourself, it went through all of the different places that it has access to. Whether that's the support tickets that the company tracks, whether it's Salesforce or any kind of CRM information, as well as of course user testing, and pulled back a variety of information that helps her better understand what's going on, including some of those highlight reels from user testing.

It gives her a little bit better of an idea of maybe what's going wrong, and it suggests, should we go ahead and create a test for this? So, she's gonna go ahead and ask it to create a test, simple as it can be. Yeah, let's go ahead and do that. And just like we saw a minute ago, it can help craft that test, Make sure that it's got what she needs. You can interact back and forth with it, telling it how you want to change it. And then when you're ready, launch the test.

At this point, we're going to go ahead and let Priya go off and get a cup of coffee. Her test is running. Her participants are being acquired, and we'll hopefully get some results from her shortly.

At the same time, Devon is working in their favorite tools in Figma. In this case, Devon's in Figma Make. Many of you may be using Figma Make today to do some early ideation, to get some thoughts and to start to get a better understanding of what could possibly be an option here. And similarly, Devin is asking user testing MCP to work with Figma make and and give some ideas on what possibly a new design might get from synthetic participants or previous results.

In this case, what comes back is that it was able to ascertain from prior results or prior information how someone might respond to this. That's really helpful information. It also has some suggestions of things that maybe would be, would have been responses from the actual humans, again, on prior human responses.

In this case, Devin likes what is being suggested and says, yes, let's go ahead and implement these changes.

And Figma make does its thing and comes back with an updated view, which Devin thinks is okay, but still needs some tweaking. So, like many of you, probably takes this design over to Figma design, puts it in a canvas, does any final adjustments, tweaks that need to happen.

You know, they want to make sure they put their final touch on this, not just take that 80% done from the AI.

And then here, now that they have a prototype that they're ready to test, goes ahead and launches that Figma plug in to also get the feedback directly from actual customers who are going to better understand what's happening here. So, now Devin's going go off and get their cup of coffee while we check-in on Marcus.

Marcus is logging in to user testing, where Marcus spends a lot of his time. But Marcus doesn't come in to the application and maybe just start a test. Instead, Marcus comes in and has an experience where he's able to look at what's going on across the organization. He's able to look at maybe some tests that are running on, always on, or in the background against some of the common themes and topics that they're looking for constant input on.

He has his arc score on his screen here, as was just talked about with Baran. So, he's able to keep an eye on what's happening with their AI chat bot. And he's also able to see maybe some of these other always on or always running tests that are happening, in addition to what's what's going, what's being done independently by other team members.

In this case, he gets his eye flagged, so the fact that that Spanish audience seems to have has had a drop in in their trust of the application recently. He really wants to dig into that. Unfortunately for Marcus, he doesn't speak Spanish. That's okay. He can go ahead and have user testing help him with this and go ahead and create the test that he wants and translate it into Spanish, so that it can it can run and and speak to the participants in the language that that they are speaking as opposed to having to always respond English. He also is going to have that AI moderator ask follow-up questions along the way in case there's anything that he wants to have additional detail on. And as you can see, he's able to give it a little bit of guidance on what he wants the AI moderator to dig into.

So, he's set this up, he's going to go ahead and launch it. The AI moderator is going to run the test in Spanish for him, so that he doesn't have to worry about, again, any kind of misspeaking or language concerns, and those results will come back into the application that he can then look through later.

So, Marcus' test is running, Priya's test is finished up now, we're almost ready for Devon's as well. Let's check-in on Priya.

Priya's going back into her LLM asking for some updates on what her how her test results look, and she wants to see, you know, again, what's been what's been captured from the test that she ran. She sees those results that we showed a minute ago, as well as again, some summaries, direct links to those highlight reels so she can dig in a little bit more.

Devin goes back into Figma and is able to see the results to their test, and see what's happening with what customers are saying about the prototype that they presented to them. In this case, of course, the heat map and other aspects as well.

And Marcus is going to interrogate with an insights discovery within the user testing platform, what's happening with the results for his group. And better understand what's potentially being seen there as well. So that he can work in the interface that he's most comfortable in, while the rest of his team is working in their interface. Well, I've talked about these three people as if they almost don't know each other, but as I mentioned, they are a team.

They do work together. And so Marcus has has deemed that he needs to, of course, bring this up. He's found something that seems like it's a problem. He goes to his messaging application, whichever one you might choose, and sends a note to to his two colleagues about, hey, you know, think something might be going on with our chatbot, particularly in Spain.

Let's dig into this. Priya says, oh my gosh, I saw something similar with some tests that I ran this week. Here's some of the results that I saw. And Devin, who was thinking ahead, has already said, you know what?

I saw some things as well. I heard that feedback from both of you. I've also incorporated that and gotten some early reads on designs. And what I'm seeing is that people really want more context in in the response.

So Priya says, great, that really helps us, that re evaluates exactly what we should work on, let's make sure we go fix that first.

So, we have three different people working in the tool that they're working in, collaboratively as a team to make the decisions that's best for their product without ever having to leave the interface that they need to work in, but still get the capabilities that they need and talk to the customers they want.

Hopefully, this gives you an idea of where we think all of this process is going. The AIS DLC or PDLC or workflow, whatever you want to call it these days, is still a little bit fuzzy for all of us, My team included. But, as we start to figure out what's working and what's not working, we want to make sure that user testing is there to help you and be part of that customer feedback loop, that customer insights loop that you and your teams are building.

So, we have had a lot of great things over the past year that I've talked about, from, you know, our MCP and AI moderation, which was super exciting, down to some of the maybe less recognized items like balance comparison, but also some fun stuff like our new browser extension, or not extension, extension less experience.

And so, I'm gonna have Bobby come back and and think help me think a little bit about how this is all coming together.

Speaker 2

Awesome. Thank you, Jennifer. Your team has clearly been hard at work, and we cannot wait to see what else you have in store. Absolutely. Thank you. Round of applause for Jennifer, everyone.

Alright. So when we started, we were back in 1967 at that wall in London. And somewhere right now, maybe in this room, somebody is inventing the next ATM, the next iPhone moment. Right? The next thing that in ten years we're all going to struggle to remember life without.

But here's the thing. Right? The speed is real. The pace is unreal. The tools are unreal.

But when you stay grounded in what your customers really need, what your customers really want, everything becomes easier. Now, one more thing before I let you go. Right? Every photo that I showed you, everything that we walked through earlier of those first time moments, how many noticed that it was generated by AI? Right? Everyone.

Right? The photos look real. Some of them look very real, but if you look closely, or in some cases not so closely, there's little things that just don't really make sense, and you notice. That's your judgment. Human insight notices, and that's really the whole point. When you spot those little imperfections, you prove that the thing you do is still the most valuable thing. Thank you very much.

Speaker 7

I personally am refusing to believe that the picture of the guy taking money out of the cash machine was generated by AI.

That that seemed so realistic, was unbelievable. And they managed to get a close close-up on him in 1967 as well, and the pure quality of it just looked real.

Thank you for thank you for the laugh for the people at the front. People at the back, not your friends anymore. Some really interesting insights from that first part of the session for me. Brian, you spoke about the insights gap.

As a researcher, the insights gap is something I think has existed for quite a long time. It's something that we struggled with, especially when we're communicating across our product team. Who agrees with me, as a show of hands?

Hands are slowly going up.

And I think what's really important is, another thing that Brian said was, when AI can build anything, knowing what to build is everything. And for me, that's the opportunity that we've got now, as researchers, as designers, as people working in product. We've got the opportunity to drive decision making forward. We've got an opportunity to have that strategic viewpoint on what should be done and when it should be done.

We've got to change the way that we communicate around that kind of stuff as well. We've got to think about what the business is trying to achieve, how we shape the conversation around what the business is trying to achieve, and make sure that we're helping them use the AI responsibly and and and driving that AI technology in the right kind of direction.

I think the people that are hands up if you're in my design workshop this morning.

That's a lot of the stuff that we covered, so we can take a lot of that kind of stuff forward and move it into some of the stuff that Bran and Jennifer and the guys were speaking about.

Now, my one of my favorite parts of these sessions, as we get to learn a lot more about our customers, we've got our customer awards coming up. These are people who have done really well. They've used user testing and really brought user insights to fore. And they're leading the way in many ways in how this next generation of AI development is going to work.

So please welcome on stage, Johan, our Chief Marketing Officer, who is going to tell us a lot more about that and he's gonna give out the awards. So Johan, please come to the stage.

Speaker 8

Amber. Thank you. Thank you.

Speaker 9

Well,

Speaker 8

leave it leave it to me to stand between you and the break.

But I have a good reason for standing before you. First, we've had some incredible speakers, and I've heard lots of applause for our speakers. I wanna take a moment to thank the team that's organized this experience and crafted it for you. So the marketing team, the production team. Could we just give them a big round of applause to say thank you?

This is this is my favorite part of the job I do, and that is to not stand on stage to talk about us, but rather to talk about you and the work that all of you do.

To do this, we run a program called the Illumi Awards. For those of you who have been part of our community for a while, you've heard of these. We're in our eighth year of running these.

And really, we want to recognize all of the work that every one of you do each day to drive innovation and growth and customer experience, especially now more than ever as we enter into this new AI era.

We had a record number of submissions this year, so many of you shared your stories with us about the change that you've driven inside of your organizations and how you've advanced the importance of human insight in the process of building experiences for customers. And if you didn't submit this year, I would urge you please, when we open and call for submissions, please share your stories with us. They're incredible to read, and it's always difficult for our team to make the judgment call about which winners win in each category. And this year, we have 27 organizations that we're recognizing. We have 12 category winners, and we have 15 distinguished luminaries, which are stories that were so compelling, we couldn't put them into a category.

And this, for us, represents the global excellence across categories, across industries, across markets.

And so I'm going to walk through and I'm going share some of the stories of some of the people who are sitting among you today. And when I announce the name of the person in the company, would ask that person to come up to the curtain here, come up, join me on stage for a photo with our CEO, Eric Johnson, who I would ask to come up on stage. He'll come up and be here in a moment to share this with me.

So, the first award that we have the winner in the room for today is the Human Insight to Action Award. This award is for leadership in transforming human insights into tangible business impact.

This organisation saw two journeys underperforming. One of them had a 66% drop off rate. Two thirds of people were just dropping off. And in the other journey, it just seemed like there was no motivation to progress in that journey.

So they looked at both at the same time, and they decided they were only going to spend one week on each, which isn't a lot of time, as we all know, but they were committed to making this happen and making it happen quickly.

What they discovered, that in both journeys, real issue wasn't choice, it wasn't motivation, it was clarity. And when they fixed it, they saw a 438% increase in bookings, they saw a similar increase in revenue, they saw their flyers get 223% more status upgrades, and saw an increase of 57% in monthly member revenue. So they had tracked two journeys in two weeks, and they transformed the way that Cathay Pacific does it. Ernest Hui, head of design, please come up to stage.

Speaker 2

Alright. Thank you, Ernest.

Speaker 8

This next award is the Synergy Excellence Award, which is for scaling insights across teams to drive alignment and better decisions. It's all about better decisions always. Right?

So this organisation saw research being slow because it was centralised, and bottlenecks were creating no timely evidence that could be used in the decision making process.

So they went back to the basics and worked hard to build a research enablement model.

They created a training program, they provided tools, and they created guardrails so that they could get over 20 colleagues running their own studies.

So they could move the decision making from opinion to evidence.

I think we all want that in our organisations, right? We want to go from opinion to evidence. What they saw was a 17% increase in savings account openings. They saw a 168% instant access hub engagement. Their help and support bounce rate dropped by 11%.

And they had a six x increase in category page visits. How's that for results? Those are some really tangible results from better alignment, better decisions, and listening is now embedded in how they make decisions at the cooperative bank. Please welcome Katie Houghton, the UX research manager there, and give a big round of applause to her team.

Our next award first of all, I'm so happy that so many of our winners are in the audience today. I love the fact that this community here is so strong that you came out today.

The next award is the digital innovator award. This award is for creating exceptional digital experiences powered by insights.

So this team, they found that these really high impact decisions were being made without hearing from their customers.

I'm a customer, actually.

Occasional testing needed to become high frequency testing.

They actually started to test more, 10x in 2025.

They took this continuous improvement and they fed it into product and design, analytics, ecommerce, and leadership, and got everybody aligned on evidence.

There's a theme here. Right? Decision making based on evidence coming from real customers, real users.

Their app NPS went up 15 points.

26% increase in stickiness, and conversion went up nearly a full point. So in one year, a new team created a new culture around insights at Royal Caribbean.

Please welcome to stage VP of Digital, Heather Bishop, and Melinda Carbonell, lead UX researcher for Royal Caribbean.

Alright. Our next award is for transformation. It's a transformation award aptly named for using human insight to drive customer experience transformation.

This organisation works with a lot of UK small businesses.

And they saw the accounting was underperforming, and the data couldn't explain why.

The empirical evidence just didn't seem to add up. So they built three prototypes, and they put them in front of eight high intent UK decision makers over the course of a week.

And rather than picking a winner, they took the best parts of each one of those, the strongest elements, to create something that was modern and relevant.

And then they AB tested that, and they saw a 35% increase in transactions, a 78% uplift in mobile, 51% of users returned, and they turned this multi prototype method into the blueprint for the way that they do research going forward at Sage.

So let's celebrate Christian Barnes, the UX researcher, and the team at Sage.

Alright. So this is the last award, and this award is for a customer evangelist.

This award is about championing a customer first culture in a business.

And this is a call this is this is I love all of these awards, but this might be my favorite one because it really is about about you. It is about the end customer.

And this person was the only researcher in an 8,000 person organisation.

One person doing all the research for this big organisation.

And so what they did was they built and accredited a 100 person super user network within the business, and they did it through coaching, templates, frameworks, guardrails.

In this process, they started uncovering things in the business, some of it was harm data that QA had missed. In fact, there was this loop in the chat bot that just with the customer saying thank you, created this registration name and card information loop over and over again that the customer stuck in. Just from a thank you. Right? It got missed by QA.

So, another challenge that they faced was these claims that were being filed. As you can tell, is in the insurance space. Right? Claims were being repudiated because the contents of a vehicle that were stolen were underestimated in value.

And it wasn't because the people, the customers were being dishonest, it was because it was too hard to figure out the value and the customers got confused. The instructions weren't good, the guidance didn't help them get to the actual value of the contents that they needed insured.

And so these challenges were going out to live customers, and this person and this 100 person super user network discovered that they could run these studies. In 2025, they ran over a thousand studies.

It cut their complaints and calls, they reduced their bounce rate on their quote pages, and the conversion rates held. Customers were happier.

And the whole organization went from going, does it work? To does it actually work for customers? Which is a massive mindset shift. They started asking, could it cause harm?

That's what Ciara Davy, principal UX research and consumer duty lead at Allianz did.

Congratulations.

Please come to state.

Alright. So those are all of our winners. We're also joined today by a number of distinguished luminaries.

Rather than keep you all in your seats longer and delay the break, I think we'll just bring up all of their logos and have a huge round of applause for our distinguished luminaries.

And with that, I will invite you to enjoy the break, and I think we'll be back in about fifteen minutes or so. So please pay attention for the the notification and enjoy the refreshment.