AI experiments need more than a pile of vibes

Trying AI isn’t the same as testing it

There is plenty of experimentation happening already.

Teams are trying AI-powered user research, synthetic personas, AI-assisted analysis and new ways of communicating insights. Rose and Claire described experiments ranging from synthetic customers to interactive personas and AI-generated research storytelling.

The problem is that novelty can masquerade as evidence.

“I’ve tried this, I’ve tried that, and I’ve tried that,” Rose said, describing a pattern she sees around AI adoption. The more important question comes afterward: “Did you capture anything?”

Did a task that once required 20 minutes take five? Did the output improve? Did researchers feel more confident in the result? Did stakeholders understand an insight more clearly?

Those questions turn an AI trial into an experiment.

Start with the outcome, not the tool

This may be one of the more useful principles for AI in UX research because the technology itself encourages the opposite behavior.

A new capability appears and our instinct is to find somewhere to use it. The tool becomes the starting point: What could we do with this?

Rose’s approach effectively reverses the question.

Start with the problem. Define the intended outcome. Decide what evidence would indicate success. Then determine whether AI belongs in the experiment at all.

That sounds almost painfully conventional. But that may be precisely why it matters.

AI is changing so quickly that organizations can easily spend their time chasing capabilities rather than improving outcomes. Rose noted that even people who claim to have the answers may only have them “for about five minutes” before the technology moves again.

The better response to that uncertainty isn’t certainty. It’s disciplined curiosity.

Researchers already know how to do this

There’s something reassuring about this challenge for UX researchers: The skills required to navigate AI are not entirely new.

Form a hypothesis. Establish what you’re trying to understand. Gather evidence. Question the result. Look for bias. Consider context. Decide how confident you should be.

That same mindset also provides a useful check on synthetic user research and AI-generated insights. A synthetic persona producing a plausible answer is interesting. Whether that answer is representative, reliable or useful for a particular decision is a different question entirely.

The distinction becomes especially important as AI makes polished outputs almost effortless. A convincing result can still be the wrong result.

The future of AI-assisted research may therefore depend less on becoming expert users of every new tool and more on applying familiar research discipline to unfamiliar technology.

Rose’s closing advice captured that mindset better than any AI playbook could: “What’s the smallest thing I could do to test that?”

Related resources

• The future of insight: how information workers leverage AI + human understanding to drive smarter decisions: In this on-demand webinar, research leaders from T-Mobile and Progressive discuss where AI accelerates research, where it introduces risks such as bias and over-reliance on automation, and how researchers safeguard quality—closely matching Rose’s argument for intentional, measurable AI experimentation.
• Guide to human insight for UX research teams: This guide focuses on integrating human insight into everyday decisions and increasing research’s organizational influence.
• How AI in UX research is augmenting (not replacing) human insight: In this podcast episode, Dr. John Whalen discusses AI moderation, simulated users and AI-assisted research across the workflow. His point is to treat AI as something to test and validate rather than simply trust—“Trust, but click.”
• The responsible path to AI-accelerated Customer Insights: This blog post pairs particularly well with the “pile of vibes” argument. It explicitly addresses the tradeoffs between places where AI genuinely accelerates research, places where it creates risk, and places where human judgment remains essential.