The experience is less like operating software and more like working with an assistant.

"We're not testing flows. We're testing relationships," Sara said.

That distinction becomes especially important with AI agents, which can perform tasks with varying degrees of autonomy.

Traditional software generally follows predefined paths. AI systems are probabilistic. Their responses can vary, even when people ask similar questions.

Success is no longer simply reaching the right destination. It's also about whether customers understand the journey, feel comfortable with the decisions being made, and believe they can intervene when something goes wrong.

For product teams, this means traditional usability testing needs another layer.

One that accounts for the human side of human-AI interaction.

Five dimensions of a better AI experience

The need for a more comprehensive approach emerged from customer research projects.

Kira Orlova, a UX manager on the Professional Services team, described how an initial project evaluating an AI-powered chatbot eventually grew into a broader research initiative.

Multiple researchers experimented with existing measurement approaches, including established trust scales and UserTesting's QXscore methodology. Another team examined academic research on human interaction with automated systems.

Their findings converged around five elements that became the foundation of AI Relationship Quality™ (ARQ™), a framework for evaluating AI user experience.

Those elements are:

1. Trust: Can people appropriately rely on the AI?
2. Understanding: Do they understand what the AI is doing and why?
3. Control: Can they steer, correct, or stop its actions?
4. Outcome: Does the AI help them accomplish something valuable?
5. Emotional Affinity: How does the interaction feel, and what kind of relationship does it create?

Individually, each dimension offers useful information. Together, they provide a more complete picture of the customer experience.

Consider an AI assistant that successfully books travel arrangements but never explains its decisions or allows changes.

The outcome might be excellent. The experience? Not necessarily.

Kira explained that the relationships between these scores can be more revealing than the individual numbers.

High outcome scores paired with low understanding, for example, can signal trouble ahead. Customers may happily accept the results until the system makes a mistake.

Then confidence can collapse.

It's like riding in a car with someone who knows the destination but refuses to tell you which route they're taking. Everything feels fine until they make an unexpected turn.

More trust isn't always better

Trust is arguably the most important—and misunderstood—dimension of AI experience measurement.

Companies understandably want customers to trust their products. Greater trust should encourage adoption, engagement, and continued use.

Right?

Not necessarily.

"Almost everyone frames trust as something that they want to maximize," Kira said. "I don't agree with that."

Her point is that AI systems need appropriate trust, not unlimited trust.

Too little trust discourages adoption. Too much creates a different problem: People may accept incorrect information without questioning it.

During development of the ARQ framework, researchers tested AI assistants using a question that couldn't truthfully be answered.

One AI assistant refused to provide an answer. Another confidently invented one in nearly two-thirds of the cases.

Across more than 200 participants, researchers observed over 26 different answers, with prices ranging from $1 to $8,000.

Yet an individual user saw only one response.

They had no way of knowing how inconsistent the answers were.

That finding exposes a serious weakness in measuring AI trust through surveys alone.

Someone might report high confidence in an AI assistant precisely because they don't recognize its mistakes (they don’t know what they don’t know).

For UX teams, the lesson is clear: Don't simply ask whether users trust your AI. Observe whether that trust changes when the system fails.

Can people recognize an error? Do they question the answer? Can they correct it?

That's a far more useful measure of trust calibration.

Why the numbers don't tell the whole story

Product leaders love dashboards.

Adoption rates, engagement metrics, task completion, and customer satisfaction scores provide valuable signals about performance.

But numbers rarely explain why something happens.

That's where ARQ takes a page from traditional UX research.

The framework combines quantitative scoring with qualitative interviews and observed user behavior.

The scores establish a benchmark. The interviews reveal what's behind them.

Consider another pattern Kira described: An AI experience scores reasonably well across most dimensions, but emotional affinity remains low.

The underlying technology may work perfectly. The problem could be something as simple as tone.

Perhaps the assistant sounds dismissive, overly confident, or unnecessarily formal.

Those problems may be relatively inexpensive to address compared with rebuilding the underlying model.

But without qualitative research, a product team could spend months fixing the wrong thing.

Kira cautioned that unusual scoring patterns should guide investigation rather than serve as definitive conclusions.

An outlier is a hypothesis, not a finding.

Watch the sessions. Listen to customers. Understand what happened.

Then decide what to change.

Turn AI experience measurement into business decisions

Companies are investing heavily in AI capabilities, often under pressure to deliver something before competitors do.

But shipping an AI feature doesn't guarantee customers will use it.

And adoption alone doesn't prove that the experience is working.

Sara explained that ARQ gives organizations a way to establish baseline measurements, compare experiences against competitors, and track improvements over time.

It also helps translate sometimes intangible UX problems into something executives can understand.

A measurable score provides a starting point for conversations about product investment and priorities.

The qualitative findings help teams decide what to fix.

For product and UX leaders, that suggests a practical approach:

• Test before launch. Evaluate prototypes and early AI experiences with real users rather than relying exclusively on automated model evaluations.
• Measure the relationship. Look beyond task success to understand trust, control, understanding, and affinity.
• Investigate the gaps. Use interviews and behavioral observations to explain unusual scoring patterns.
• Benchmark and repeat. Establish a baseline, make improvements, and measure whether the experience actually gets better.

This isn't about replacing technical evaluations of AI accuracy.

Those remain essential.

It's about recognizing that technical performance and customer experience answer different questions.

One tells you whether the system works.

The other helps explain whether people will want to use it again.

The next competitive advantage in AI is human

The companies that succeed with AI won't necessarily be the ones with the most features, the fastest models, or even the most accurate answers.

They'll be the ones that understand how people experience those capabilities.

That requires a shift in perspective.

Instead of asking only whether AI completed the task, ask whether customers understood what happened, felt appropriately confident in the result, and retained control over the interaction.

Most importantly, ask whether the experience earned another visit.

Because when customers increasingly interact with software as a partner rather than a tool, the quality of that relationship becomes part of the product itself.

"Trust is the element that to many it matters most,” Kira said. “And it's the one that you can least afford to look at alone."

Additional resources

• Sara Logel on LinkedIn
• Kira Orlova on LinkedIn
• Mike Mace on LinkedIn
• Webinar: The AI-native product loop: build, test, learn without slowing down — Learn how product teams can integrate real customer feedback into AI-powered development workflows, continuously validate ideas, and make better decisions without sacrificing speed.
• Guide: How to be sure your AI experience is excellent: Introducing ARQ — Explore UserTesting's AI Relationship Quality framework and how measuring trust, understanding, control, outcomes, and affinity can help teams evaluate and improve AI experiences.
• Podcast: UX research for AI: Building trust in experiences — Microsoft senior UX researcher Priyanka Kuvalekar explains why evaluating AI experiences requires looking beyond traditional usability metrics to understand trust, emotion, and human-centered design.
• Blog post: AI broke brains and research protocols: Here's how researchers can adapt — Researcher Michele Ronsen examines why AI's unpredictable, personalized responses complicate traditional UX research and how researchers can adapt their methods to evaluate these experiences.

Frequently asked questions (FAQs) about AI experience measurement