Why AI became a research companion

Mario’s entry into AI began inside Google Cloud, working directly with engineers who were testing early AI assistance for troubleshooting, monitoring, and optimizing cloud applications. The experience gave him a front-row seat to how real users interacted with AI, and what they expected from it.

He recalls that engineers, in particular, had high standards, “They had this mental model that a senior software engineer was sitting next to them. So they had very high expectations for the tool.”

At the same time, early adopters showed patience. Latency wasn’t ideal, answers were sometimes incomplete, and hallucinations happened—but users saw potential. When AI could summarize complex documentation, link to official sources, or provide a quick starting point for troubleshooting, it reduced cognitive load and accelerated workflows.

That mix of skepticism and curiosity shaped Mario’s belief that AI’s role in research is not to replace experts, but to assist them. And today, the tools available to researchers extend far beyond what was possible even two years ago.

How AI adds value across the research workflow

One of the core themes from the interview is the expanding role of AI in user research. To make sense of this transformation, Mario referenced a framework from a former Google DeepMind colleague that breaks the research lifecycle into three phases: planning, execution, and activation. AI now shows up meaningfully in each.

Planning: Turning ambiguity into clarity

Researchers often begin with loosely defined questions or unfamiliar domains. AI tools can help:

• Generate and refine research questions
• Summarize existing knowledge
• Identify assumptions and gaps
• Draft briefs or early frameworks

“This is the moment where you need to define and get your research ideas,” Mario said. “AI can definitely help on that, especially when it’s a new domain you’re not familiar with.”

For new researchers especially, this can feel like having an on-demand subject-matter guide.

Execution: Scaling and accelerating the heavy lifting

During fieldwork and analysis, AI handles many repetitive workflows, including:

• Drafting surveys or interview guides
• Suggesting variables and hypotheses
• Coding open-ended responses
• Identifying themes in qualitative data
• Generating early analysis plans

Tools can even protect participant privacy. As Mario notes, “There are tools that can replace the voice and appearance of a participant while keeping the emotion, so you don’t dilute the qualitative richness.”

This expands possibilities for ethnography and usability testing in sensitive contexts.

Activation: Turning findings into compelling stories

Once data is analyzed, AI becomes a storytelling partner—able to:

• Summarize transcripts
• Draft reports, memos, and executive summaries
• Produce audience-specific versions of findings
• Transform long academic-style documents into conversational blog posts

Mario demonstrated this by uploading one of his research papers to a model to create other content, such as blog posts. “The output was actually pretty good,” he said. “It saved me a lot of time converting 5,000 academic words into a 500-word summary with the right tone.”

This phase is where AI copilots truly shine: translating complexity into usable insights.

Prompt engineering: The new language researchers must learn

One of Mario’s strongest messages is that the quality of AI-generated insights depends heavily on the prompt.

“The way you prompt makes a massive difference in the quality of the answer—in the depth and the length,” he explains. “Prompting is learning a new language.”

Unlike traditional search, where brief keywords typically suffice, large language models respond differently to subtle prompt changes. That variance becomes even more pronounced when comparing outputs across tools.

Mario describes reviewing academic papers where the prompt is hidden in the appendix. I want to know the tool they used, their prompt, and see if I get the same results, he said. 

For researchers, this skill is becoming fundamental—not just using AI tools, but directing them.

The rise (and limits) of synthetic users

While AI in user research has matured quickly, synthetic users remain one of the most debated topics. Many teams hope they can accelerate early testing, reduce cost, or generate insights without recruiting participants. Mario offers a balanced, evidence-based take.

He outlines three types of synthetic data:

• Data boosting or augmentation: Using AI to expand or infer additional quantitative data from existing samples.
• Fully synthetic data: Generating complete datasets or responses without human input.
• Synthetic personas: AI-generated users researchers can “talk to” for early-stage qualitative exploration.

But despite the hype, current performance remains inconsistent. Quoting a recent summary paper, Mario said, “Studies show some replications, but many non-replications. LLMs cannot be assumed to mimic human behavior reliably across items and across countries.”

Key concerns include:

• Bias in demographic representation
• Reduced variability, flattening nuanced opinions
• Sensitivity to small prompt changes
• Lack of emotional and contextual depth, especially in qualitative scenarios

Mario warns that the lack of transparency from many vendors makes evaluation even more difficult, “If on top of AI being a black box, the company is another black box, we really don’t know what’s happening.”

Still, he believes synthetic personas have potential as tools for very early ideation or piloting—just not as substitutes for real users.

“My personal view is that I prefer to talk to real people if I can,” he said. “And now there are so many tools out there that can allow you to do that, including, obviously, UserTesting.”

Why the human researcher still matters

AI can summarize themes, cluster responses, and write beautifully formatted insights. But as Mario reminds us, qualitative research in particular involves nuances that AI often misses.

He points out that transcripts lack tone, pacing, facial emotion, and subtle contradictions, “You can say the same thing with a different tone and it means the opposite.”

A model may understand syntax, but not always intent.

Mario also describes a test where he asked a model to recreate a data visualization by reading a table. “The image looked great, but the numbers were wrong,” he said. “After an hour and a half of prompting, I gave up.”

This is the tension many researchers face: AI can produce outputs that look compelling but contain subtle (or significant) inaccuracies.

This is why he advocates for testing AI tools using the datasets researchers already understand and vetted. “Try to reproduce results you’ve already done,” he said. “That’s the best way to evaluate quality.”

Rather than fearing replacement, researchers should view AI as amplifying their judgment—not diminishing it.

How teams can responsibly integrate AI into user research

Mario offers practical guidance for teams experimenting with AI in user research, including:

• Start with known data. Evaluate models by retesting open-ended coding or reproducing charts you've already validated.
• Use the right licenses. “Never use a personal license. Always use commercial licenses and review terms carefully.”
• Protect participant privacy. Ensure data isn’t being used to train the model.
• Experiment frequently. Model behaviors change constantly. Familiarity matters.
• Expect variation between tools. Different models bring different strengths.
• Keep the human in the loop. Use AI for acceleration—not for bypassing participant reality.

The opportunity: Reclaiming time for deeper work

For Mario, the promise of AI isn't about replacing researchers—it’s about enhancing their ability to focus on what matters most: asking better questions, interpreting human behavior, and influencing decisions.

He suggests a long-term vision where companies use AI to synthesize past research rather than reinventing it each time. 

“You can feed it hundreds of research reports and get a summary so you’re not reading 200 documents just to get started,” he said. 

Imagine every researcher entering a project already informed by years of organizational knowledge. AI could make that possible.

“Do not wait to experiment—just do it carefully. Test the tools, learn how they behave, and use them to make your research better.”

Episode links

• Mario on LinkedIn
• Amrit on LinkedIn
• Callegaro Research
• Quantitative User Experience Association
• There’s an AI for that
• Nathan Isaacs on LinkedIn
• Webinar: Mastering AI-Powered surveys with UserTesting
• Guide: The UX research methodology guidebook
• Podcast episode: AI in UX research with John Whalen (Insights Unlocked – Episode 191)
• Blog post: How to test AI: A practical guide for evaluating AI user experience and product design