Caitlin believes many teams are falling into what she calls the "optimization trap," chasing productivity gains without questioning whether the work needed automating in the first place.

She sees organizations experimenting with AI everywhere but struggling to demonstrate meaningful returns.

The problem isn't necessarily the technology. It's the approach.

Teams are doing too many things too quickly, without taking the time to make any one process particularly effective.

Her recommendation is surprisingly simple: Find something about your existing workflow that genuinely frustrates you.

Maybe it's cleaning interview transcripts, organizing research notes, or identifying customers who haven't participated in feedback studies.

Start there.

"If you're a user researcher or a product manager who just loves sitting with data," Caitlin asked, "then do you need to create a dozen agents that automate your entire data pipeline?"

Probably not.

The goal isn't to eliminate work. It's to eliminate the work that prevents you from doing something more valuable.

A clever prompt isn't a trustworthy workflow

There's a difference between asking ChatGPT to summarize an interview and building AI research workflows your team can depend on.

The first might save you 20 minutes.

The second requires a process you can run repeatedly, evaluate objectively, and improve over time.

Caitlin recommends starting with tasks that occur frequently enough to provide opportunities for testing.

Consider an automated process that identifies customers who haven't provided feedback and generates outreach lists every Monday.

After several weeks, you can evaluate the results, identify problems, and refine the workflow.

But an automation you run once every three months?

You might never discover its weaknesses.

The same principle applies to AI qualitative data analysis. A workflow that produces compelling findings once isn't necessarily reliable. It needs to deliver reasonably consistent results across repeated runs.

And that requires documentation.

"The thing that most teams don't do is document," Caitlin said. "You don't measure anything."

Without a baseline, there's no credible way to demonstrate improvement.

It's like renovating your kitchen without measuring the room first. You might end up with something beautiful, but there's no guarantee the refrigerator will fit.

How do you know whether AI-generated insights are accurate?

This is where the conversation gets particularly important for anyone using AI in UX research.

Large language models can produce remarkably convincing analysis. They can identify themes, summarize interviews, and generate recommendations that sound like they came from an experienced researcher.

But convincing isn't the same as correct.

And a beautifully written finding can still send a product team in the wrong direction.

Caitlin recommends testing AI research analysis against what researchers call a golden test set.

Essentially, it's a collection of previously analyzed research data with findings you already trust.

Suppose your team conducted 30 customer interviews and identified several important usability problems, unmet needs, and behavioral patterns.

You've reviewed the evidence. Your findings have been validated. You're confident in the analysis.

Now give that same research data to your AI workflow.

Does it identify the same problems? Does it overlook important themes? Does it introduce findings unsupported by the evidence?

These comparisons provide a starting point for evaluating AI-generated insights.

Caitlin also emphasizes defining measurable criteria for quality, not simply counting how many findings a model produces.

For teams experimenting with AI-powered customer research, this suggests a practical framework:

1. Establish a baseline. Document how long the existing process takes and what good results look like.
2. Test against trusted findings. Compare AI outputs with previously completed human analysis.
3. Measure consistency. Run the workflow repeatedly and identify where results change.
4. Document improvements. Track corrections, limitations, and performance over time.

The objective isn't to prove that AI is brilliant.

It's to determine whether the results are reliable enough to inform decisions.

Start small, especially when customer data is involved

Of course, building AI workflow automation introduces another challenge: What happens to the customer data?

Research teams routinely handle sensitive information, including interview recordings, personal experiences, and details about customers' lives.

Moving that information into external AI systems deserves careful consideration.

Caitlin uses a straightforward personal test.

"If what I'm doing right now were to leak," she asks herself, "how would I feel about that? And who would it affect?"

It's a useful question, particularly for organizations still developing AI research ethics and governance policies.

Her advice for teams operating under restrictive corporate policies is to experiment with synthetic data first.

Create representative but fictional research datasets. Test different AI tools. Compare outputs. Learn how the workflows behave before introducing actual customer information.

Then start with a small, approved pilot.

Rather than automating an entire research operation, test one clearly defined task using appropriately protected data.

Document the results and use that evidence to make the case for expanding.

It's the same principle good product teams have followed for years: Test the idea before committing to the build.

Make AI workflows a team sport

Once you've developed a reliable workflow, another problem emerges.

How do you share it?

An AI process that works beautifully on one researcher's laptop isn't particularly valuable if nobody else can use, understand, or improve it.

Caitlin recommends documenting workflows in shared, version-controlled repositories using tools such as GitHub.

That might sound intimidating to researchers and marketers who've never considered themselves technical.

But the underlying concept is familiar.

Instead of keeping your best processes locked away in personal notes, create shared instructions, correction logs, and documented workflows that teammates can test and refine.

Someone discovers an error. They document it.

Another researcher improves the process. The team can adopt the updated version.

Over time, the workflow becomes more dependable.

The important shift isn't from researcher to programmer.

It's from individual experimentation to collective learning.

Don't automate the human out of customer research

For all the discussion about productivity, measurement, and AI workflow automation, the most revealing part of Nathan and Caitlin's conversation came when they explored what shouldn't be automated.

Caitlin's first answer was unexpected.

Don't automate the things you enjoy.

She worries about allowing AI agents to take over the parts of research that originally made the work interesting.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer vel massa nec orci consectetur maximus. Phasellus tempor eros sit amet imperdiet hendrerit. Duis mollis, sapien nec molestie elementum, velit ante rhoncus leo, sit amet suscipit massa nulla quis neque. Nam vulputate dui posuere urna congue posuere. Nunc tristique condimentum mauris. Nullam elementum orci magna, congue efficitur nulla vulputate elementum. Ut ultrices quam molestie, laoreet diam id, facilisis eros. Etiam metus massa, ultrices non ornare non, facilisis nec lorem. Proin gravida efficitur metus, vel dapibus velit facilisis quis. Aenean ullamcorper purus tellus, sit amet accumsan augue mollis sit amet. Nam vitae lacus condimentum sapien ullamcorper sollicitudin. Ut sollicitudin velit felis, quis pharetra ante molestie sit amet. Ut vulputate, ante at gravida venenatis, lorem leo venenatis urna, nec mollis leo justo vel ipsum. Duis semper, est sed semper ornare, massa odio hendrerit nibh, vel tincidunt sem velit eget erat.

Duis convallis vulputate arcu, vel lacinia urna tempor sodales. Nullam commodo, magna in lobortis semper, lectus risus rutrum elit, eget feugiat arcu elit non turpis. Sed eu fermentum arcu, porttitor bibendum tellus. Curabitur quam nulla, euismod eget sem a, dapibus consectetur arcu. Aliquam erat volutpat. Fusce fringilla porttitor scelerisque. Etiam sodales elit condimentum eleifend pharetra. Mauris blandit enim id felis convallis, ut finibus justo hendrerit.

There's also a deeper concern: What happens to our ability to think critically when we outsource too much of the thinking?

Consider customer interview videos.

AI can summarize what participants say, identify recurring themes, and extract supporting evidence.

But watching someone struggle with a product can reveal something a transcript alone might miss.

A hesitation. A confused expression. The frustration behind a seemingly simple comment.

Nathan recalled how watching unmoderated research videos had sometimes taught him more about empathy than the specific questions he was trying to answer.

Those experiences stayed with him years later.

That's difficult to capture in an automated summary.

And it raises an uncomfortable possibility: The more efficient our research processes become, the less time we may spend understanding the people behind the data.

The real opportunity isn't doing less research

AI research workflows have enormous potential to help teams move faster, analyze more information, and make better-informed decisions.

But speed alone isn't a business outcome.

For product, UX, and customer experience leaders, the more meaningful questions are whether AI helps uncover better insights, reduce avoidable rework, and create experiences customers actually need.

That requires measurement, experimentation, and a willingness to question the technology rather than simply admire its output.

It also requires protecting the human judgment that gives research its purpose.

Automate the repetitive work. Test the results. Share what you learn.

But don't surrender the curiosity that helps you recognize an important insight when you see one.

As Caitlin put it, "I'm really afraid of automating the joy out of everything I do, and I see that happening."

Additional resources

• Webinar: Evolve your UX research strategy to win in an AI world — Discover how leading UX teams combine AI and human expertise to accelerate research, validate insights, and make more confident design decisions.
• Guide: The AI in UX Research Report — Explore how researchers are adopting AI, the challenges they're encountering, and what the technology means for the future of UX research.
• Podcast: How AI in UX research is augmenting (not replacing) human insight — Cognitive scientist Dr. John Whalen discusses AI-powered research, simulated users, and why human expertise remains essential to understanding customers.
• Blog post: How a UX researcher uses AI in their daily workflow — A practical look at incorporating AI into research planning, interview analysis, note-taking, and reporting without losing sight of research quality.