AI Personas: When Users Aren’t Available
Role: Proposed and directed the approach | Team: 1 designer, 1 product director | Product: ClaimXperience | Outcome: cut a month-long discovery cycle to two weeks, validated with 75 real users
Note: some workflow and design details below are described generically rather than shown exactly, to protect confidential product work. The approach, my role, and the outcomes are accurate.
The problem
ClaimXperience is a collaboration tool that allows policyholders to contribute to resolving their insurance claims as quickly as possible. Only one problem: policyholders weren't using it, particularly for creating personal property inventories. Feedback and product analytics both pointed to the same root cause: the existing workflow for building an inventory was hard to use, and people were abandoning it.
We needed to understand why, but we had a real constraint. Most of our research runs on a community of volunteer users who are invested in improving the tools they use for their jobs. Policyholders aren't that. We can't reach out to them directly for research, and finding people on a research panel who'd actually been through the ClaimXperience inventory process after a real property claim would have been slow and expensive, if they were findable at all.
The idea
Rather than wait on a recruiting problem we couldn't easily solve, I proposed building AI persona agents to run an initial round of discovery — synthetic interviews we could run immediately, to form a starting hypothesis we could then test with real people.
I taught Seth, the designer on the project, how to build and interview a persona agent. My role from there was direction and oversight: I talked through refining the prompts with Seth, reviewed the agents' output, gave feedback on the workflows Seth proposed from it, and secured the budget for the real-user validation round that followed. Seth built the agents, ran the interviews, and designed and analyzed the validation study.
Examples of personas similar to those used in the project.
What the AI interviews surfaced
We created three personas, spread across different comfort levels with technology, and interviewed each about how they'd naturally approach tasks structurally similar to a property inventory — submitting forms, building shopping lists, building to-do lists — since creating an inventory is, at its core, building a very long list of everything you own. We also asked each persona to imagine they'd just been through a property claim, and asked what had gone well, what hadn't, and what the experience felt like.
Going in, each of us had a different assumption and the AI interviews did an excellent job at challenging our assumptions.
Assumptions
Product Director
About: Ryan is a busy person who files his taxes via mobile app; and he prefers the voice input method when using AI. Aligns closest to our Riley persona.
Assumption: Most policyholders would prefer to just talk through their inventory out loud.
Reality: Voice input was a strong preference for exactly one persona, and only in specific scenarios
UX Director
About: I love spreadsheets and sometimes joke that my brain works in columns and XLOOKUPs.
Assumption: Policyholders would prefer tabular entry.
Reality: None of the personas preferred spreadsheet or table entry.
Seth was smart and didn’t tell me what he assumed.
Insights
Seth's synthesis found three input methods that would work for all three personas: uploading media, picking from an existing list, and searching for and adding items one at a time. That last one was the genuine surprise — no matter which primary method a persona leaned toward, every one of them hit moments where they just needed to add a single item on its own.
Despite the speed we arrived at these insights, they were still AI generated and AI can make mistakes. We needed to test the accuracy of the information.
Testing with real people
Seth built wireframes for three proposed workflows — identical except for the item-entry method — and tested each with 25 unique participants recruited through our research panel, 75 people total. Each test had two tasks: add items and submit an inventory for review.
While over 84% of participants were able to complete the first task in all three tests, there was over 40% misclick — tapping somewhere the prototype didn't expect — rate. Despite that (or perhaps because of that) we were able to see the path that people wanted to take even though the prototype only had one valid path forward.
Click path from one test
What I'd do differently
The tests each showed participants a specific input method and asked them to complete a task with it. In hindsight, I'd change that design: I would be more confident in the results if we gave participants an open task and see which method they reach for on their own, rather than assigning them one and asking how it went. We got clean completion data, but we didn't get a clear read on unprompted preference outside of the misclicks — which is a gap I'd close if I could go back and do it again.
Conclusion
The AI-agent round didn't replace the need to test our concepts. It replaced the slower, riskier version of getting to that step. A discovery round with 5-6 real participants — scheduling, interviewing, synthesis, reporting — usually takes our team about a month. Seth went from a standing start (including the time to learn how to build and refine a persona agent) to a synthesized hypothesis in two weeks, for a population we couldn't have recruited in that window at all. That's what let us walk into the real-user validation round with a hypothesis that had already survived a round of scrutiny, instead of testing a single untested assumption cold.
Current Status
This workflow hasn't shipped yet, so we don't have production usage data — the numbers above are from concept validation, not live behavior. I'll update this case study once it's in the product.
This project is a good example of the part of my role that doesn't always show up in a job description: deciding how a team should go find an answer when the obvious path is blocked, not just reviewing the answer once someone else finds it.
