A/B testing is the gold standard for understanding what works. But running tests with human participants is slow, expensive, and logistically complex. What if LLMs could stand in for human users in some of these tests?
The question is not whether LLMs can simulate human behavior—they can, to varying degrees. The question is when that simulation is reliable enough to replace actual human data.
Where LLMs might work
LLMs could be useful for early-stage exploration: generating hypotheses, identifying potential issues with user flows, or surfacing edge cases that human testers might miss. They could also help with tests that involve text comprehension, sentiment analysis, or basic decision-making tasks where human judgment is relatively consistent.
For example, if you're testing different wording for a call-to-action button, an LLM might reasonably predict which version is more compelling. If you're testing a new onboarding flow, an LLM might simulate how users navigate through the steps.
Where they fall short
But LLMs are not humans. They don't have the same cognitive biases, emotional responses, or real-world constraints. They don't get distracted, tired, or confused in the same ways. They don't have the same background knowledge or cultural context.
Crucially, LLMs are trained on vast amounts of text data, which means they reflect the average of what people say, not necessarily what they do. There's a gap between stated preferences and revealed preferences—between what people say they want and what they actually choose.
The research gap
There's not yet a clear framework for when LLM-based simulation is valid. Some studies have shown that LLMs can replicate certain human judgments in controlled settings. Others have found significant deviations, especially for tasks that require real-world experience or emotional resonance.
The key question is calibration: how do we know when an LLM's prediction is accurate enough to replace a human test? We need benchmarks, validation studies, and a better understanding of the conditions under which LLMs and humans converge.
A hybrid approach
The most promising path is probably hybrid: use LLMs for rapid iteration and hypothesis generation, then validate with smaller-scale human tests. This could dramatically accelerate the experimentation cycle while maintaining statistical rigor.
But we're not there yet. The technology is moving fast, and the research is catching up. For now, LLMs are a tool for exploration, not a replacement for human judgment.