Don't Replace Your Human Sample Just Yet: An Evidence-Based Guide to Synthetic Respondents

The question has officially gone mainstream: Should we be using synthetic respondents in research?

Whether you lead an insights team, drive marketing strategy, or steer innovation, you are likely feeling the pressure to conduct research faster and cheaper. Synthetic respondents (aka, AI models programmed to answer survey questions as if they were target customers), are being offered up as the solution.  

It is a tempting pitch. Some vendors claim they can instantly fill hard-to-reach quotas and replace 15 to 85 percent of the real people taking your surveys with "synthetic respondents." Eventually, they say, this will fully replace real people. And the promise is to cut fieldwork costs by 40 percent. However, this framing is aggressive and ahead of what the evidence supports.     

At the same time, serious researchers are pushing back. They are raising foundational questions, based on peer-reviewed critiques, and should be taken seriously. They argue there is no theoretical reason for naive LLM respondents to be valid as samples. Because human answers and AI answers are generated in fundamentally different ways, running standard statistical comparisons between the two groups is technically meaningless.    

So, who is right about synthetic respondents?

Both sides cannot be entirely correct, and a research partner who tells you otherwise is not serving your interests. Moving ahead with synthetic respondents without thorough consideration can risk injecting hidden bias into your data. You could end up basing major strategic decisions on fiction.

The search for an answer

Instead of picking a side or staking claim to the middle ground, we sought a thoughtful answer. We spent the past year running pilots, reviewing academic research, and putting synthetic respondents head-to-head with humans in a live study to see exactly where they help, and where they fail.

In this article, we answer questions leaders are asking about synthetic respondents:

• Can synthetic respondents actually replace human survey takers?
• Is synthetic data a new concept in market research?
• What are the risks of using AI models as survey respondents? 
• How do Large Language Models (LLMs) create contamination and bias in survey data?
• Should AI tools replace or amplify human intelligence in market research?

Data modeling isn’t new, but its history can inform how to use it in the future. 

One of the most important things to understand about synthetic data is its age. The underlying idea traces back to the 1940s, when scientists Ulam, von Neumann, and Metropolis used early simulations to solve complex physics problems. 

In 1987, a researcher named Donald Rubin created a formal way to fill in missing survey data with plausible modeled answers. By 2011, using complex math to fill in data gaps became a standard tool in epidemiology and official statistics. In 2014, deep generative models opened an entirely new era of simulated data.  

AI-powered synthetic respondents, which hit the commercial market around 2022, are just the latest chapter in this story. Decades of academic discipline have already shown us how to use modeled data with rigor. That lineage matters because it tells us what is possible and where the boundaries lie.   

What is genuinely new (and more complicated) about today’s LLMs?

Large language models expand what synthetic data can do. They produce coherent, open-ended rationales instead of just numbers. A demographic backstory in a prompt can be deployed thousands of times in minutes. A single model can span consumer goods, healthcare, and financial services, instead of needing a custom mathematical model for every single study. But this expansion comes with new risks.

What are the risks of using LLMs in research? 

Data contamination. The model may have already been trained on the very category you are studying. It is built on past data, meaning it is often just recalling public sentiment rather than predicting how a consumer will react to a totally new launch.

The plausibility trap. Fluent, well-written synthetic responses create a false sense of confidence. Just because an answer sounds true does not mean it is.

‍Flattening of nuance. Models tend to regress to the "typical" response, hiding the outliers and flattening the nuance where true human insight usually lives. If you ask a model to rate something on a scale of 1 to 5, it clusters its answers around the middle. This is a risk because the opinions on either end are often where the most valuable human insights live. Instead of capturing how people actually think and feel, flattening leaves you with an artificial, middle of the road consensus.   

Amplifying bias. Generative models replicate the biases in their training data, meaning underrepresented groups often suffer the most.  

Opaque sampling. With older math models, researchers could describe what population they were sampling from. With an AI model, we cannot.  

The Directions Group position on synthetic respondents 

Instead of tossing out AI or going all in, we are asking better questions about when it is appropriate to use it. Any technology that powers our work must serve to answer our clients' business questions in the best way possible. 

Our core position remains unchanged: Synthetic tools should amplify the best of human intelligence.  

We believe synthetic respondents are a real tool with real limits. Anyone can access AI resources, but AI cannot replicate human empathy. Do not let algorithms replace what only human respondents can answer.

Up next: The playbook for using synthetic respondents 

Over the next few weeks, we will share the data from our own live brand tracker test. We will show you where the evidence is encouraging, where the critiques are valid, and what guardrails you should demand before any AI data is used to guide decision making.  

Need more guidance now? 

If you need help right now, reach out to the Directions Group Centers of Excellence.  

We help clients decide if synthetic data fits their questions and goals, scope pilot designs, evaluate vendors, and put validation in place before any insight is finalized.