Five Principles for Using Synthetic Respondents Responsibly

In this article, we answer important questions to guide research design: 

• What are the standards for using AI survey takers responsibly?   
• Why must every synthetic deployment require a human benchmark? 
• Why does fluent, natural-sounding AI text create an illusion of validity? 
• Why is synthetic data accurate for relative ordering but unreliable for stand-alone absolute scores?   
• Why does how you ask questions matter far more than which model you use?  
• Why should you be cautious about flashy new methods?   

‍Every new research method eventually requires a clear set of standards.  

As synthetic respondents enter the market research landscape, the insights profession has reached that critical moment.   

Over the past year, The Directions Group has studied the peer-reviewed science, run controlled head-to-head comparisons in live studies, and built an evidence-based approach to using synthetic respondents.   

In the previous three articles, we looked at the eighty-year history of modeled data, the findings from our own restaurant brand tracker, and a practical decision framework grounded in evidence.   

This final article in our series outlines the five core principles we apply to any study using synthetic respondents. These principles establish a standard of rigor that protects your research metrics and ensures decision readiness.   

Principle 1: Amplify, Don't Replace

Synthetic respondents serve human researchers. They should support our work, not speak for the real consumers we are studying.  

Surveys are an act of motivated communication between a researcher and a real person. A real person has unique experiences, opinions, incentives, and emotional reactions. An AI model can approximate those outputs, but it cannot reproduce the actual human experience. The moment you substitute AI survey takers for real humans on a strategic question where the final number is consequential, you have stopped doing research and started running a simulation.

Using AI to pre-screen concepts, prioritize large lists, or provide additional depth of understanding for hard-to-reach quota populations (always calibrated against real human data) is a valid, powerful extension of research. Technology expands the range of what we can do. It must never replace the human voice. 

Principle 2: Validate Before You Trust

Every synthetic deployment we recommend requires a held-out human benchmark. This requirement is not optional or negotiable. 

The reason is simple: sounding realistic is not the same as being correct. Fluent, well-written AI responses create an illusion of validity. Because the text reads so naturally, researchers can easily fall into a trap of false confidence, even when the underlying data is distorted. Our review of the available evidence is pretty clear: demonstrations that a synthetic model is consistent with human responses on one study does not predict performance on the next. 

Requiring a human benchmark does not mean running a full parallel study every single time. It means building a validation component into your study design. This allows the research team to identify where and how the synthetic responses diverge before those results are used to inform a strategic decision.   

Principle 3: Rank, Don't Level

Use synthetic data for relative ordering and screening. Use real human respondents when you need absolute, stand-alone scores.  

This principle is supported by clear, empirical evidence. The landmark Colgate-Palmolive study found that synthetic respondents achieved 90% of human correlation for product rankings, which is a relative ordering task. Similarly, our own restaurant brand tracker found that the relative order of brands was mostly preserved, even while the absolute scores were heavily inflated by 4 to 19 points.  

Synthetic survey takers show potential to tell you which flavor idea ranks first, which marketing message is the weakest, or which product concept should be dropped. They cannot reliably predict that sixty-two percent of your target market will recommend your brand. Those stand-alone metrics will be distorted by training data bias in ways that vary by brand and category.    

You cannot trust AI survey takers to provide accurate brand awareness, favorability, or purchase intent scores. Reporting these simulated scores as business benchmarks means drawing strategic conclusions that the methodology simply does not support. 

Principle 4: Pick the Right Elicitation Method

How you ask the LLM questions matters more than which LLM you use.

Using direct numeric prompts (such as asking the AI to rate an item on a scale of one to five), the model produced narrow distributions clustered around the middle. This is a common occurrence that strips away the valuable nuances of human opinion. 

To unlock 90% accuracy, researchers had to use a specific, advanced approach: they asked the AI to write a free-text response first, and mapped that text back to a rating scale using mathematical similarity.  

And there was another vital requirement: demographic persona conditioning. When researchers did not condition the AI personas on age and income, the overall shape of the distribution curves still improved, but the ranking accuracy collapsed to just 50%. In other words, without proper demographic instructions, the AI's ability to put product preferences in the right order was no better than a coin flip.   

This has a direct impact on how you evaluate research vendors. The question is not simply "does your model work?" The question is: "What method does your model use to extract ratings, and do you have evidence that it outperforms direct numeric prompting?"  

Principle 5: Choose Robustness Over Novelty

Workflows validated by peer-reviewed science beat flashy, proprietary techniques.

The market research industry is currently experiencing a wave of AI experimentation that is running well ahead of validation. New models and vendor claims arrive every quarter. At The Directions Group, we are deliberate about where we place our bets.

Methodological best practices beat shortcuts. Proven business impact (the ability to help a team make a confident, validated choice) beats methodological novelty designed to impress a conference audience.  

When our pilot data contradicts a vendor claim, we publish it. When the literature identifies a failure mode, we build it into our evaluation process. And if a technique we have invested in turns out to underperform, we say so.       

Our Evolving Roadmap  

We continue to investigate synthetic respondents. As the technology continues to evolve, we are investing in several areas to build a sharper, faster, more validated research pipeline:     

• In the near-term: We are adding several things to our standard toolkit: Human calibration will be a standard step in all our AI designs, text-to-score mapping will be added to our tools, and we will use our proprietary River panel and past survey waves to retrospectively test and grade the AI's historical accuracy.    
• In the mid-term: We are building a detailed domain coverage map to identify precisely where AI models have enough training data to be useful, and where data is too thin to trust.  
• In the long-term: We are designing pre-registered prediction studies on real product launches to hold AI models accountable to real-world business outcomes. That means we are asking the AI to predict outcomes, recording those predictions, and eventually grading them against the results post-launch. 

Download the Complete Study  

You can download the entire four-part research series as a single report:
An Evidence-Based Guide for Using Synthetic Respondents in Market Research.   

Download the playbook >  

Ready to figure out how to use AI in your studies?

Our Centers of Excellence can help you evaluate when synthetic data fits your goals, design the right hybrid research program, and put validation in place before insights are used to inform decision making.   

Or we can simply have a conversation.  

Contact the Directions Group Centers of Excellence  > 

‍

‍