Synthetic Data Use Cases: Where AI Survey Takers Are Helpful and Where They Are Harmful

In this article, we answer critical questions leaders are asking about AI survey takers:

• How do you draw a safe line between where synthetic respondents help and where they are harmful?  
• Why is AI accurate at putting product options in the correct order, but unreliable at predicting stand-alone scores? 
• What are the six specific research tasks where real-world evidence supports using simulated survey respondents?
• What are the four areas where using AI-generated data puts your business decisions at risk?  
• What seven questions should you ask any research vendor before agreeing to use synthetic data?

Business leaders need a practical way to evaluate synthetic respondents before putting their research (and their decisions) at risk.    

We have spent the last year reviewing evidence and running our own studies so that  real-world validation dictates where AI tools belong in research design. This level of discipline is essential for achieving decision readiness, because a study only succeeds if it provides business teams with enough reliable evidence to launch a product, change brand positioning, or invest capital with confidence.   

‍

The Dividing Line: Comparing Relative Rankings and Absolute Scores

The single most useful distinction we have found is this: AI-generated respondents perform best when relative order matters, but they fail when you rely on absolute, stand-alone scores.  

To understand this distinction, it helps to look at how these two types of data work in practice. 

• Relative rankings (putting options in order of preference): For example, asking: "Does a consumer prefer Product A over Product B?" or "Rank these five new flavor ideas from favorite to least favorite." The evidence shows AI is good at putting options in a meaningful sequence. A landmark 2025 study by  Colgate-Palmolive and PyMC Labs found that AI-generated respondents achieved a 90% match to real human concept rankings. Similarly, our own Q4 2025 tracker found that the general ranking of fast-food brands was mostly preserved. 
• Absolute scores (stand-alone percentages and ratings): This is about measuring a specific number on its own, without comparing it to anything else. For example, asking: "What percentage of our target audience will recommend our brand?" or "Rate this product's appeal on a scale of 1 to 5." When we look at these stand-alone metrics, AI-generated respondents consistently show inconsistent value—sometimes inflating, sometimes deflating, and in most cases flattening out the value. For example, in our Q4 2025 tracker, the AI inflated the top recommendation ratings by 4 to 19 points across seven major restaurants. 

In short, ordering and screening tasks tend to work. Trying to rely on stand-alone, absolute scores to make a business projection does not.

‍

Five Strategic Use Cases for Synthetic Respondents, Supported by Evidence

There are six specific areas where the empirical evidence supports using synthetic respondents. 

1. Sorting large lists of features and claims 

Ranking features, claims, or messages where only the hierarchy matters.

For example, if you need to quickly understand which of 30 concepts deserves a real study, simulated respondents can shorten a prioritization exercise from weeks to hours.  

2. Concept Pre-Testing

Screening 8 concepts down to 3 before committing to full-scale choice modeling or monadic fielding 

Pre-testing helps teams avoid chasing a bad idea. Simulated survey takers let you weed out the duds early for a fraction of the cost. The Colgate-Palmolive study suggested this could be done, along with validation against real people.   

3. Survey Instrument Pre-Tests

Sanity-checking question wording, flow, and skip logic before your study launches. 

Simulated survey takers can quickly point out confusing questions, items that accidentally ask two things at once, or broken paths in your survey. As a result, you can catch glitches and clean up the questionnaire before a real panel ever sees the survey.  

4. Directional Discovery

Initial exploration before a robust primary study 

If you are asking "what questions should we be asking?", synthetic respondents can generate early hypotheses that sharpen the design of the human study that follows.

5. Hypothesis Generation

Generating qualitative rationales for why a concept might land with a target audience. 

This is a qualitative benefit unique to large language models. It gives you the ability to produce coherent narrative responses that can inform creative and strategy work, with the understanding that these are hypotheses rather than final findings.  

A hypothesis yet to be validated: Quota Boosting 

Filling 15 to 20 percent of a hard-to-find subgroup  

If you need to survey a very hard-to-reach group, such as specialized cardiologists or a rare type of consumer, finding enough real people can be slow and expensive. Suppliers are suggesting that synthetic respondents are a key use case (using AI to fill in gaps within the population), but historical evidence suggest that statistically speaking, this does not hold.  

‍

Four Areas Where Simulated Data Does Not Belong 

Just as the evidence shows us where AI tools can assist, it is equally clear about where synthetic data should not be used to guide business decisions.   

1. Tracking your brand's performance over time 

Our brand study showed that AI inflated favorability scores by 6 to 12 points, and recommendation scores by 4 to 19 points. This is not a minor statistical wobble. It is a predictable bias because the AI's "opinions" are heavily distorted by public sentiment and chatter it has already scanned on the Internet.  

2. Creating financial models and market size projections

Estimating sales volumes or product share requires highly accurate, real-world numbers. Simulated data consistently misses these targets, meaning it cannot be used to justify major capital investments.

3. Making high-stakes, either-or decisions

We cannot assume the AI will get the basic direction of consumer reaction right. In independent tests of political polling, the AI got the direction wrong for about one out of every five people, predicting that an opponent of a candidate would actually support them. 

4. Sizing up rare, business-to-business (B2B) audiences

For an AI to successfully simulate a consumer, its underlying model must have already processed a massive amount of high-quality data about that specific group. For highly specialized professional roles or narrow business audiences, that training data is incredibly thin, making the AI's guesses highly unreliable.

‍

The Critique You Need to Take Seriously 

Synthetic respondents have no theoretical reason to be considered true statistical samples of any meaningful population.  

There is no built-in mathematical guarantee that an AI model's simulated answers will match what your specific customers think. Just because a vendor proves their model worked well on one study does not mean it will work on your next project. 

The disciplined response to this challenge is not to ban AI from your research. It is to demand proof.  

Every hybrid research design should include a small group of real human respondents to serve as a benchmark. You should never base a business decision on AI responses simply because they sound realistic and fluent. Fluency is not the same as accuracy.

‍

Seven Critical Questions to Ask Before You Deploy AI Survey Takers 

If a vendor pitches you a study using synthetic respondents, we recommend asking these seven questions before you agree to the methodology. The answers will tell you if the data is safe for decision making.   

1. Will you set and write down clear accuracy targets for our specific study before we start? 
2. Has your AI model already been trained on the category or data we are trying to predict? (If it has, the AI might just be repeating old public sentiment rather than predicting how consumers will react to your new launch.)
3. Where does your model perform well, where does it fail, and how do you prove it?
4. When you update or change the underlying AI model, do results change?
5. What tests has your model failed that you have not published?
6. How is the persona prompt constructed, and on which demographics? 
7. What elicitation method is used? (Direct, follow-up, or semantic-similarity?) In other words, how do you get ratings from the AI? Do you ask for direct numbers or do you have the AI write text first?    

The answers to these questions will tell you more about the credibility of a study design than any hand-selected success story.   

‍

The Practical Starting Point 

The most honest summary we can offer? Synthetic respondents are a legitimate tool that belongs in a well-designed research program—in specific roles, with documented validation, and never as a replacement for human respondents when the answer matters. 

Scoping the role of AI correctly for your specific study is exactly the kind of work the Directions Group Centers of Excellence was built to do. 

We help clients evaluate when synthetic data fits the business question, design the right hybrid research program, and put validation in place before any insights reach a deck. 

If you’d like to discuss how these findings can apply to your research, feel free to contact the Directions Group Centers of Excellence. 

COMING SOON  

In the fourth and final article in this series, you’ll find out Five Principles for Using Synthetic Respondents Responsibly.

‍