Once a model can draft a questionnaire and run an interview, the next question arrives quickly: if the questions and the follow-ups can be generated, why not generate the respondents too? A blanket yes or no answers the wrong question, because two separate claims sit inside it. The first is whether model-generated answers resemble human answers at the population level. The second is whether that data can carry the conclusion you intend to make. There is a real evidence base for the first claim. The answer to the second depends on whether your conclusion is about question wording, the relative standing of several concepts, or a subgroup proportion and difference. The way to settle it is not to read a model's account of itself, but to run a comparison against a small human benchmark sample with the right structure, and then put that comparison in the methods section.
Resemblance has research support; substitutability has to be earned
Prompting a language model to answer as a member of a target population is now usually called silicon sampling or synthetic respondents. The optimistic half of the literature is specific: one study conditioned a model on thousands of real socio-demographic backstories, proposed the term algorithmic fidelity, and showed that conditioned models could reproduce subgroup response distributions from large US surveys, with the match going beyond surface wording.
Follow-up work supplies the other half. Replicating items from the 2016-2020 American National Election Study, the average scores generated by the model came close to the survey averages, but three other things went wrong: synthetic responses carried less variation than the real surveys, regression coefficients often differed significantly from estimates based on real data, and minor changes in prompt wording shifted the distribution, as did running the same prompt three months later. Close averages can therefore coexist with unreliable structure and poor reproducibility.
A review in the marketing research literature turns scattered comparisons into a proportion that is easier to reason with: across 285 silicon-to-human comparisons, about 24.9% produced similar results, 65.3% diverged, and 9.8% only partially aligned. Stable constructs such as personality traits and political preferences travelled better, while well-established consumer behaviour effects often did not. The review's operational conclusion is blunt: do not take a model's output at face value when making decisions.
| The conclusion you need | Evidence it requires | What current research suggests | Defensible boundary |
|---|---|---|---|
| Is this item ambiguous, are these options mutually exclusive, does the routing work | Direction agrees with human cognitive interviews | Pretesting stimuli and survey items is listed among the better-supported uses, useful for spotting double-barrelled questions and vague options | A discovery tool; the final decision still rests on human pretesting |
| Which concept is preferred, which version performs better | Rank order and relative gaps agree | Stable constructs travel better, behavioural effects often do not, and results move with prompt wording | Requires at least one human benchmark; report direction only, never a precise magnitude |
| How large is a subgroup, how far apart are two groups, what would a price change do | Marginal distributions, variance, subgroup direction and magnitude | Variance is compressed, coefficients often differ from human data, and the structure of bias varies across countries and corpora | Not used directly; at most a source of hypotheses, labelled as model-generated |
Tier the uses by risk instead of issuing one verdict on synthetic data
In practice it is more workable to tier the study's use by risk and then decide how much human data it needs. Tier one is sharpening materials and items: having a model flag double-barrelled questions, vague options, or overloaded stems is cheap and reversible, and a wrong call costs one editing round. Tier two is directional comparison: the relative strength of several concepts or versions now steers budgets, so a human benchmark has to back it. Tier three is inference about levels and differences: market size, penetration, subgroup gaps, price sensitivity, and any causal phrasing. Once those conclusions reach a budget line or a public statement, the cost of being wrong is at its highest.
Tiering also changes what the human sample is for. In tier one, humans are a spot check confirming that the flagged problems are real. In tier two, humans are the ruler that shows whether the ordering has flipped. In tier three, humans remain the substrate and synthetic output only generates hypotheses. Budget then goes where it changes the conclusion, rather than into volume of generated answers.
A six-step benchmark protocol
The protocol below applies to tier two and tier three. Its cost sits mainly in the human benchmark, which does not need to match the synthetic sample in size but must cover the subgroups the study will draw conclusions about.
- Freeze materials and generation settings. Record model name and version, the full prompt, sampling parameters, the background fields assigned to each synthetic respondent, and the number of repetitions. Any change makes it a new version and invalidates the earlier validation; mixing outputs from different versions in one table is among the hardest errors to catch later.
- Build a human benchmark. Collect real respondents with the same questionnaire, the same stimuli, and the same time window. Structure matters more than size: if the study compares age bands, every band needs a usable number of responses, as in regional and stratified sample allocation.
- Compare four layers, not only the mean. Marginal distributions show the absolute gap; rank order shows whether positions are stable; subgroup direction shows whether the internal ordering reverses; variance and response shape reveal midpoint clustering and missing extremes. Write the pass criteria for all four before running the numbers.
- Test prompt robustness. Prepare at least three wording variants and two option orders, then check whether the conclusion flips. The logic parallels handling order effects through randomization, except that the uncertainty is introduced by the prompt rather than the questionnaire.
- Test stability over time. Keep the prompt and parameters, rerun after two to four weeks. If the conclusion drifts with no intervention, the output cannot support trend claims, and any real change would be indistinguishable from that drift.
- Predefine the fallback. Whichever layer fails, the claim drops to the tier that layer supports, and the failure goes into the methods section. Deleting failed runs and starting over turns validation into a ritual.
A recomputable example: rank order survived, levels and subgroups did not
The numbers below are illustrative and constructed to demonstrate the judgement, not results from a real project. A team must choose among four product concepts. The questionnaire has 30 items, eight of them concept ratings, with purchase intent on a five-point scale scored as the share choosing 4 or 5. The synthetic sample contains 600 responses (two models crossed with three persona templates); the human benchmark contains 240 respondents across four age bands, 60 each.
| Concept | Synthetic intent | Human benchmark | Bias (synthetic minus human) | Rank |
|---|---|---|---|---|
| A | 62% | 44% | +18 points | First in both |
| B | 55% | 41% | +14 points | Second in both |
| C | 48% | 30% | +18 points | Third in both |
| D | 41% | 26% | +15 points | Fourth in both |
The overall ordering is identical, so the relative standing of the four concepts survives and the "screen out the clearly weaker concepts" step in tiers one and two is defensible. Absolute levels, however, are overstated by roughly 16 points on average, and the per-concept bias ranges from 14 to 18 points rather than sitting at a constant. The subgroup comparison is worse: in the human benchmark, the 45-and-over group rated C at 36%, above B at 33%, while the synthetic sample inverted that ordering. When price moved from 199 to 249, the human benchmark's high-intent share for A dropped 9 points; the synthetic sample dropped only 2, with a further 4-point spread across prompt variants. Response shape diverges too: the standard deviation on the five-point scale is 0.82 for the synthetic sample against 1.14 for humans, with 41% versus 27% choosing the midpoint.
Read side by side, the four layers separate what may be said from what may not. A defensible report can state that under these materials the synthetic sample and the human benchmark agreed on concept ordering. It cannot report market size, cannot report age-band differences, and cannot use this data to set price. None of that rests on an opinion about model capability; all of it comes from a recomputable comparison.
Can a correction factor rescue the bias?
A stable overstatement invites an obvious shortcut: estimate the bias from the human benchmark, apply it to the synthetic sample, and buy a large sample at low cost. Whether that works depends on whether the bias itself is stable, which two checks can establish.
- Is the bias constant? List the bias for every item, subgroup, and stimulus version and look at the spread. In the example it ranges from 14 to 18 points and reverses direction in one subgroup, so it varies with the question and the population. Applying one coefficient there extrapolates the benchmark's own uncertainty into territory the benchmark never covered.
- Is the bias explained by composition alone? If it comes mainly from the synthetic respondents' age and regional mix differing from the target population, that is a sample-structure problem addressable through weighting, at the usual cost in precision, following the logic of survey weighting. Compressed variance is a different failure: the variation that regression and segmentation depend on is largely absent, and no weight restores it. NIST's generative AI profile lists this over-uniform output as a bias and homogenization risk and recommends comparing output against known ground truth using multiple evaluation methods.
- Is the bias tied to country and culture? An audit comparing Chile and the United States found that the model reproduced US survey responses more faithfully, and that the most pronounced bias dimensions differed by country: race and political identity in the US, gender, education, and religion in Chile. Passing validation in one population does not transfer to another, which matters for any multi-region or multi-language study.
- Does the bias move with model versions and time? Model updates change historical results, and the same prompt produces different distributions at different points in time. Reproducing a conclusion from a year ago may require recovering a retired model version.
The calibration test therefore reduces to three conditions: bias direction is consistent, bias magnitude is stable across items and subgroups, and the residual difference is traceable to measurable composition. Only then does a single correction factor deserve a trial, and it still needs out-of-sample validation on a held-out subgroup. Otherwise the synthetic sample should be demoted to hypothesis generation.
Failure modes and the check that catches each one
- Comparing means only. A close average is the most reassuring and the most misleading result, hiding compressed variance and unstable coefficients. Check: report standard deviations, extreme-option shares, and at least one interaction.
- Treating the prompt as a setting rather than a material. Wording, example count, and output format all shift the distribution. Check: archive the full prompt as a study material and disclose it with the report.
- Mistaking fluent plausibility for factual support. Generative systems are documented to produce confidently stated but inaccurate or internally inconsistent content. Check: never let a model output be the only basis for a statement presented as fact.
- Using synthetic answers to summarise open text. What a model says open-ended text contains reflects its corpus, not your users. Check: build open-text conclusions from human answers, logs, or interviews, and use synthetic output only to rehearse a coding frame.
- Sample-size illusion. Growing a synthetic sample from 600 to 6,000 narrows intervals that express generation uncertainty, not model bias. Check: report generated volume and usable effective sample size separately, as in sample size planning.
Disclosure: put the generation method in the methods section
Once a conclusion involving synthetic respondents is released or delivered to a client, disclosure standards apply. AAPOR's disclosure standards require a description of how the sample was generated and recruited, including whether it came from probability or non-probability methods and, where a list or panel was used, the supplier. Translated to synthetic respondents, a methods section should state that the data were model-generated, name the model and version, provide the prompt and parameters, describe how synthetic respondents were constituted, report the human benchmark's size, source, and field dates, and list which of the four comparison layers passed and which failed.
A workable sentence looks like this: concept-ordering conclusions in this study come from a model-generated synthetic sample (model, version, and prompt in the appendix), validated against a benchmark of 240 human respondents; the two samples agreed on ordering, but synthetic purchase intent ran about 16 points higher on average and the ordering reversed within the 45-and-over group, so this report does not state proportion estimates, subgroup differences, or price sensitivity. Nothing here needs softening. Its value is that a reader can see which conclusions may travel and which remain hypotheses.
Do not drop the human sample in these situations
- The conclusion will drive budget allocation, pricing, media spend, or a public statement.
- The population is underrepresented in training corpora: minority-language regions, specific occupations in lower-tier cities, people with rare experiences.
- The study measures change: pre-post comparisons, tracking, campaign evaluation.
- The conclusion concerns subgroup differences where cells are small, so model error and genuine subgroup variance compound.
- The topic is health, finance, law, or employment, where readers may treat model output as professional judgement.
- The design needs non-survey evidence such as behavioural logs, administrative records, or recorded interviews.
Conversely, when the use stays in tier one with a defined human spot check, synthetic respondents can compress the time spent sharpening materials substantially. Treating them as a first reader that hunts for problems is closer to what current evidence supports than treating them as respondents.
Running this comparison chain in WeJot
The chain fits inside one workflow. Use WeJot's AI generation to produce draft items and stimuli, run a round of simulated responses for the target population, and let it surface wording and routing problems. Then obtain the human benchmark through the cognitive pretest and soft-launch monitoring steps, and compare ordering, levels, subgroup direction, and response shape before deciding what may be reported. Where the study needs the reasons behind real behaviour, combining a questionnaire with AI interviews buys more than additional generated responses.
The division of labour is the point: generation provides speed, the human benchmark provides direction. Recording both in a single methods note, together with model version, prompt, and benchmark comparison, supports the conclusion far better than a closing sentence noting that AI was used.
Method and source references
- Argyle et al. (2023), Political Analysis — Out of One, Many: Using Language Models to Simulate Human Samples (algorithmic fidelity: conditioned models can reproduce subgroup distributions from large surveys)
- Bisbee et al. (2024), Political Analysis — Synthetic Replacements for Human Survey Data? The Perils of Large Language Models (close averages with compressed variation, divergent coefficients, prompt sensitivity, three-month drift)
- Sarstedt, Adler, Rau & Schmitt (2026), NIM Marketing Intelligence Review 18(1) — Your Next Respondent Might Be an LLM: Guidelines for Using Silicon Samples in Marketing Research (composition of 285 comparisons, stable constructs versus behavioural effects, generation and validation workflow)
- EPJ Data Science (2026) — Auditing socio-demographic and cross-societal fairness in LLM-simulated public opinion (Chile versus United States: recovery and bias dimensions both vary by context)
- NIST (2024), AI 600-1 Generative AI Profile (confident but inaccurate output, bias and homogenization, comparing output with ground truth and documenting data representativeness)
- AAPOR Disclosure Standards (disclosure of sample generation and recruitment, measurement tools, and preceding context)