Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

AI summary

Overview

Research area: AI evaluation and computational social science, specifically the psychometrics of large language models used as synthetic survey respondents ("silicon samples").

Technical level: Intermediate. The paper is readable without specialist mathematics, but it relies on psychometric concepts (Cronbach's alpha, factor congruence, mediation), a composite scoring metric, and bootstrap inference.

Scope: A single-benchmark audit comparing 14 "decision models" from 11 providers and 47 generative LLMs against human survey data from 263 Lithuanian employees, using a composite Psychometric Similarity Score (PSS) and several statistical baselines.

What This Paper Is About

Researchers increasingly substitute AI-generated "synthetic respondents" for real people in surveys, but plausibility is not the same as validity. This paper tests whether a new class of language models, which return a probability for every point of a rating scale instead of writing text, can stand in for human respondents on a full organisational-psychology instrument, and whether newer generative models have closed the gap since an earlier audit of 37 LLMs. The authors ask not just whether the synthetic answers look human on average, but whether they reproduce the internal structure of the survey and forecast how a specific person actually answers.

Key Contributions

1. First psychometric evaluation of decision models as survey respondents. Fourteen decision models from eleven providers were run on 100 respondents and 68 items, returning full probability vectors, at a total cost of $1.27 across 1,400 respondent-model cells.

2. An updated generative leaderboard of 47 LLMs under a revised scoring. It includes 11 models released after the original audit and compares successive releases within 10 product lines, finding that in 5 of those lines the newest release scores lower than an earlier release by more than resampling noise.

3. A dissection of what probability vectors actually contain. Through control grids with no profile, repeated requests, swaps of gender, position and education, a profile regression read out like a decision model, and proper-scoring-rule forecasts, the paper separates sample-level fidelity from person-level information and calibration.

4. A set of nine recommendations for evaluating synthetic survey respondents, together with corrected reference points including a second sample of 100 real people and a copula reference that excludes its own training data.

Main Findings

• No model reaches the human ceiling. The best generative cell, MiniMax M3, reaches PSS 0.813. The real answers of the same 100 people (the oracle) reach 0.886, and a Gaussian copula fit in sample reaches 0.927.

• The best models are level with a copula fit only on other respondents. The out-of-sample copula reaches 0.824, and the difference to MiniMax M3 (+0.01, interval −0.07 to +0.09) is within noise. The top of the generative field is dense: 12 generative cells have paired intervals to MiniMax M3 that contain zero.

• Decision models lag the generative field. Under the default rounded readout, 13 of 14 decision models score below the fixed reference gpt-5.4-mini (0.743), with intervals excluding zero. The best decision model, Jev 1.13, reaches 0.681.

• Readout choice matters enormously. Read as the most probable category rather than rounded, Jev 1.13 rises to 0.786, which would place it 5th of 47 generative cells instead of 27th. Its difference to MiniMax M3 then becomes −0.03 (−0.10 to +0.04), within noise. This is the best of 42 decision cells.

• Decision models capture marginals but not correlation structure. Averaged over respondents, their probability vectors come much closer to the human item marginals than rounded answers do, yet 33 of the 47 generative models reproduce the pattern of item correlations better than any decision model.

• Newer is not better. In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it. For example, GPT-6.1 Sol reaches 0.510 while the older gpt-5.4-mini reaches 0.743.

• The same model name behaves differently through the two interfaces. GPT-6 Luna as a generator reaches 0.621 with a correlation component of 0.566; as a decision model it reaches 0.547 (rounded) with correlation falling to 0.339. The decision interface also shows more acquiescence (1.48 human SD against 0.53).

• Probability vectors do not forecast individuals well. For every decision model, the vectors are worse than the answer frequencies of other respondents as forecasts of each person's own answers, mainly because the model's average vector differs from the population's. Mixing vectors with the population distribution helps, but does not reverse the ordering.

• Profiles carry little information. A multinomial profile regression fit on 163 real people improves on the answer frequencies of other respondents by only 1.0%, suggesting the eleven profile fields say little about how a person answers.

• The human reference bar is itself noisy. A second sample of 100 real people averages 0.79 (0.76–0.84 over 20 draws), and 8 generative cells score above that average. At n = 100, PSS cannot cleanly rank the best models against real respondents.

Methodology in Plain English

The authors took an existing open dataset of 263 Lithuanian employees who answered four work-psychology scales totalling 68 items in 12 subscales, with 13 reverse-keyed items. They selected a fixed sample of 100 respondents, stratified by gender and position, and described each one with eleven profile fields such as age, education, sector and organisational tenure.

Each of these profiles was fed to every model as a "state," and every survey item was turned into a scored question with the Lithuanian scale anchors as a rubric. Decision models returned a full probability distribution over the scale points for each item; generative models returned ordinary text answers to all 68 items in one call. The authors then converted the distributions into answers in three ways: the rounded expected score (the default), the single most probable category, and a random draw from the distribution.

Every synthetic sample was compared with the 263 real people using the Psychometric Similarity Score, a weighted composite of five parts: how close the item distributions are (a Jensen-Shannon divergence term), how similar the 68×68 item correlation matrices are, how close Cronbach's alpha is, how close a mediation effect is, and how well 96 demographic contrasts match in sign and size. Uncertainty came from a paired bootstrap with 200 draws that resamples both respondents and humans.

Crucially, the authors added baselines that do not involve language models at all: a Gaussian copula fit on the human data (both in sample and fit only on the 163 people outside the headline sample), independent draws from the human marginals, and a regression on the profile fields. They also ran control grids with no profile, repeated identical requests, and profiles with gender, position or education swapped. Finally, they scored each model's probability vector as a forecast of the individual's own answers using the ranked probability score, a strictly proper scoring rule, against the answer frequencies of other respondents.

Why This Matters

Impact on research. The paper separates two things that are easily conflated: a synthetic sample that looks plausible in aggregate and one that behaves like human data in its internal structure. Because no language model beats a copula fit on other respondents, the authors argue that synthetic respondents built on demographic profiles should aim at sample-level fidelity rather than claims about individual people.

Real-world applications:

• Survey pretesting and pilot studies in market research and the social sciences, where synthetic samples have been recommended mainly for ideation and pilot work.
• Organisational psychology and HR analytics, where instruments like the Individual Work Performance Questionnaire and UWES-17 could be simulated to plan studies or stress-test questionnaires.
• Questionnaire design, since the audit shows how readout and elicitation format change what a model appears to believe, which is directly relevant to pretesting item wording.
• Model procurement and benchmarking, giving buyers a template for checking whether a newly released model version is actually an improvement.

Industry relevance. Decision models are cheap (all fourteen cost $1.27 combined) and avoid several failure modes of generated text, such as parsing errors, refusals and long reasoning traces, which makes them attractive for scaling survey simulation. The finding that newer releases are not reliably better, and that the readout choice can move a model from 27th to 5th place, is an operational warning for anyone building pipelines on vendor version numbers.

Future Directions

1. Disentangling the confounds between interfaces. The authors state their design cannot separate output format from route (OpenRouter versus vendor APIs), prompt language (English JSON state with Lithuanian items versus a fully Lithuanian prompt), and sampling (deterministic endpoints versus sampled generation). Controlled experiments varying one factor at a time are the obvious next step.

2. Making the model's vector depend on the person, not just the item. Since the average vector differs from the population distribution and profile fields add only 1.0% over population frequencies, future work could test richer respondent representations, such as interview-based personas of the kind used for General Social Survey prediction.

3. Improving calibration of probability vectors. The vectors carry about as much person-level information as a profile regression, which suggests a repair path: post-hoc calibration against human data may recover more of their value.

4. Validating the recommendation set against other instruments and languages. The benchmark is one open Lithuanian organisational-psychology survey at n = 263 and n = 100, so whether the PSS ordering and the nine recommendations transfer to opinion surveys, other languages and larger samples remains open. The authors also explicitly flag that a single mediation value at n = 100 should not be read as a model property.

Target Audience

Researchers and practitioners in computational social science, survey methodology and psychometrics who are considering LLM-generated synthetic samples; AI evaluation engineers who benchmark model versions; and applied data scientists in market research or organisational psychology who need to judge whether a synthetic sample can support the same interpretations as human data. Readers with a basic grasp of correlation and reliability statistics will get the most from it, though the core argument is accessible to beginners willing to follow the scoring logic.

Authors’ abstract

Language models are increasingly used to create synthetic survey samples, also called silicon samples. Decision models are a new class of language models that return a probability for each point of a rating scale instead of text. We ask whether these models, or the newest generative LLMs, can stand in for human respondents on a psychometric survey. We test fourteen decision models from eleven providers and 47 generative LLMs on an open Lithuanian organisational-psychology survey (n=263, 68 items), with 100 real respondent profiles, the Psychometric Similarity Score (PSS) and statistical baselines. The best generative model, MiniMax M3, reaches PSS 0.813. This is level with a Gaussian copula fit only on other respondents (0.824) and below a copula fit in sample (0.927). A second sample of 100 real people scores below the best models, so at this size PSS cannot rank them against real respondents. In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it. With the default rounded readout, 13 of 14 decision models score below the generative reference gpt-5.4-mini. Of the 47 generative models, 33 reproduce the pattern of item correlations better than any decision model. Averaged over respondents, the probability vectors are much closer to the human item marginals than rounded answers. They also rank respondents about as well as a profile regression fit on 163 real people. As forecasts of each person's own answers, however, every decision model is worse than the answer frequencies of other respondents, mainly because its average vector differs from them. The profile regression improves on these frequencies by only 1.0%, so the profile fields say little about how a person answers. Synthetic respondents built on such profiles should therefore aim at sample-level fidelity.