Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models

AI summary

Overview

Research area: Natural Language Processing / computational social science — specifically, the in-silico simulation of human survey responses with large language models.

Technical level: Intermediate. The paper is readable without deep statistics background, but familiarity with survey methodology, classification metrics (F1-score), and LLM prompting helps.

Scope in one sentence: A systematic comparison of 8 different ways of extracting closed-ended survey answers from LLMs, evaluated across 32 million simulated responses, 4 political attitude surveys, and 10 open-weight models.

What This Paper Is About

LLMs are trained to write open-ended text, but most human survey data is closed-ended — each question comes with a fixed set of response options. Researchers have invented many ad-hoc tricks for forcing LLMs to pick from those options, but nobody had compared these tricks head-to-head to see which ones actually reproduce human answers. This paper tests 8 such "Survey Response Generation Methods" against real human survey data from the United States and Germany, and finds that the choice of method changes results dramatically.

Key Contributions

1. A taxonomy and empirical comparison of 8 Survey Response Generation Methods, organized into three families: Token Probability-Based Methods (First-Token Probabilities, First-Token Restricted, Answer Prefix), Restricted Generation Methods (Restricted Choice, Restricted Reasoning, Verbalized Distribution), and Open Generation Methods (Open-Ended Classification, Open-Ended Distribution).
2. A large-scale evaluation of 32 million simulated survey responses, spanning 4 political attitude surveys (ANES 2016, GLES 2017, GLES 2025, ATP 2021) across 2 countries and 2 languages, 10 open-weight LLMs (Llama 3, OLMo 2, Qwen 3 families), 4 response-option scale variants, and multiple decoding strategies.
3. Evidence that the method choice significantly affects both individual-level and subpopulation-level alignment, established through OLS regressions per dataset with the First-Token Probabilities Method as reference.
4. Practical recommendations for which method to use, including that Token Probability-Based Methods should be avoided and Restricted Generation Methods considered first.

Main Findings

• Method choice matters enormously: Individual-level alignment "varies strongly" between Survey Response Generation Methods. Even among specifications that beat the stratified baseline, the spread exceeds 0.35 on the GLES 2017 and GLES 2025 datasets.

• Restricted Generation Methods win overall: The Restricted Choice Method leads to significant improvements in individual-level alignment, followed by the Restricted Reasoning Method and the Open-Ended Classification Method. Restricted Choice coefficients in Table 3 are .148* (ANES 2016), .242* (GLES 2017), and .218* (GLES 2025).

• Reasoning does not help: Open-ended "reasoning" output does not consistently improve alignment, and in some cases degrades it — for example for Qwen 3 8B on GLES 2025 with the Restricted Reasoning method. Reasoning is also "orders of magnitude" less computationally efficient.

• A different method wins on distributions: Instructing the model to verbalize a probability for every response option (Verbalized Distribution Method) "generates well-aligned survey responses across all datasets" at the subpopulation level, with coefficients of -.028 (ANES 2016), -.219* (GLES 2017), -.296* (GLES 2025), and -.016 (ATP 2021), where lower total variation distance is better.

• Token Probability-Based Methods are unreliable: They perform poorly for smaller models, are more prone to invalid responses (especially for reasoning models), show little agreement across response-option scale variants, and often perform worse at the subpopulation level.

• Model size helps but doesn't explain everything: Larger LLMs generally outperform smaller ones at individual-level alignment, but there is no clear pattern for response-option scales and decoding strategies.

• Robustness differs by method: Mean Fleiss's κ across datasets for Restricted Choice is 0.18 (1–3B), 0.49 (7–8B), 0.74 (32–70B); for Open-Ended Classification it is 0.20, 0.54, 0.76. Token Probability-Based Methods show "poor individual-level robustness across response option scales."

• Individual-level predictability has limits: Table 5 shows the five most difficult-to-predict ANES 2016 cases (identified via a calibrated logistic regression with 5-fold cross-validation out-of-fold predictions), all with a true class probability of approximately 0, where the reported vote choice runs counter to what the persona attributes suggest.

• A global alignment view agrees: Using distance correlation as a measure of dependence between subpopulation and response-option distributions, Restricted Generation Methods again yield good global alignment.

Methodology in Plain English

The authors take four real survey datasets with known human responses. For each respondent, they build a persona description (the same attributes used in the original studies being replicated) and ask a language model to predict that person's answer to a survey question.

They vary how the answer is extracted in 8 ways:

• Token probability methods read off the model's probability for tokens that spell each answer (e.g., the token "Don" for Donald Trump), either directly on the first output token, with the vocabulary restricted, or after a fixed prefix.
• Restricted generation methods use formatting instructions plus a restricted vocabulary (via JSON schemas) to force output such as {"answer_option": ...}, optionally with {"reasoning": ...} generated first, or a full probability distribution over all options.
• Open generation methods let the model answer freely first, then ask the same model, in a second step, to classify that free text into one answer option (or into a distribution).

To test robustness, each question is asked with 4 variants of the answer scale: Full Text vs. Indexed options ("Clinton", "Trump" vs. "A", "B"), in original vs. reversed order. They use 3 random seeds (only 1 for ATP 2021) and compare greedy decoding with the model default temperatures.

Evaluation happens at two levels: individual-level alignment, measured as macro average F1-score against the human answer (with the most probable option selected for distribution-generating methods), and subpopulation-level alignment, where responses are aggregated within subgroups defined by persona attributes (e.g., gender, state, age brackets floored to multiples of ten) and compared with total variation distance for categorical options and 1-Wasserstein distance for ordinal options. They also compute distance correlation for a global view. Baselines: a stratified random-shuffle baseline, plus upper bounds from tuned random forest cross-validation (individual level) and repeated 1/3 sub-sampling (subpopulation level). OLS regressions isolate each method's effect, with Benjamini-Hochberg-adjusted significance at p < 0.05. All other methods used the same persona prompt templates from the original studies, with the system prompt "You are a political scientist predicting responses to the following question:".

Why This Matters

Impact on research: In-silico survey simulation is a fast-growing area, but methods have been under-reported, which harms reproducibility and transparency. This paper shows that a methodological detail — how you read the answer out of the model — can change results as much as changing the model itself, and it provides concrete guidance plus open code and data.

Real-world applications:

• Survey pre-testing: piloting questionnaire wording and response scales before expensive human fieldwork.
• Imputing missing survey data: simulating individual responses, which the authors note is more versatile than simulating subpopulations directly.
• Cross-national attitude research: the evaluation includes non-English, non-US data (GLES 2017 and GLES 2025 in German).
• Benchmarking and model alignment work: the same technique families are used when probing model opinions and values.

Industry relevance: The findings matter for anyone running synthetic-user or synthetic-respondent pipelines, and for LLM serving providers: Restricted Choice is both more accurate than Open Generation Methods and far more computationally efficient — Open Generation Methods, larger models, and especially reasoning models require orders of magnitude more GPU time.

Ethical relevance: The authors warn that uncritical application, particularly to directly predict survey outcomes, risks distorting public opinion and misrepresenting marginalized populations, and raises inferential-privacy concerns about simulating individuals who chose not to participate.

Future Directions

• Extend beyond Western, political-attitude contexts to other countries and to surveys about attitudes, opinions, and values beyond the topics studied here.
• Disentangle language from political context: computational constraints prevented the researchers from investigating the impact of language and country/political context independently between the ANES and GLES datasets.
• Test further response-option perturbations, such as a missing midpoint option, which prior work found affects both human and LLM survey responses.
• Explore alternative persona prompts and decoding strategies: interview-style prompting has been shown to improve alignment and might interact differently with different Survey Response Generation Methods, and more advanced decoding strategies were out of scope. Future work could also investigate more structured or theory-informed reasoning strategies, since only the default reasoning traces of Qwen 3 8B and Qwen 3 32B were tested.
• Develop evaluations that do not treat human survey responses as ground truth, since this assumption ignores biases in human survey responding.

Target Audience

Computational social scientists and survey methodologists who want to simulate human respondents with LLMs; NLP researchers working on LLM evaluation, prompting, or structured output generation; and practitioners building synthetic-respondent pipelines who need to know which extraction method is reliable, robust, and computationally affordable. The paper's recommendations section and regression tables are useful to applied researchers; the benchmark setup is most valuable to those designing new in-silico survey studies.

Authors’ abstract

Many in-silico simulations of human survey responses with large language models (LLMs) focus on generating closed-ended survey responses, whereas LLMs are typically trained to generate open-ended text instead. Previous research has used a diverse range of methods for generating closed-ended survey responses with LLMs, and a standard practice remains to be identified. In this paper, we systematically investigate the impact that various Survey Response Generation Methods have on predicted survey responses. We present the results of 32 mio. simulated survey responses across 8 Survey Response Generation Methods, 4 political attitude surveys, and 10 open-weight language models. We find significant differences between the Survey Response Generation Methods in both individual-level and subpopulation-level alignment. Our results show that Restricted Generation Methods perform best overall, and that reasoning output does not consistently improve alignment. Our work underlines the significant impact that Survey Response Generation Methods have on simulated survey responses, and we develop practical recommendations on the application of Survey Response Generation Methods.