Graph-Based Alternatives to LLMs for Human Simulation

AI summary

Overview

Research area: Human behavior simulation, graph neural networks, and large language model alternatives.

Technical level: Intermediate. Readers familiar with basic machine learning concepts (training, evaluation, embeddings) will follow most of the paper, though some familiarity with graph neural networks and LLM fine-tuning helps.

Scope: The paper introduces GEMS, a graph neural network framework that reformulates close-ended human simulation tasks (survey response prediction, test-taking) as link prediction on a heterogeneous graph, and shows it matches or beats fine-tuned LLMs while using roughly 1,000 times fewer parameters.

What This Paper Is About

Large language models have become the default tool for simulating human behavior—predicting how people answer survey questions, vote, or perform on tests. This paper asks whether LLMs are actually necessary for so-called "close-ended" simulation tasks, where the goal is simply to pick one option from a fixed set. The authors propose that the relational structure among people, questions, and choices contains enough signal on its own, and that a much smaller graph neural network can do the job just as well.

Key Contributions

1. GEMS framework: A formulation of close-ended human simulation as link prediction on a heterogeneous graph with three node types (subgroups, individuals, and choices) and two relation types (membership and response).

2. Systematic comparison across three evaluation settings: Missing response imputation, prediction for entirely new individuals, and prediction for entirely new questions—each capturing a widely studied use case in prior LLM-based simulation work.

3. A lightweight LLM-to-GNN projection: A linear mapping trained with ridge regression that transfers frozen LLM hidden states into the GNN embedding space, enabling generalization to new questions without ever querying the LLM at inference time.

4. Efficiency and transparency analysis: Empirical demonstrations that GEMS uses roughly 10³ fewer parameters and 10² less compute than LLM fine-tuning, can be affordably ensembled, avoids pretraining data contamination, and remains permutation-equivariant to input ordering.

Main Findings

• GEMS matches or beats strong LLM baselines. Across OpinionQA, Twin-2K, and Dunning-Kruger datasets, GEMS outperforms zero-shot, few-shot, and chain-of-thought prompting, and matches 8-shot supervised fine-tuning—often exceeding it slightly (e.g., 57.00 vs. 56.76 on OpinionQA setting 1).

• No text is needed for settings 1 and 2. For missing responses and new individuals, GEMS achieves competitive accuracy using only a learnable feature table for choices and subgroups, without any language representations. The relational structure alone provides sufficient signal.

• Setting 3 requires only a thin language bridge. For new questions, a single linear projection from frozen LLM hidden states to GNN embeddings suffices, trained via ridge regression over as few as 20–500 questions depending on dataset.

• Dramatic efficiency gains. GEMS trains in under an hour on OpinionQA, whereas fine-tuning a 70B LLM on the same data is extrapolated to require roughly 500 GPU-hours. Scaling to 10× larger datasets pushes LLM costs toward 10³ GPU-hours while GEMS remains in the low single digits.

• Ensembling is affordable and effective. Because GEMS is so cheap to train, the authors ensemble 11 differently initialized models—still roughly 10× cheaper than one LLM fine-tuning—and gain consistent accuracy improvements (+0.31, +0.13, and +1.52 points across the three settings).

• GNNs and LLMs make similar errors. Contingency tables on OpinionQA show shared failures dominate (39.6%, 46.7%, 38.8% across settings), with limited headroom for cross-model ensembling in settings 1 and 2. Setting 3 shows more complementarity (~20% off-diagonal mass).

• Classical baselines fall short. XGBoost and matrix factorization underperform GEMS in settings 1 and 2, and cannot handle setting 3 at all—XGBoost lacks cross-question transfer, and matrix factorization cannot construct embeddings for unseen entities.

• Embeddings reveal interpretable structure. Principal components of GEMS embeddings on OpinionQA correspond to political ideology and class. Individual-level embeddings show substantial within-subgroup heterogeneity, in contrast to LLM stereotyping. Choice embeddings capture ideological similarity across different wordings, whereas LLM hidden states cluster by surface features like the phrase "a top priority."

• GEMS is more transparent. Training from scratch on task-specific data eliminates pretraining contamination and inherited social biases, and attention-style ordering sensitivities vanish because GNN aggregation is equivariant to neighbor order.

Methodology in Plain English

The authors reframe the simulation task as a graph problem. Each person is a node. Each possible answer to a question is also a node. Each demographic profile (like "female, age 30–44") becomes a subgroup node. When a person answers a question, that creates an edge from the person to the choice node they selected, and each person is linked to the subgroups they belong to.

A graph neural network then passes messages along these edges to learn embeddings—numerical representations—of every individual and every choice. To predict what someone would answer to a question they have not yet been asked, the model computes a dot product between the person's embedding and each candidate choice's embedding, then applies a softmax to get a probability distribution over the options.

Training works like a fill-in-the-blank exercise. During each training step, the model randomly hides some response edges and learns to reconstruct them from the remaining graph, with the hidden edge as the positive example and the other options for that question as implicit negatives.

For new questions, no responses exist yet, so the choice nodes are isolated and have no learned features. The authors solve this by freezing an LLM, extracting its hidden state for each choice's text, and training a simple linear projection that maps those hidden states into the GNN's embedding space. At test time, a new question's choices are represented via this projection rather than via graph message passing—no LLM calls required at inference.

Why This Matters

Impact on research: The paper challenges a default assumption in the human simulation literature that LLMs are the natural or only viable tool. It provides direct head-to-head evidence that relational structure carries most of the predictive signal for close-ended tasks, which reframes what the field should be studying and lowers the compute barrier for participation.

Real-world applications:

• Survey research and polling: Predicting public opinion on new questions without fielding expensive new surveys, useful for pollsters, social scientists, and market researchers.

• Pluralistic AI alignment: Simulating diverse viewpoints for new individuals to test whether AI systems serve different demographic groups equitably.

• Educational assessment: Modeling how students will respond to grammar and reasoning test items, including predicting responses to newly designed questions.

• Social science experimentation: Predicting treatment effects in experiments that ask participants to choose among discrete options.

Industry relevance: Any organization that spends money running LLM inference for user modeling, recommendation, or prediction of discrete choices could substitute a graph-based approach at a fraction of the cost. The dramatic reduction in compute requirements also brings simulation capabilities within reach of academic labs and smaller companies.

Future Directions

1. Extending to open-ended simulation. The paper explicitly restricts itself to close-ended tasks. Whether graph-based methods can complement or replace LLMs for free-text simulation (e.g., generating narrative responses) remains open.

2. Hybrid architectures. The error analysis shows GEMS and LLMs make different mistakes in setting 3, suggesting an ensemble or joint training approach could capture complementary strengths.

3. Scaling and dataset diversity. The current evaluations use three datasets; testing GEMS on much larger, more heterogeneous behavioral datasets (like SubPoP) would clarify how far the relational approach stretches.

4. Richer graph structure. The current graph uses only subgroup membership and response edges. Adding edges between related questions, temporal dynamics, or social network connections could further improve predictive power and interpretability.

5. Mitigating residual biases. Even though GEMS avoids pretraining bias, subgroup features come from potentially biased demographic categories. Whether the model's learned subgroup embeddings reintroduce stereotyping is an important question the paper touches on but does not fully resolve.

Target Audience

This paper is most valuable to computational social scientists, AI alignment researchers, and machine learning practitioners working on human behavior simulation or survey prediction. It is also relevant to researchers interested in efficient alternatives to LLM fine-tuning, and to anyone studying the trade-offs between predictive accuracy, cost, and transparency in machine learning models of human choices. Readers with a background in recommender systems or graph learning will find the technical framing particularly intuitive.

Authors’ abstract

Large language models (LLMs) have become a popular approach for simulating human behaviors, yet it remains unclear if LLMs are necessary for all simulation tasks. We study a broad family of close-ended simulation tasks, with applications from survey prediction to test-taking, and show that a graph neural network can match or surpass strong LLM-based methods. We introduce Graph-basEd Models for Human Simulation (GEMS) which formulates close-ended simulation as link prediction on a heterogeneous graph of individuals and choices. Across three datasets and three evaluation settings, GEMS matches or outperforms the strongest LLM-based methods while using three orders of magnitude fewer parameters. These results suggest that graph-based modeling can complement LLMs as an efficient and transparent approach to simulating human behaviors. Code is available at https://github.com/schang-lab/gems.