Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting

AI summary

Overview

• Research area: Machine learning and computational social science — specifically, mechanistic interpretability applied to LLM-based political preference prediction (election forecasting).
• Technical level: Advanced. The paper assumes familiarity with transformer internals (residual streams, MLP decomposition, value/key vectors, unembedding matrices), linear probing, and distributional distance metrics.
• Scope: A single paper introducing "mechanistic forecasting," which predicts human political preferences by aggregating party-aligned activations of internal MLP value vectors rather than relying on next-token output probabilities, tested across 7 models, 6 national elections, and more than 24 million persona configurations.

What This Paper Is About

When researchers prompt LLMs with synthetic personas to forecast public opinion, they almost always judge the model by its final output — the probabilities it assigns to words like a party name. The authors argue that this surface-level view is the bottleneck: models may encode richer, more structured knowledge about human preferences internally than they ever express in generated answers. The paper asks whether reading a model's internal representations directly can produce more accurate population-level preference forecasts than reading its outputs, using election forecasting as the test case.

Key Contributions

1. Introduces mechanistic forecasting. A method that identifies party-aligned value vectors in MLP layers, measures how persona prompts activate them, and aggregates those activations into group-level preference distributions directly comparable to representative survey data.
2. Demonstrates exploitable latent information at scale. Across models, countries, and parties, the authors show that LLMs encode substantial latent information about political preferences that is not reliably expressed in output probabilities, and that exploiting this signal often improves party-level forecasts against real-world survey benchmarks.
3. Documents systematic differences across persona attribute categories. Demographic attributes (e.g., age, education, employment) are often better captured by latent estimators than by output probabilities, while opinion-based attributes show greater cross-national variation.
4. Proposes a gating criterion. Attribute-level normalized entropy of output probability distributions is identified as a simple criterion for deciding when mechanistic forecasting should be preferred over probability-based prediction.

Main Findings

• Probes reliably capture party structure. Probe F1 scores consistently exceed 96% on a 10% hold-out split. Control conditions perform near chance (macro-F1 ≈ 0.33 for randomly shuffled labels; ≈ 0.24 for random-weight probes), and regression baselines on the same activations reach 0.93 ± 0.05 versus 0.99 ± 0.01 for the value probe (p < 0.001).
• Latent aggregation often beats output probabilities. Across models and national contexts, preference distributions derived from party-aligned latent activations are frequently closer to real-world survey data than those derived from next-token probabilities.
• Instruction tuning widens the gap. Comparing fine-tuned to base model versions, win-rates increase further for most countries and across models, suggesting alignment shifts estimations away from real-world survey predictions and makes the latent approach relatively more effective.
• Gains are not explained by external data access. An in-context learning baseline that receives the same party-position data through the prompt often remains worse than mechanistic forecasting across models.
• Party-level errors differ substantially. For nearly all parties and models, some mechanistic forecasting estimators improve party-share predictions for at least some categories. Overall error levels are broadly comparable across models, and the contribution of latent information becomes more variable as estimation error grows — latent signals matter most for parties that are hard to predict from probabilities alone.
• Demographic attributes benefit most, and most consistently in the US and UK. Age, education, and employment show high win-rates in both countries. Household income shows strong improvements in Canada but mixed performance in the United States, and demographic attributes are less well captured in New Zealand. Germany and the Netherlands show comparatively uniform performance across demographic categories.
• Opinion-based attributes vary more across countries. Latent gains for political orientation are strongest in the United Kingdom but substantially weaker in the Netherlands. Germany shows comparatively low win-rates for immigration despite strong demographic performance, while New Zealand shows the opposite pattern.
• High entropy predicts where latents help. Mechanistic forecasting is most effective for attributes with high normalized entropy — where no single category dominates and accurate prediction requires aggregating weak, distributed signals. This is especially pronounced for instruction-tuned models, whose output probabilities concentrate mass on a small subset of categories.
• Entropy gating yields consistent gains. Evaluating mechanistic forecasting only for attributes with normalized entropy exceeding 0.85 reveals consistent median improvements in prediction error across attributes and models. In low-entropy settings, gains are correspondingly limited.

Methodology in Plain English

The authors treat a transformer MLP as a sum of "sub-updates": static value vectors, each multiplied by an input-dependent scaling number. The direction of a value vector determines which tokens it promotes or suppresses; the scaling number determines how strongly it fires for a given input. This separation lets them ask two separate questions — which internal directions are associated with a political party, and how strongly a persona activates them.

First, they train simple linear probes on mean-pooled residual-stream representations from intermediate layers (specifically the 0.5L–0.9L window, chosen via an ablation because it had the highest lower whisker in the selectivity-gap distribution across seeds, models, and countries). Probe training data comes from voting-advice applications: Wahl-O-Mat for Germany, StemWijzer for the Netherlands, and Vote Compass for the United Kingdom, United States, Canada, and New Zealand.

Second, they compare each probe's weight vector against every MLP value vector using cosine similarity, and select value vectors whose similarity lies outside a 2.5-IQR fence relative to other directions in the same layer. Vectors are split into probe-aligned (positive cosine) and diametric (negative cosine). Each selected vector is validated with a sign-inversion test: flipping the sub-update's sign and checking whether the log-probability of the party token drops. A separate intervention on 1,280 non-political persona-question pairs shows that scaling the selected vectors significantly raises the party-token logit versus a no-scaling baseline (paired t-test, p < 0.001), while a matched random-vector control of equal magnitude shows no significant effect.

Third, they build persona prompts from attributes such as age, gender, education, household income, employment, political orientation, and issue positions on immigration and inequality. Each persona is instantiated with 10 independently designed prompt templates (hand-crafted in German, then machine-translated and verified by native speakers), yielding 280,000 persona combinations whose attribute marginals match the corresponding real-world survey.

For each prompt, they record the scaling coefficients of the selected party value vectors, weight them by cosine similarity, average over vectors to get a per-party activation score, and aggregate across personas within each attribute category using survey-matching sampling weights. This produces an attribute-level distribution over parties. They compare three normalized distributions — latent activation-based, next-token probability-based, and weighted survey-based — using Jensen–Shannon distance for nominal attributes and first Wasserstein distance for ordinal attributes. An attribute counts as a "win" for mechanistic forecasting when the probability-based distance minus the latent distance is positive.

Why This Matters

Impact on research. The paper reframes LLM-based social simulation from a prompting problem into a representation-aware estimation problem. It extends hidden-knowledge research — which has focused on binary factual correctness — to distributional, contextual preference prediction, where ground truth is a survey distribution rather than a verifiable answer. It also provides a concrete, reproducible method (with code released) rather than a purely diagnostic finding.

Real-world applications:

• Election and polling augmentation. Supplementing expensive representative surveys with model-derived estimates of conditional vote shares, particularly for demographic subgroups.
• Market and consumer research. Applying the same persona-activation pipeline to non-political preferences, the setting the authors use for their control intervention.
• Survey methodology. Using the entropy gate as a diagnostic for which survey items or subgroups an LLM can be trusted to estimate, and which require human respondents.
• Model auditing and evaluation. Treating the gap between latent and surface-level knowledge as a measurable signal of what alignment and instruction tuning suppress.

Industry relevance. Any organization using LLMs as synthetic respondents — polling firms, marketing agencies, product researchers — now has a reason to inspect internal representations rather than only outputs. The entropy criterion gives practitioners an actionable, cheap screen before investing in latent-space analysis, and the finding that instruction tuning widens the latent-versus-surface gap is directly relevant to teams choosing between base and aligned models.

Future Directions

• A principled framework for prompt variation. The authors note that no criterion currently exists for determining how many prompt paraphrases are sufficient; they use 10, exceeding comparable studies, but explicitly call for future work to establish a rigorous design framework.
• Tuning the layer window. The 0.5L–0.9L range is presented as a robust default identified by ablation, not a universal optimum, and the authors state it should be treated as a tunable hyperparameter.
• Extending beyond elections. The paper argues the representation-aware approach has implications beyond the electoral setting for estimating collective human preferences, but only tests the electoral case.
• Explaining cross-national variation. The reasons why latent gains vary so much between countries — for example, strong for political orientation in the United Kingdom but weak in the Netherlands — remain unresolved, and the authors link them tentatively to how explicitly each dimension is expressed in surface-level predictions within each national context.
• Refining the gating rule. The entropy threshold of 0.85 is presented as a practical filter; whether it generalizes across domains and model families is not established.

Target Audience

Researchers in mechanistic interpretability and computational social science who work on LLM-based survey simulation; survey methodologists evaluating whether language models can stand in for human respondents; and applied machine learning engineers at organizations deploying personas or synthetic respondents, who need to know when internal representations outperform generated answers. Readers without a background in transformer internals or distributional statistics will find the methodology sections demanding, though the results sections are accessible.

Authors’ abstract

Large language models are increasingly used to predict human preferences in both scientific and business endeavors, yet current approaches rely exclusively on analyzing model outputs without considering the underlying mechanisms. Using election forecasting as a test case, we introduce mechanistic forecasting, a method that demonstrates that probing internal model representations offers a fundamentally different - and sometimes more effective - approach to preference prediction. Examining over 24 million configurations across 7 models, 6 national elections, multiple persona attributes, and prompt variations, we systematically analyze how demographic and ideological information activates latent party-encoding components within the respective models. We find that leveraging this internal knowledge via mechanistic forecasting (opposed to solely relying on surface-level predictions) can improve prediction accuracy. The effects vary across demographic versus opinion-based attributes, political parties, national contexts, and models. Our findings demonstrate that the latent representational structure of LLMs contains systematic, exploitable information about human preferences, establishing a new path for using language models in social science prediction tasks.