Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

AI summary

Overview

Research area: Computational social science and political science methodology, specifically the reproducibility of research pipelines that use large language models to turn text into data.

Technical level: Intermediate. The chapter assumes familiarity with encoder versus decoder architectures, cross-validation, F1 scores, fine-tuning, and regression-based downstream analysis, but it explains its central argument in conceptual terms.

Scope: This is Chapter 5 of the book LLMs in Political and Social Science: Principles and Applications of Transformer Models (Oxford University Press, forthcoming 2027) by Joan C. Timoneda (Purdue University) and Sebastián Vallejo Vera (University of Western Ontario); it defines replicability, reproducibility, and robustness for LLM-based research, proposes a probabilistic standard for each, and offers practical procedures and disclosure rules for applied social scientists.

What This Paper Is About

LLM output is probabilistic, differs across models and model versions, and can shift the statistical results that researchers ultimately report. The chapter separates this problem into distinct components, asks which components researchers can fix and which they can only measure and report, and supplies concrete procedures, a standardized prompt script, and reporting standards so that LLM-derived findings in political and social science can be checked and trusted.

Key Contributions

1. A precise vocabulary plus a probabilistic standard for replication. Following Barrie et al. (2024), the authors define replicability as the same data, code, and model yielding the same reported results, and add robustness as a separate dimension: the same data analyzed under different reasonable LLM choices (different models, versions, or prompts) yielding similar findings. Reproducibility means the same model applied to new data produces similar findings, and transparency means an outside observer can verify models, data, and methodology. The chapter argues that a replication should be judged successful when it recovers the same distribution of results (performance scores, dispersion, item-level disagreement, downstream coefficients) rather than identical output.

2. A standardized prompt script and a treatment of prompt design as a source of variation. The script fixes role, task, category definitions with positive and negative indicators, boundary-case rules, and a JSON output format, and it slots in few-shot examples (5–10 labeled examples per category) or the rolling-window context of prior annotations. The authors pair this with the rolling-window approach from Section 4.8, where accumulating prior labeling decisions in the context reduces the influence of any single phrasing choice, and with the four factors tested in that section (prompt length, specificity, order of categories, and specified role).

3. Separate replication procedures for encoders and decoders, plus error-structure analysis. For encoders the chapter formalizes 10-times repeated 10-fold cross-validation (100 out-of-sample F1 scores) as already-probabilistic replication; for decoders it proposes repeated same-version runs whose mean, spread, and item-label instability are reported. It then introduces error-structure analysis to diagnose where in the data a model errs and whether different models' errors align, rather than only how often they err.

4. The benchmark-compare-harmonize (BCH) pipeline and a stacked ensemble. The five-step procedure is: (1) use RoBERTa as the encoder benchmark, (2) run an ensemble of open-weight models with few-shot chain-of-thought, (3) QLoRA-fine-tune the best performer as a second benchmark, (4) produce downstream estimates across models to approximate variation in resulting betas, and (5) apply design-based supervised learning (DSL) to correct remaining biases. The abstract reports that a stacked ensemble of open-weight models recovers much of the precision lost in the DSL correction step; the detailed results for that step fall beyond the excerpt available here.

Main Findings

• Measurement choice can reverse a substantive conclusion. Using three datasets, the chapter shows that the choice of LLM measurement can flip the sign of downstream regression coefficients. The names and sizes of these three datasets are not reported in the excerpt available here.

• Existing corrections work but cost precision. Design-based corrections with gold-labeled subsamples restore validity of downstream estimates, but the improvement comes at the cost of estimate precision.

• Decoder replicability is reassuring at the level of distributions. Decoders are especially consistent across same-model runs; Gemma 4 produced F1 scores of 0.5128, 0.5124 and 0.5144 on the political advertising task, with other models behaving similarly. The relevant question is the shape of the distribution, not the identity of any single run.

• Encoder replicability is a problem of data splits, not inference. Inference with a classification-tuned encoder is a fixed sequence of arithmetic operations, but performance depends on where the train-test split falls; with RoBERTa one fold can score 0.81 and another 0.9. Hence the recommendation of 10-times repeated 10-fold cross-validation, whose mean and standard deviation across 100 runs approximate true out-of-sample performance.

• Stability is not accuracy. A perfectly stable measure can be systematically biased, producing a coefficient that is consistent yet wrong across runs. Probabilistic replication plus ensembling makes downstream estimates stable and lowers the variance of downstream betas; DSL remains necessary to address bias, and the chapter states that this combination yields narrower confidence intervals and fewer type-II errors (developed in Section 5.6, beyond the excerpt).

• The reproducibility problem is structural, not unique to LLMs. Human coding has always diverged across coders and even across the same coder over time. Northcutt et al. (2021) audited ten widely used benchmark test sets, including ImageNet and the IMDB sentiment corpus, and found pervasive labeling errors averaging at least 3.3 percent, with higher rates in some datasets — enough to change which models appear best.

• The human-versus-LLM robustness comparison in prior work is asymmetric. Barrie et al. (2024) measure variation across several different models on the LLM side but within a single pool of crowd workers on the human side; the apples-to-apples comparison would set multiple models against multiple pools of coders. Their own evidence also shows their locally installed open-weight Llama implementation varying less than the human coders, which the authors read as pointing to a proprietary-versus-open-weight distinction rather than a general LLM failing.

• Model selection guidance is task-specific. Encoders such as RoBERTa remain strong for annotation of shorter texts with complex categories and for clustering documents in topic modeling; decoders generally fall short on annotation tasks and need QLoRA fine-tuning to approach or beat encoder performance, while complementing encoders by labeling and interpreting topic-model output and enabling semi-supervised variants. Among open-weight decoders, models at or below 9 billion parameters perform admirably but have less general pattern knowledge and weaker reasoning; models in the 20–40B range often offer the best trade-off, with Gemma 4 31B, Llama 4 Maverick, and Mistral 3.2 24B named as strong options at the time of writing (with 7B and 8B equivalents as fallbacks); models at 70B or above are difficult and expensive to deploy locally and their extra reasoning does not reliably translate into task gains.

Methodology in Plain English

The authors treat every LLM annotation as one draw from a distribution of possible outputs, rather than as a fixed measurement. Instead of running a model once and reporting a single F1 score, they run the same task repeatedly on the same model version and prompt, then report the center and spread of the resulting scores, along with how many individual items changed labels between runs. For encoder models, the same logic is implemented through 10-times repeated 10-fold cross-validation, which produces 100 out-of-sample scores; two researchers with the same data, model, and hyperparameters should obtain practically the same distribution.

For the problem of choosing among models, they recommend building a gold-standard, human-coded training or validation set and then treating measurement as a maximization problem against it: run an ensemble of open-weight decoders spanning different developers and sizes with few-shot chain-of-thought, benchmark RoBERTa as an encoder baseline, and QLoRA-fine-tune the best decoder as a second benchmark. They then look at the structure of the errors — where in the data models disagree and whether their mistakes line up — because a higher F1 does not automatically mean a better measure of the underlying concept. Finally, they carry the whole ensemble through to downstream regression estimates so that variation in the betas themselves can be observed, and apply design-based supervised learning with the gold-labeled subsample to correct residual bias. A standardized prompt script and a fixed reporting checklist (exact model version, prompt, temperature and sampling settings, software, access date, seeds, hyperparameters, training data, fine-tuned models and tokenizers) make the whole pipeline auditable.

Why This Matters

The chapter turns a vague worry ("LLM results might not replicate") into a set of separable, diagnosable problems with matching remedies, and it argues that critics sometimes hold LLMs to a standard that human coding never met. It also reframes replication as a distributional claim, which gives researchers a defensible criterion for declaring success — for instance, replication estimates falling within two standard deviations of the reported mean performance, with comparable bands for dispersion, item-level disagreement, and downstream coefficients.

Real-world applications:

• Legislative and political speech analysis — the prompt script's own worked example classifies Chilean congressional speeches from 1980 to 2020 by how they talk about poverty.
• Political advertising research — the task on which Gemma 4's repeated F1 scores were reported.
• Topic modeling and document clustering — using encoders to cluster documents and decoders to label and interpret the resulting topics, including semi-supervised variants.
• Any applied pipeline where a text-derived variable enters a regression — where the chapter shows measurement choices can flip the sign of a coefficient.

Industry relevance: the chapter gives concrete guidance on version pinning (for example, API models such as GPT-5.5, GPT-5.5-pro, and GPT-5.5-chat-latest, where the "latest" variants silently update), on preferring open-weight models that can be saved and re-run, and on cost, noting GPU rentals in the range of 0.5 to 1.5 dollars per hour for the models and tasks tested. These practices map directly onto internal model governance, evaluation, and audit workflows outside academia.

Future Directions

• A symmetric human-versus-model robustness comparison. The authors explicitly ask whether the human side would look as stable if the task were re-run each month with three fresh research assistants, or with assistants trained in-house rather than working only from written instructions.
• Direct testing of prompt stability. The chapter observes that Barrie et al. (2024) discuss prompt stability but do not test it explicitly in their design, leaving room for both honest error and selective reporting (Baumann et al., 2025).
• How much precision the stacked ensemble recovers, and under what conditions. The abstract claims substantial recovery of precision lost in the correction step; the excerpt does not report the size of that recovery, and it notes that fine-tuning gains and model rankings are task-specific and likely to shift as the state of the art changes.
• Extending probabilistic replication to closed-weight models and future model generations. The chapter notes that the persona/role element of prompts has shown little effect on outcomes (Zheng et al., 2024, confirmed in Section 4.8) but standardizes it anyway "to guard against unknown sources of variation in future models," and flags that closed models aggravate the problem because researchers have little control over the deployment environment.

Target Audience

Applied researchers in political science and the social sciences who use LLMs to produce variables for statistical analysis; graduate students and instructors working through the book's pipeline chapters; methodologists and journal reviewers evaluating reproducibility claims; and applied NLP practitioners in industry who need to document model versions, prompts, and evaluation distributions for auditable, version-pinned systems.

Authors’ abstract

This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.