Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based human behavior simulation (computational social science), probability calibration, and low-resource/multilingual NLP (Luxembourgish).

Technical level: Intermediate. The core concepts (accuracy, calibration, Expected Calibration Error) are explained from first principles, but readers benefit from familiarity with LLM inference and supervised fine-tuning.

Scope: A case study testing whether open-weight LLMs' token probabilities can serve as trustworthy confidence estimates when predicting individual users' like/dislike behavior on Luxembourgish news-comment data, and whether task-specific fine-tuning fixes the observed miscalibration.

What This Paper Is About

When researchers use LLMs to simulate human behavior, they usually judge success by accuracy alone, which implicitly assumes human behavior is deterministic. The authors argue that on noisy digital trace data, a model must also know when it is likely wrong — so its stated confidence should match its actual accuracy. They test this on a "Next Like Prediction" task: given a user's past likes and dislikes on the Luxembourgish news platform RTL.lu, predict whether that user will like or dislike a new comment, and check whether the model's token probabilities are usable as confidence scores.

Key Contributions

1. Confidence as an evaluation axis. The authors argue that alignment for generative agents should be measured through calibration alongside accuracy, and operationalize this by normalizing the token probabilities of the two binary answer tokens (0 and 1) into a confidence estimate for a human behavior simulation task.

2. Demonstrating that accuracy and calibration diverge. On the Next Like Prediction task, both tested off-the-shelf models are overconfident and are beaten on both accuracy and Expected Calibration Error (ECE) by a trivial statistical baseline. Scaling from 3B to 8B improves accuracy while worsening calibration, and longer behavioral histories raise accuracy while confidence falls.

3. Fine-tuning as a calibration intervention. Task-specific supervised fine-tuning sharply reduces calibration error and widens the gap between confidence on correct versus incorrect predictions, showing that lightweight adaptation can align predicted probabilities with observed accuracy.

4. A diagnostic toolkit for the task. The paper combines accuracy, ROC-AUC, Brier score, Top-1 ECE, and conditional confidence (confidence on correct vs. incorrect predictions), plus a contextual-scaling analysis of how accuracy and confidence each change with history length.

Main Findings

• Off-the-shelf models underperform a naive baseline. The statistical baseline (predicting from the user's historical like rate) reaches accuracy 0.5900, ECE 0.0642, and Brier score 0.2476. Ministral 3 3B reaches accuracy 0.4906 with ECE 0.1650; Ministral 3 8B reaches accuracy 0.5355 with ECE 0.2004. Neither off-the-shelf model beats the baseline on accuracy or calibration.

• The models are confidently wrong. The 8B model assigns a mean probability of 0.7202 to incorrect predictions, close to its 0.7494 confidence on correct ones. The 3B model shows confidence 0.6482 on incorrect predictions against overall accuracy of only 0.4906. The baseline's confidence on incorrect predictions is 0.6346.

• Brier scores indicate severe overconfidence. The 8B model's Brier score of 0.2937 is described as worse than simply predicting a constant 0.5 probability for every interaction.

• Bigger is not better calibrated. Going from 3B (accuracy 0.4906, ECE 0.1650) to 8B (accuracy 0.5355, ECE 0.2004) improves accuracy but raises ECE — accuracy and calibration move apart.

• Fine-tuning fixes both, but mainly calibration. Ministral 3 8B with SFT reaches accuracy 0.7578, ROC-AUC 0.8253, Brier score 0.1618 and ECE 0.0056. Ministral 3 3B with SFT reaches accuracy 0.7089, ROC-AUC 0.7600, Brier score 0.1880 and ECE 0.0087 — near-parity in calibration at much smaller scale.

• Fine-tuning widens the confidence gap. For 8B SFT, confidence on correct predictions rises to 0.7828 while confidence on incorrect predictions falls to 0.6616, which the authors read as the model lowering its probability estimates when it is more likely to be wrong.

• Longer histories help accuracy but hurt confidence for off-the-shelf models. For 8B, accuracy correlates with history length at r = 0.50 while confidence correlates at r = -0.64; for 3B the confidence correlation is r = -0.71. After SFT, both become positive and significant for 8B (accuracy r = 0.33, confidence r = 0.43).

• A proprietary model is accurate but unmeasurable. Gemini 3.5 Flash-Lite reaches accuracy 0.6868 — above the statistical baseline and both off-the-shelf models, but below the fine-tuned 8B — yet its API withholds logits, so no calibration metrics (ROC-AUC, Brier, ECE) are reported for it.

Methodology in Plain English

The task. The researchers reconstructed which RTL.lu users liked or disliked which comments. For each user, they formatted earlier interactions as prior turns in the model's chat history, then asked the model to predict the next one as a 0 (dislike) or 1 (like). Context windows ranged from a minimum of 3 prior interactions up to 72. Evaluation always conditions on the user's real history, never on the model's own earlier predictions, so errors do not compound.

The data. Comments span March 2020 to May 2024. The top 20% of users by interaction count were removed to avoid overfitting to power users, and users with fewer than four interactions were excluded. Splitting was done at the user level so no user appears in both train and test: 3,247 training users with 57,185 training cases, and 783 test users with 13,759 test cases. The class balance is 61% likes and 39% dislikes; users average 20.5 interactions (median 13) and comments average 49 words. Comment languages are Luxembourgish (75%), English (15%), French (4%) and German (3%).

Turning probabilities into confidence. The model is forced to answer with a single token, 0 or 1. The authors apply a softmax over the vocabulary and renormalize the probabilities of those two tokens so they form a proper binary distribution. This confidence is then compared against whether the prediction was actually right.

The measures. Accuracy for discrete performance; Top-1 ECE with 10 confidence bins for calibration; Brier score to penalize overconfident errors; ROC-AUC for threshold-free discrimination; and "conditional confidence" — mean confidence on correct predictions versus mean confidence on incorrect ones.

The comparisons. Three points of reference: a naive statistical baseline that predicts each user's historical like rate; Ministral 3 in 3B and 8B versions used off-the-shelf with in-context learning only; and the same two models after supervised fine-tuning. All fine-tuning runs used identical settings — one epoch, maximum sequence length 4,500 tokens, the paged AdamW optimizer with 8-bit quantization, a single NVIDIA L40S GPU (48GB VRAM), and TRL defaults otherwise. Gemini 3.5 Flash-Lite with reasoning disabled served as a proprietary accuracy-only reference.

Why the task is hard by design. Three sources of irreducible ambiguity are named: linguistic complexity (Luxembourgish with heavy code-switching and spelling variation), contextual sparsity (the underlying news articles are deliberately withheld, and only 3 to 72 prior interactions are available), and behavioral non-determinism (offline events and day-to-day whims are invisible, and a user might upvote a comment one day and ignore a similar one another day). Perfect prediction is explicitly not expected.

Why This Matters

Impact on research. The paper reframes what counts as a valid LLM-based behavioral simulation: reporting accuracy alone can actively mislead, because in this study accuracy and calibration moved in opposite directions under both model scaling and context scaling. It also supplies a cheap, transparent sanity check — if a naive historical-frequency baseline beats your LLM on accuracy and calibration, the inference-only pipeline, not the model architecture, is likely the problem.

Real-world applications:

• Comment moderation and ranking. A calibrated predictor of what users will endorse could support ranking or surfacing comments while exposing genuine uncertainty rather than false confidence.
• Platform engagement modeling. Distinguishing "the model is unsure" from "the model is confidently wrong" changes how downstream decisions should be weighted.
• Low-resource language tooling. The work contributes an evaluation of confidence estimation on authentic Luxembourgish text, a language the paper describes as data-rich but method-poor for NLP.
• Survey and social-science simulation. Calibrated per-prediction confidence lets researchers aggregate individual predictions probabilistically instead of treating each as a hard fact.

Industry relevance. Anyone deploying LLMs on noisy user-generated or non-English text — where the model drifts from its pre-training distribution — faces the same failure mode documented here: high stated confidence on wrong answers. The paper also makes a practical argument for open-weight models, since closed APIs that hide logits make calibration measurement, and therefore uncertainty-aware deployment, impossible.

Future Directions

• Move beyond binary actions to open text. The current setup collapses behavior into a single-token 0/1 decision; extending calibration measurement to free-text generation requires going past single-token estimates, since the authors note perplexity is an inadequate proxy for uncertainty in unconstrained generation.
• Test internal-state and other uncertainty methods. Because token probabilities suffer from surface-form competition and first-token deviations, the authors point to probing internal activations and mechanistic interpretability techniques (circuit analysis, sparse autoencoders) as promising routes to better confidence signals.
• Evaluate multi-turn, self-conditioned simulations. Every experiment here is single-turn and conditioned on ground-truth history. How uncertainty compounds once a model conditions autoregressively on its own synthetic outputs is untested and, in the authors' framing, the central open challenge.
• Benchmark on out-of-distribution behavioral data. Whether the observed overconfidence and its correction under fine-tuning generalize to other model families, other languages, and other noisy behavioral datasets is explicitly left open, as the study covers one model family (Ministral 3) and one dataset.

Target Audience

Researchers in computational social science and LLM-based agent simulation who need to justify their evaluation choices; NLP practitioners working with multilingual, code-switched, or low-resource text; ML engineers deploying models on noisy user-generated data who need to know when to trust a prediction; and calibration and uncertainty-estimation researchers looking for an applied, non-English case study. Readers focused purely on state-of-the-art accuracy benchmarks will find the paper's value is in its negative results and its evaluation framing rather than in a new model.

Authors’ abstract

As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.