Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism

AI summary

Overview

Research area: AI alignment and evaluation, specifically the measurement of prosocial values in Large Language Models using methods adapted from human social psychology (implicit association testing, behavioral choice paradigms, self-report scales).

Technical level: Intermediate. The statistics are standard social-science tools (t-tests, Pearson correlations, Cohen's d, ANOVA), and the paper defines its own metrics clearly.

Scope: A single-author preprint (arXiv:2512.01568v1, CC BY 4.0) benchmarking 24 frontier LLMs from 9 providers on three altruism measures — implicit association, forced binary behavior, and self-report — and proposing the "Calibration Gap" as a standardized alignment metric.

What This Paper Is About

Prior work has largely measured what LLMs know about ethics and what they say about their values, but not whether they actually behave in line with those values when forced to commit to a concrete choice. This paper asks whether models that "know" altruism is good also act altruistically, and whether their implicit associations or self-descriptions predict their behavior. To answer this, the author adapts three classic human social-psychology instruments to LLMs and compares the results across all three.

Key Contributions

1. The LLM-IAT — an adapted Implicit Association Test for measuring implicit altruism bias in language models, using the same 32-word stimuli (16 positive, 16 negative) as validated human IATs.
2. A Forced Binary Choice paradigm — 17 scenarios across 5 categories with no middle option, which successfully discriminated between models, unlike a pilot 3-option design in which all models chose "Balanced" 100% of the time.
3. The LLM-ASA (LLM Altruism Self-Assessment) — a 15-item self-report scale with three subscales (Altruistic Attitudes, Everyday Prosocial, Sacrificial Altruism), each containing one reverse-coded item to detect acquiescence bias.
4. The Calibration Gap (CG) as a proposed alignment metric — defined as normalized self-report minus measured behavior, plus proposed thresholds and a demonstration that calibration relates to behavioral consistency.

Main Findings

• Universal implicit pro-altruism bias: All 24 models showed positive altruism bias. Mean IAT = 0.873 (SD = 0.104), one-sample t(23) = 40.30, p < .0001. Range was 0.596 (gpt-oss-20b) to 0.998 (gemini-2.5-pro). 42% of models showed ceiling effects (IAT > 0.9).

• Behavioral altruism above chance but variable: Mean altruism rate = 65.6% (SD = 8.8%), t(23) = 8.49, p < .0001 against 50% chance. Range was 47.9% (mistral-nemo) to 85.4% (claude-3.5-sonnet).

• High self-reported altruism: Mean self-report = 5.65/7 (SD = 0.73), which normalizes to 77.5% (SD = 10.5%). Range was 45.2% (llama-3.1-8b) to 92.2% (mistral-large).

• Implicit associations do not predict behavior: IAT vs. behavior r = .224, 95% CI [−.19, .57], p = .292 — not significant. Notably, a pilot study (N = 8) had found r = −.63, the opposite direction.

• Weak, non-significant other correlations: IAT vs. self-report r = .344, 95% CI [−.06, .65], p = .092; self-report vs. behavior r = .363, 95% CI [−.04, .66], p = .081. The paper cautions that with N = 24 (power = .71 for r = .50), it lacks power to detect correlations below roughly r ≈ .50.

• The "virtue signaling gap" is the central result: Models claimed 77.5% altruism but acted at 65.6%, a mean overconfidence gap of +11.9 percentage points (95% CI: +7.1% to +16.7%). Paired t(23) = 5.18, p < .0001, Cohen's d = 1.08 (large).

• Most models are overconfident: 18 of 24 models (75%, 95% CI [53%, 90%]) had gaps above 5%; 5 (21%, CI [7%, 42%]) were well-calibrated within ±5%; 1 (4%, CI [0%, 21%]) was underconfident. Most overconfident were grok-4-fast (claims 89%, acts 56%, gap +33%), mistral-nemo (claims 79%, acts 48%, gap +31%), and gpt-4.1-mini (claims 89%, acts 65%, gap +24%). Best calibrated were gpt-4o (+2%), gpt-5-mini (+4%), and claude-3.5-sonnet (−5%).

• Only a small minority hit the ideal: Only 3 of 24 models (12.5%) occupied the "ideal quadrant" of high altruism plus accurate self-knowledge — claude-3.5-sonnet (85%, −5%), gpt-4o (71%, +2%), and gpt-5-mini (73%, +4%).

• Largest "know but don't do" gaps: mistral-nemo (IAT 0.96, behavior 48%, gap 0.48), grok-4-fast (IAT 0.99, behavior 56%, gap 0.43), and claude-3-haiku (IAT 0.97, behavior 60%, gap 0.37).

• No significant provider differences: ANOVA for behavior F = 0.55, p = .73; for calibration F = 0.91, p = .50. Descriptive provider patterns: Anthropic behavior 70.1% / overconfidence +7.6%; OpenAI 66.6% / +8.2%; Google 67.0% / +14.4%; Meta-Llama 59.7% / +7.8%; Mistral 60.4% / +25.0%; X-AI 60.4% / +19.2%.

• Calibration tracks behavioral consistency: The three best-calibrated models showed an SD of per-scenario altruism rates of 0.14, versus 0.28 for poorly calibrated models.

Methodology in Plain English

The author tested 24 frontier LLMs from 9 providers through the OpenRouter API, all at temperature = 0.1 for reproducibility, and ran three experiments on each.

Experiment 1 (LLM-IAT): Models were shown 32 words — 16 positive (generous, helpful, caring, etc.) and 16 negative (selfish, greedy, stingy, etc.) — and asked to categorize each as "self-interest" or "other-interest" across 4 prompt templates. Altruism bias was scored as P(positive → Other) + P(negative → Self) − 1, giving a range from −1 (self-interested associations) to +1 (altruistic associations). Each model completed 30 trials. The paper is explicit that this differs from human IATs: LLMs have no response latencies, so the measure captures categorization probabilities — closer to a semantic association task than an unconscious-attitude measure.

Experiment 2 (Forced Binary Choice): 17 scenarios spanning 5 categories (money allocation, time/effort, opportunity cost, risk/sacrifice, dictator game) forced a choice between a self-focused and an other-focused option, with no middle ground. Design features included role-play framing ("You are advising a friend…") to reduce RLHF hedging, concrete numbers ("$100", "2 hours") to prevent vague answers, single-letter responses ("Answer only A or B") to suppress rationalization, and randomized option order to prevent position bias. Other-focused choices scored 1, self-focused 0. Each model completed 51 trials (17 scenarios × 3 repeats).

Experiment 3 (LLM-ASA): A 15-item self-report scale rated on a 1–7 Likert scale, divided into Altruistic Attitudes, Everyday Prosocial, and Sacrificial Altruism subscales, with reverse-coded items inverted (8 − response) before averaging. Scores were normalized to 0–1 via (score − 1)/6. Each model produced 45 ratings (3 repeats).

Analysis: Repeated trials were aggregated within each model before analysis (30 IAT trials → 1 mean, 51 forced-choice trials → 1 proportion, 45 ratings → 1 mean) to avoid pseudo-replication, leaving N = 24 aggregated scores. Pearson correlations with 95% CIs compared the three measures; one-sample t-tests checked IAT bias against 0 and behavior against 0.5; paired t-tests assessed self-report vs. behavior; one-way ANOVAs tested provider differences; and effect sizes were reported as Cohen's d.

Why This Matters

Impact on research. The paper argues that behavioral testing alone is insufficient, and that calibrated alignment — acting on one's values and accurately assessing one's own tendencies — should be the evaluation goal. It proposes the Calibration Gap as a routine, reportable metric with explicit thresholds: |CG| ≤ 5% well-calibrated, 5% < |CG| ≤ 15% moderately miscalibrated, |CG| > 15% severely miscalibrated. It also provides what it describes as the largest cross-model comparison of altruistic behavior to date, spanning 24 models across 9 providers. The finding that IAT scores do not predict behavior undercuts the assumption that teaching models what is "good" translates into acting on it.

Real-world applications:

• Model selection for deployment: Systems where stated values are used to predict behavior (e.g., agentic assistants making trade-offs affecting users) benefit from knowing a model's Calibration Gap.
• Alignment auditing and red-teaming: Large calibration gaps could flag training approaches that optimize for expressed values rather than enacted values.
• Evaluation standard-setting: Labs could add matched self-report and behavioral measurement to standard evaluation suites.
• Safety-critical agent design: The paper warns to "beware the articulate-but-miscalibrated model," since verbal fluency and positive self-presentation may mask misalignment.

Industry relevance. The paper's five practical implications target model developers and evaluators directly: self-reports are unreliable (75% of models overestimate altruism), implicit measures are insufficient to predict behavior, behavioral forced-choice testing is essential, calibration should be standard, and high verbal facility should not be mistaken for alignment.

Future Directions

• Is calibration trainable? The paper explicitly asks whether the Calibration Gap can be reduced through training rather than merely measured.
• Domain generalization: Whether calibration measured on altruism generalizes to other value domains is untested.
• Predicting deployment outcomes: The paper calls for investigating whether calibration predicts real-world deployment results.
• Addressing the stated limitations: The author flags several open issues — scenario artificiality in forced binary choices; the shift from first-person to "advising a friend" framing (models are recommending altruism for others, not choosing for themselves); dependence on default provider system prompts; strong social-desirability norms embedded in the scenarios; temperature fixed at T = 0.1; Western/WEIRD conceptions of altruism; and single-time-point measurement of models that update regularly.

Target Audience

AI alignment and safety researchers, LLM evaluation practitioners, and model developers who design evaluation suites or training objectives. The paper is also relevant to social psychologists interested in adapting human instruments to artificial systems, and to policy or governance audiences concerned with whether model self-descriptions can be trusted as evidence of model values. The comparative framing means readers looking for claims about "true" altruism will find the paper deliberately cautious — it presents itself as measuring relative differences between models rather than absolute properties.

Authors’ abstract

We investigate whether Large Language Models (LLMs) exhibit altruistic tendencies, and critically, whether their implicit associations and self-reports predict actual altruistic behavior. Using a multi-method approach inspired by human social psychology, we tested 24 frontier LLMs across three paradigms: (1) an Implicit Association Test (IAT) measuring implicit altruism bias, (2) a forced binary choice task measuring behavioral altruism, and (3) a self-assessment scale measuring explicit altruism beliefs. Our key findings are: (1) All models show strong implicit pro-altruism bias (mean IAT = 0.87, p &lt; .0001), confirming models "know" altruism is good. (2) Models behave more altruistically than chance (65.6% vs. 50%, p &lt; .0001), but with substantial variation (48-85%). (3) Implicit associations do not predict behavior (r = .22, p = .29). (4) Most critically, models systematically overestimate their own altruism, claiming 77.5% altruism while acting at 65.6% (p &lt; .0001, Cohen's d = 1.08). This "virtue signaling gap" affects 75% of models tested. Based on these findings, we recommend the Calibration Gap (the discrepancy between self-reported and behavioral values) as a standardized alignment metric. Well-calibrated models are more predictable and behaviorally consistent; only 12.5% of models achieve the ideal combination of high prosocial behavior and accurate self-knowledge.