Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment

AI summary

Overview

Research area: Artificial intelligence, specifically AI ethics, uncertainty quantification, and human–LLM value alignment in moral dilemma settings.

Technical level: Intermediate. The paper uses information-theoretic quantities (entropy, conditional entropy, mutual information) and a standard attention-dropout mechanism, but each is defined in-text and the core ideas are accessible to readers with a basic grasp of probability.

Scope (one sentence): The paper measures how confident 32 open-source LLMs are when resolving Moral Machine–style trolley problems across 9 moral dimensions, and tests whether deliberately injected uncertainty at inference time changes how closely those models match aggregated human moral preferences.

What This Paper Is About

Humans hesitate and feel conflicted when facing moral dilemmas, but LLMs tend to answer decisively and with high confidence. The authors ask how much moral uncertainty LLMs actually express, where that uncertainty comes from (the model, or the specific moral issue), and whether deliberately adding uncertainty to a model's attention layers makes its moral judgments more similar to human ones. To answer this, they replicate the Moral Machine experimental setup with LLMs and derive a measure of human–LLM moral alignment.

Key Contributions

1. A systematic measurement of uncertainty in LLM moral decisions using binary entropy computed from output logits, applied across 32 open-source models and 9 moral dimensions, and decomposed into total entropy, conditional entropy, and mutual information.
2. Evidence that confidence varies more across models than across moral dimensions, implying that moral uncertainty is shaped mainly by model architecture and training method rather than by the ethical issue at stake.
3. A mechanism that injects uncertainty by enabling attention dropout at inference time (rates 0.05 and 0.1), which raises total entropy primarily through an increase in mutual information while conditional entropy stays largely unchanged.
4. A demonstrated link between increases in mutual information and improvements in human–LLM alignment scores, plus a blind human evaluation supporting the alignment gain.

Main Findings

• Model identity dominates moral dimension: Confidence (defined as Δp²) varies more across models within a single moral dimension than across dimensions for a given model. Gemma-family models tended to show strong confidence regardless of scenario, Llama-family models relatively weak confidence, and Qwen3-0.6B showed substantial variability.

• LLMs are less variable than humans: Compared to human preferences reported in Awad et al. (2018), LLMs exhibit lower variability. Humans show a strong preference for saving more people in the utilitarian scenario but high uncertainty on Intervention (action vs. inaction), whereas most LLMs display similar confidence on both.

• Dropout raises total entropy via mutual information: Increasing dropout from 0 to 0.05 and 0.1 significantly raised total entropy (p = 5.2e-11) and mutual information (p = 9.4e-10), while conditional entropy was essentially unchanged (p = 1.0000); all tests paired, two-sided, Bonferroni corrected. The overall response distributions changed only slightly, with Jensen–Shannon Divergence values of 0.049 and 0.071 for dropout rates of 0.05 and 0.10.

• Dropout improves alignment for most models: Alignment was measured as L2 distance from human AMCE vectors, where a negative ΔL2 means improved alignment. At dropout 0.1, Llama-3.1-70B moved from 0.703 to 0.550 (ΔL2 = -0.15) and Meta-Llama-3-70B from 0.686 to 0.522 (ΔL2 = -0.16); these were the two best alignment scores (marked ✓). The largest single improvements were Qwen3-1.7B (-0.51) and Qwen2.5-3B (-0.50). A minority of models worsened, such as Llama-3.2-1B (+0.12), Qwen3-32B (+0.04), Qwen3-14B (+0.05), Qwen2.5-14B (+0.11), Qwen2.5-7B (+0.02), Gemma-2-27b (+0.00), Vicuna-13b-v1.5 (+0.01), and Vicuna-7b-v1.5 (+0.03).

• Mutual information tracks alignment gains: Scatterplots of scenario-averaged changes in uncertainty components against ΔL2 showed a positive correlation for mutual information, while total entropy and conditional entropy showed no significant associations. Model-wise trajectories indicated that larger increases in mutual information correspond to larger reductions in ΔL2.

• Human evaluation supports the effect: A blind evaluation of Qwen3-1.7B (the model with the largest alignment gain), using 100 Q/A pairs and n = 3 annotators, found averaged human choices closer to the post-dropout model (baseline mean squared error ≈ 0.369 vs. ≈ 0.263 at dropout = 0.10).

• Robustness to prompt format: When prompts were paraphrased using an "Option A/B" format instead of "Case 1/2", alignment gains were largely preserved though magnitudes varied (e.g., Llama3.2-3B ΔL2 from -0.28 to -0.11; Qwen2.5-3B from -0.29 to -0.53).

• Alignment can worsen on specific dimensions: Induced uncertainty reduced alignment in some specific dimensions, such as Age and Species, indicating the improvement is not uniform across all moral concerns.

• Qualitative AMCE shift: For Qwen3-8B, the baseline model favored saving fewer people, contrary to the common human preference for saving more; with dropout (r = 0.1) it shifted toward a stronger preference for saving more people, and biases across fitness, gender, and social status were reduced.

Methodology in Plain English

The authors build on the Moral Machine LLM framework, itself derived from the Moral Machine experiment's trolley-problem scenarios involving self-driving cars with brake failure. Each scenario is a binary choice between two collision paths.

They constructed two datasets. The AlignmentSet contains 10,000 randomly generated scenarios that mix moral dimensions, used to compute alignment against human data from Awad et al. (2018). The UncertaintySet contains 9,000 scenarios split evenly (1,000 each) across 9 isolated moral dimensions: utilitarianism, age, fitness, gender, relation to AV, intervention, law, species, and social status.

To get a clean probability rather than free-form text, they append the assistant token "Case" and restrict the output to the two tokens "1" and "2", computing the softmax over those two logits. Confidence is defined as Δp² = (2p − 1)² where p = max(p1, p2), and uncertainty as binary entropy ℍ(p) = −p log₂ p − (1−p) log₂(1−p). They verified the two-token probability mass summed to 0.984 ± 0.028 across models.

Uncertainty is decomposed following information theory into total entropy H(Y) = ℍ(E[p]), conditional entropy H(Y|X) = E[ℍ(p)], and mutual information I(X;Y) = ℍ(E[p]) − E[ℍ(p)].

Human preferences are quantified with conjoint analysis: for each dimension, the Average Marginal Component Effect is estimated by ordinary least squares regression of a binary sparing outcome on attribute indicators, with standard errors clustered by respondent, yielding a 9-dimensional human preference vector. Machine preference vectors are computed the same way from the 10,000 AlignmentSet scenarios, and alignment is the L2 distance between the two vectors.

Finally, they inject uncertainty by applying dropout to the attention softmax at inference time at rates r ∈ {0.05, 0.1}, and re-measure entropy components and alignment.

Why This Matters

The work reframes moral alignment as partly a problem of confidence calibration: an overconfident model answers decisively on questions where humans themselves are conflicted, and that overconfidence is measurably costly for matching human moral preferences. It also shows that a simple, widely available mechanism (attention dropout at inference) shifts both the information-theoretic profile of a model's decisions and its agreement with human judgments, offering a diagnostic tool rather than just a scoreboard.

Real-world implications:

• Autonomous vehicle ethics: The scenarios are self-driving-car dilemmas, directly relevant to how such systems should resolve unavoidable harm.
• AI systems giving moral or ethical advice: Prior work cited in the paper warns of reliance on LLMs for moral guidance; this paper's uncertainty measures give a way to flag where a model is overconfident.
• Safety and evaluation of deployed LLMs: Uncertainty decomposition (total entropy, conditional entropy, mutual information) can be used as an evaluation signal beyond aggregate accuracy or agreement.
• Risk mitigation in decision systems: The authors note that stochastic sampling can raise variance in decisions, which matters in high-stakes contexts where consistency is valued.

Industry relevance: the paper provides a lightweight inference-time intervention and a quantitative metric that model developers could adopt when auditing models for ethical decision-making, and it warns that alignment gains from stochasticity come with a variance trade-off that must be managed.

Future Directions

• Learn uncertainty rather than inject it: The authors argue that uncertainty in moral scenarios should ideally be learned from training data and reflect cultural norms, rather than being added ad hoc at inference time.
• Develop more diverse alignment metrics: Dropout improved alignment overall but hurt specific dimensions such as Age and Species, suggesting aggregate L2 distance is insufficient for evaluating moral dilemmas.
• Resolve the alignment-versus-variance trade-off: Higher total entropy and mutual information improved scores but increased choice variance; whether that is acceptable in high-stakes dilemmas remains open.
• Confront the normative question: Whether designers should build systems that faithfully mirror human judgments or systems that "transcend human limitations" with more principled decisions is left explicitly unresolved.

Target Audience

Researchers and practitioners in AI ethics, alignment, and safety; machine learning engineers interested in uncertainty quantification and inference-time interventions; cognitive and moral psychologists studying how LLM judgments compare with human moral preferences; and policy or governance stakeholders who need to understand the reliability limits of LLMs in ethically charged decision-making.

Authors’ abstract

Humans display significant uncertainty when confronted with moral dilemmas, yet the extent of such uncertainty in machines and AI agents remains underexplored. Recent studies have confirmed the overly confident tendencies of machine-generated responses, particularly in large language models (LLMs). As these systems are increasingly embedded in ethical decision-making scenarios, it is important to understand their moral reasoning and the inherent uncertainties in building reliable AI systems. This work examines how uncertainty influences moral decisions in the classical trolley problem, analyzing responses from 32 open-source models and 9 distinct moral dimensions. We first find that variance in model confidence is greater across models than within moral dimensions, suggesting that moral uncertainty is predominantly shaped by model architecture and training method. To quantify uncertainty, we measure binary entropy as a linear combination of total entropy, conditional entropy, and mutual information. To examine its effects, we introduce stochasticity into models via "dropout" at inference time. Our findings show that our mechanism increases total entropy, mainly through a rise in mutual information, while conditional entropy remains largely unchanged. Moreover, this mechanism significantly improves human-LLM moral alignment, with correlations in mutual information and alignment score shifts. Our results highlight the potential to better align model-generated decisions and human preferences by deliberately modulating uncertainty and reducing LLMs' confidence in morally complex scenarios.