Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment

AI summary

Overview

• Research area: Natural Language Processing and computational ethics, specifically the aggregation of human annotator judgments about moral content (Moral Foundations Theory).
• Technical level: Intermediate. The paper assumes familiarity with Bayesian modeling, entropy, and calibration, but the core ideas are explained through worked examples.
• Scope: The paper introduces "Moral Entropy," a Bayesian consensus model whose posterior entropy serves both as a calibrated ground truth for auditing standard annotator-voting rules and as a soft training target for moral-foundation classifiers.

What This Paper Is About

Almost all moral-annotation corpora turn many annotator votes into one label using a cheap heuristic — majority vote, or the permissive "any-annotator" rule that marks an item positive if even one annotator flags it. These rules throw away exactly the information moral psychology cares about: how much annotators disagreed, and why. The paper argues that disagreement should be modeled as uncertainty rather than voted away, and it builds a Bayesian consensus posterior that converts annotator disagreement into a measurable, auditable quantity.

Key Contributions

1. Moral Entropy framework. The paper defines Moral Entropy as the entropy of a Bayesian posterior over a moral-foundation label, derived from a Dawid–Skene-style model of annotator confusion matrices. Cross-entropy/KL, Brier score, and expected calibration error are described as members of one family of diagnostics (Bregman divergences of an entropy function); the main text reports the 0/1-loss case and the rest are in the supplement.
2. Auditing of consensus rules. Existing heuristic rules (any-annotator, majority, two-vote) are graded against the Bayesian posterior and against a Bayes-optimal gold label, across three corpora and fifteen discourse domains, exposing large, direction-dependent, corpus-dependent bias.
3. Entropy-based uncertainty analysis. Total Moral Entropy is decomposed into aleatoric uncertainty (irreducible disagreement about moral content) and epistemic uncertainty (arising from insufficient or noisy annotation), using the mutual-information decomposition standard in Bayesian deep learning. The paper states this reveals three qualitatively distinct regimes across corpora.
4. Soft-label fine-tuning. Classifiers are trained directly on the calibrated posterior instead of collapsed hard labels, producing consistent 2–3% accuracy gains attributed to the greater consistency of soft targets.

Main Findings

• The any-annotator rule has a large, structurally asymmetric bias. Pooled across corpora, it disagrees with the Bayesian consensus at FPR = 30.6% and FNR = 0.0%. The near-zero false-negative rate is close to structurally guaranteed: because the rule is a logical OR over annotator votes and the confusion-matrix prior is diagonal-dominant, an all-negative vote pattern almost never yields a posterior above 0.5.
• The false-positive error is large and real. Roughly a third of items flagged positive by the naive rule are not supported by the calibrated Bayesian consensus.
• Errors vary sharply by discourse domain. Per-foundation FPR ranges from 12.8% (Election) to 30.4% (MeToo), and FNR from 20.2% (Sandy) to 83.8% (Davidson). MeToo and Sandy are the only two domains where FPR still exceeds FNR; the other five (ALM, BLM, Baltimore, Davidson, Election) invert the pattern.
• The pooled asymmetry does not survive at the foundation level. Mean FPR/FNR is 19.9%/38.9% on MFTC and 10.2%/54.8% on the three MFRC domains, i.e. the errors invert.
• Strict rules fail in the opposite direction. Majority and two-vote nearly eliminate false positives (3.5% and 4.2% on MFTC, under 1% on MFRC) but miss most true positives (mean FNR 63–65% on MFTC, approximately 83% on MFRC).
• Strict rules converge where annotation is thin. On MFRC, 13 of 15 FNR cells coincide between majority and two-vote, because at 3.00 annotations per item a majority is two votes; they separate where items carry more annotators (BLM, 4.92 per item: FNR 0.64 versus 0.55).
• The two-vote rule collapses on eMFD. Checked against eMFD's independent expert gold labels rather than the authors' own posterior, it becomes nearly degenerate: mean FPR = 1.5%, FNR = 96.6%, since most eMFD items never reach two votes. No fixed vote-count rule generalizes across annotation designs.
• Soft labels beat hard labels. Fine-tuning on the calibrated posterior improves accuracy by 2–3% over identical architectures trained on aggregated voting labels, consistently across foundations, with training plateauing within the first epoch. Final ROC-AUC against the hard label was 0.955 (Care), 0.950 (Sanctity), 0.949 (Fairness), 0.944 (Authority), and 0.934 (Loyalty); Care converges highest.
• Posterior entropy is almost entirely aleatoric. Epistemic uncertainty contributes under 1% of total entropy in MFTC (mean 0.3%) and MFRC (mean 0.3%), and never exceeds 6% even in eMFD, the corpus with the fewest annotators per item. The authors read this as evidence the confusion-matrix parameters are tightly identified by the data, so the reported entropy is real moral disagreement rather than model underfitting.
• High-entropy items cluster around a known psychological boundary. Among the 100 highest-entropy Care items in BLM, disagreement concentrates on hostile political rhetoric ("terrorists," "TERRORISM," "shoot tyrants") rather than compassion-laden content. This matches Kennedy et al. (2023), who find hateful language mostly invokes Purity and Loyalty/Authority, implicating Care only when an explicit harmful act is described.
• No annotator is a fixed outlier. Annotator disagreement rates against the BLM majority range from 7.6% (A04) to 31.3% (A02), but in the five representative high-entropy items examined, annotators switch sides depending on item wording; A02, the most discordant annotator corpus-wide, agrees with the majority on four of five and dissents alone only on the least ambiguous item.

Methodology in Plain English

The authors adapt the classic Dawid–Skene model of noisy annotators. Each annotator gets a "confusion matrix" describing how likely they are to say each label when the true label is something else, with weak Dirichlet priors encoding a mild belief that annotators are more often right than wrong. Each item's true label has a prevalence drawn from a Dirichlet distribution. Given all the annotations on an item, Bayes' rule gives a posterior probability that the item genuinely carries a given foundation label — the "Bayesian consensus" p_i. The model is fitted by MAP estimation via gradient ascent on the log-posterior (Adam optimizer, log-domain computation for numerical stability), and a Laplace approximation — a Gaussian centered at the MAP estimate with covariance from the inverse negative Hessian — converts the point estimate into a full posterior. The authors note they took this route because exact sampling via NUTS proved fairly slow.

The entropy of that per-item posterior is the item's Moral Entropy. Sampling from the Laplace approximation lets this total entropy be split into an aleatoric part (the average entropy across posterior draws — uncertainty a fully informed model would still report) and an epistemic part (the residual, non-negative by Jensen's inequality — disagreement among plausible models that more annotation could reduce).

Because Moral Entropy is maximized exactly at p_i = 1/2, and thresholding there is the Bayes-optimal decision under 0/1 loss, the Bayesian consensus doubles as a calibrated ground truth against which discrete rules can be graded. The paper's headline diagnostic is hard disagreement against this gold label, expressed as false-positive and false-negative rates. Continuous diagnostics (cross-entropy/KL, Brier score decomposition, expected calibration error) are reported in the supplementary material.

Evaluation uses three MFT-annotated corpora totalling 106,627 texts carrying over 250,000 annotations: MFTC (33,858 tweets, roughly 125,000 annotations from 23 trained annotators, seven discourse domains), MFRC (17,886 Reddit posts across Everyday Morality, US politics, and French politics), and eMFD (54,883 news-article paragraphs, which the authors group by GDELT theme tags into Physical Violence, Asymmetric Threats, Internal Unrest, Civil Liberties, and an "Untagged" residual category).

Why This Matters

Impact on research. The paper reframes annotator disagreement as a resource rather than noise. It offers a way to distinguish a genuinely contested corpus from an under-annotated one, and to audit any heuristic consensus rule already in use — since a rule can look reasonable in isolation while hiding a large, systematic, corpus-dependent bias. Because any-annotator's false-positive rate grows mechanically with the number of annotators J even when every individual annotator is well calibrated, downstream classifiers inherit a bias with no way to detect it. The authors note none of the machinery is specific to Moral Foundations Theory: entropy audit can be applied to any values- or judgment-annotation pipeline before a hard-label dataset ships.

Real-world applications:

• Content moderation and hate-speech datasets. Moral-judgment labels underpin moderation classifiers; a rule that flips roughly a third of flagged items would systematically over- or under-moderate depending on the platform's annotation design.
• Dataset documentation and release. Reported entropy (split into aleatoric and epistemic) could accompany published labels so downstream users know which items are contested versus merely under-annotated.
• Training value-alignment and safety classifiers. Soft-label training gave 2–3% accuracy gains over hard labels on MFTC, suggesting a low-cost improvement for teams already training on vote-aggregated data.
• Cross-cultural and cross-lingual value measurement. The paper points to translating an existing English corpus and re-running the entropy audit as a practical route to testing whether a consensus encodes a narrow cultural view.

Industry relevance. The practical claims are that posterior entropy is cheap to approximate (a Laplace approximation suffices), that soft labels cost nothing at training time and beat the hard labels they were built from, and that no fixed vote-count rule generalizes across annotation designs — all directly relevant to teams building labeled data pipelines rather than research prototypes.

Future Directions

• Systematic validation of the uncertainty decomposition. The reported aleatoric/epistemic split rests on a Laplace approximation around the MAP fit. The authors attempted direct validation via NUTS but encountered slow mixing (effective sample size as low as 5 out of 300 draws even after tuning burn-in and trajectory depth, likely due to strong correlations in the high-dimensional confusion-matrix parameter space). On the small number of items where MCMC yielded usable draws the decomposition matched the Laplace approximation, but a systematic validation via a reparameterized sampler or an ensemble of independently seeded MAP fits is left to future work.
• Correcting annotator-pool and cultural bias. Annotators for MFTC, MFRC, and eMFD were not sampled to represent diverse cultural or ideological perspectives, so even a well-calibrated consensus over this pool could encode a narrow, plausibly WEIRD (Western, Educated, Industrialized, Rich, Democratic) view. The authors note trained annotation against a shared codebook partly mitigates this, and Skorski (2026) shows moral semantics are largely preserved under careful LLM-based translation (EN to PL, AUROC gaps of 0.01–0.02 across foundations) — pointing to translating a corpus and re-running the audit as a practical corrective, which they leave to future work.
• Generalizing the fine-tuning result. The 2–3% soft-label gains are demonstrated on a subset of corpora and foundations and a single classifier architecture; generalization across architectures and the full label set is untested.
• Extending the audit beyond Moral Foundations Theory. The authors state the entropy audit applies to any values- or judgment-annotation pipeline, raising the open question of how it performs on other label ontologies.

Target Audience

Researchers and practitioners who build or consume datasets of subjective human judgments — computational social scientists, NLP researchers working on moral or values annotation, and machine learning engineers training classifiers on vote-aggregated labels. It is most useful to readers comfortable with Bayesian modeling and calibration metrics, though the central argument (that OR-rules over noisy votes are biased and that entropy reveals it) is accessible without that background. Practitioners focused on content moderation, dataset curation, and annotation quality assurance will find the audit results directly actionable.

Authors’ abstract

Most work in computational ethics treats annotator disagreement on moral content as noise to be voted away, collapsed into majority vote or the more permissive any-annotator rule the moment a single annotator flags an item. We argue this uncertainty should instead be modeled and learned from. We introduce Moral Entropy, a Bayesian framework that keeps a full posterior over the true label and decomposes its entropy into aleatoric uncertainty (irreducible disagreement about the moral content) and epistemic uncertainty (from insufficient or noisy annotation) -- and lets any heuristic consensus rule be audited against a calibrated ground truth via entropy methods such as cross-entropy/KL, Brier score, and expected calibration error. Across three corpora and fifteen discourse domains, auditing the standard aggregation rules against this posterior reveals bias that no current pipeline reports: the any-annotator rule disagrees with the calibrated posterior on roughly 30% of items -- pooled, almost entirely false positives, though the errors invert at the foundation level (19.9%/38.9% mean FPR/FNR on MFTC) -- while the stricter majority and two-vote rules miss 63-83% of true positives.