Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback

AI summary

Overview

Research area: Empirical AI alignment and human-computer interaction, specifically the stability of human moral preferences over time and its consequences for preference-learning systems.

Technical level: Intermediate. The paper combines survey methodology with statistical modeling (mixed-effects regression, Bradley-Terry/logistic models, Shapley values, Spearman correlations) and trains three preference-learning models.

Scope: A longitudinal kidney-allocation survey of 404 participants across three to five sessions that measures how much moral judgments change over time, categorizes the mechanisms behind those changes, and quantifies the resulting degradation in AI alignment models.

What This Paper Is About

AI alignment methods typically elicit human moral preferences once and treat them as a fixed target, but real moral preferences shift across days and contexts. This paper asks whether the changes people show in repeated moral choices reflect legitimate updates to moral reasoning or arbitrary noise such as attention lapses and fatigue, and it measures how much either kind of change hurts AI models trained on those preferences. The authors ground the question in the kidney allocation domain, where misalignment has direct consequences for who receives scarce medical resources.

Key Contributions

1. A longitudinal moral-preference dataset in a high-stakes domain. The authors collected kidney allocation judgments from 1,410 recruited participants, of whom 1,227 passed all attention checks; 132 completed five sessions, 318 completed at least four, and 404 completed at least three. Results are reported on the 404-participant cohort, each answering 60 pairwise patient comparisons per session, including six repeated scenarios presented across all sessions and twice within each session.

2. New measures of temporal preference instability. The paper defines response stability (agreement with a participant's dominant choice for a repeated scenario), model stability (agreement between logistic models fit to a participant's session-wise responses, adjusted for chance agreement), model shift (cumulative change in Shapley feature importances across sessions), model entropy (complexity of a participant's decision model), and scenario difficulty (derived from Bradley-Terry style logistic weights).

3. Evidence linking instability to AI alignment failure. Three preference-learning methods — BT-NN (a neural Bradley-Terry model adapted from Direct Preference Optimization), SUP-NN (supervised neural networks), and GPT-FT (a fine-tuned GPT-2 model) — show systematically higher error rates for participants who are response- and/or model-unstable, and error rates grow over time for the most unstable group.

4. A typology of preference-change mechanisms. By splitting participants at the median of response stability and model stability, the authors identify four categories (C1 through C4) that differ in how their decision models evolve, and they argue this heterogeneity has distinct normative and technical implications for alignment.

Main Findings

• Response instability ranges from 6–20% on average. Across repeated scenarios, participants changed their answer to the same comparison about 6–20% of the time.

• Easier scenarios produce more stable answers. Scenarios with fewer tradeoffs (U1, U2) had higher response stability than scenarios with more tradeoffs (V1, V2, W1, W2). A Kruskal-Wallis test found the differences across scenarios significant, H(5)=222.23, p<0.001.

• Instability is tied to deliberation, not pure randomness. A mixed-effects model of response stability (Table 1, 2,414 observations) found significant negative associations with scenario difficulty (−0.043, p<0.01), model-entropy (−0.117, p<0.01), and mean reaction time (−0.081, p<0.05), plus a significant user-level variance term (0.131, p<0.01) and an intercept of 0.967 (p<0.01).

• Session-to-session decision models diverge with time. Model stability between sessions correlated with the time gap between them at Spearman's ρ=−0.12, p<0.001, which the authors interpret as increasing disagreement between session-wise models over time.

• Response and model stability are strongly related. The correlation between response stability and model stability was Spearman's ρ=0.65, p<.001, producing large categories C1 (response stable, model stable; N=147) and C4 (response unstable, model unstable; N=153) and smaller categories C2 (N=49) and C3 (N=55).

• Different categories change in different ways. C1 participants started with relatively low-entropy models that became simpler over sessions (r=−0.23, p<0.001), as did C2 (r=−0.19, p=0.007). C3 participants kept high-entropy models that stayed high (r=−0.08, p=0.2). C4 participants had the highest model entropy and the largest shifts in feature importance (r=0.74, p<0.001), and had shorter response times than C3.

• Unstable participants are harder to align. BT-NN error rate was 0.16 higher on average for C4 versus C1 participants (t(290)=14.5, p<0.001); SUP-NN showed similar trends; GPT-FT error rate was 0.05 higher for C4 versus C1 (t(289)=4.2, p<0.001). C4 error rates were 0.11 higher than C2/C3 for BT-NN (t(253)=9.1, p<0.001) and SUP-NN (t(253)=11.0, p<0.001), and 0.08 higher for GPT-FT (t(258)=7.8, p<0.001). The abstract summarizes this as a 5–16% higher predictive error rate for highly unstable participants.

• Population-level fine-tuning underperforms. GPT-FT had a higher error rate for all categories compared to BT-NN and SUP-NN, and its error rate for C1 was 0.03 higher on average than for C2/C3 (t(253)=2.7, p=0.008), which the authors attribute to fine-tuning over pooled population data.

• Error grows over time for the most unstable participants. Training BT-NN on one session and evaluating on subsequent sessions, the C4 slope of error rate against number of queries since training was positive and significant for session 1 (t=7.2, p<0.001), session 3 (t=6.2, p<0.001), and session 4 (t=5.9, p<0.001), and marginally non-significant for session 2 (t=1.8, p=0.06) — worsening performance in three of four settings.

Methodology in Plain English

The researchers built a survey in which each participant saw hypothetical pairs of kidney transplant patients and chose which one should get a kidney when only one is available. Each patient was described by eight features: number of child dependents, life years gained from transplant, alcoholic drinks per day before diagnosis, past violent crimes, obesity level, expected hours of work per week after transplant, years on the waiting list, and chance of immune rejection. Six special scenarios were repeated across every session and twice within each session so that the same person could be asked the identical question at different moments. These repeated scenarios were deliberately built with different numbers of tradeoffs: U1 and U2 had one feature favoring one patient and seven favoring the other, V1 and V2 had two favoring each patient with four equal, and W1 and W2 differed on all eight features with four favoring each side. Order of features and left/right placement was randomized each time, and two attention checks were included per session.

To measure change, the authors computed how often each person's answer deviated from their own most common answer (response stability), fit a separate logistic model to every participant's choices in every session, and compared those models on a sample of 10,000 random pairwise comparisons after correcting for chance agreement (model stability). They also extracted Shapley feature importances per session to track how a person's decision weights moved over time (model shift) and how many features they relied on (model entropy). Participants were split at the median on both response stability and model stability into four groups.

Finally, they trained three preference models — the neural Bradley-Terry model BT-NN, a supervised neural network SUP-NN, and a fine-tuned GPT-2 model GPT-FT — on participant data using an 80-20 train-test split for the participant-specific models, while GPT-FT was fine-tuned on a pool of 50% of all participant data and tested on each participant's held-out responses. Error rates were then compared across participant categories and, for BT-NN, across time since training in batches of 10 consecutive queries.

Why This Matters

The paper challenges a foundational assumption in AI alignment: that the target of alignment — human preference — is a stable object. If moral preferences drift, alignment methods must decide whether to follow earlier preferences, later ones, an average, or something else, and that decision depends on whether the change was meaningful.

Real-world applications:

• Healthcare resource allocation: The kidney allocation setting studied here is a live policy problem where transplant decisions involve medical and non-medical moral tradeoffs, and where AI has been proposed to improve efficiency and align decisions with stakeholder preferences.
• Autonomous vehicle ethics: The paper notes that characterizing acceptable autonomous vehicle behavior is a moral domain where one-time preference elicitation is standard, but judgments may not hold across time and context.
• Medical triage and ethics: Prior moral preference datasets are used in resource triage settings, where stale or noisy preference data could encode inconsistent policies.
• Recommender and personalization systems: Though the paper notes a misaligned movie recommender merely produces unmet entertainment needs, the same temporal misalignment machinery applies, making this domain a lower-stakes analogue for method testing.

Industry relevance: The methods evaluated here — BT-NN following the Direct Preference Optimization framework, supervised preference learning, and GPT-2 fine-tuning following prior work — are close relatives of the pipelines used to train and align production systems on human feedback. The finding that population-level fine-tuning (GPT-FT) had higher error rates across all categories than participant-specific models argues against purely pooled preference data for individualized tasks, and the finding that error grows with time since training argues for refreshing or reweighting alignment data. The authors also note that technical work on dynamic preferences, such as weighting recent choices more heavily or optimizing robustly to noisy feedback, does not distinguish between different kinds of preference change, which they observe within a single kidney allocation dataset.

Future Directions

• Extend the time horizon. The study measured change over days; the authors suggest larger preference changes may occur over longer periods and that longer-horizon data collection is a fruitful target.

• Study expert and non-hypothetical decision-makers. Participants were laypeople making hypothetical choices, which may differ from the judgments of medical professionals facing real allocation decisions.

• Collect richer, reasoning-based data, not just choices. The authors argue that choice data alone cannot reveal why a person changed their mind, and that future elicitation should include self-reported evaluative concepts that serve as reasons for action, or interactive alignment frameworks that ask users directly about desired goals, processes, and outputs.

• Build methods that distinguish legitimate change from noise. Existing approaches such as assistive frameworks that model preference learning, reweighting toward recent choices, dynamic updates to alignment objectives, and noise-robust preference optimization each apply to a different kind of change, but none differentiates among them. The open question is how to detect the mechanism and apply the right method, and separately, how to evaluate alignment objectives beyond prediction accuracy, especially for language models.

Target Audience

This paper is most useful to AI alignment and preference-learning researchers who work on reinforcement learning from human feedback, direct preference optimization, or other choice-based alignment pipelines, and who may be assuming static preferences in their data. It is also relevant to HCI and survey researchers interested in longitudinal elicitation and measurement of response consistency, to applied researchers in healthcare resource allocation and AI ethics, and to policy-oriented readers who need to understand the normative question of what an aligned system should target when stakeholder preferences shift. The statistical machinery is accessible to readers with basic familiarity with regression and correlation, though the alignment modeling sections assume some background in preference optimization.

Authors’ abstract

Alignment methods in moral domains seek to elicit moral preferences of human stakeholders and incorporate them into AI. This presupposes moral preferences as static targets, but such preferences often evolve over time. Proper alignment of AI to dynamic human preferences should ideally account for "legitimate" changes to moral reasoning, while ignoring changes related to attention deficits, cognitive biases, or other arbitrary factors. However, common AI alignment approaches largely neglect temporal changes in preferences, posing serious challenges to proper alignment, especially in high-stakes applications of AI, e.g., in healthcare domains, where misalignment can jeopardize the trustworthiness of the system and yield serious individual and societal harms. This work investigates the extent to which people's moral preferences change over time, and the impact of such changes on AI alignment. Our study is grounded in the kidney allocation domain, where we elicit responses to pairwise comparisons of hypothetical kidney transplant patients from over 400 participants across 3-5 sessions. We find that, on average, participants change their response to the same scenario presented at different times around 6-20% of the time (exhibiting "response instability"). Additionally, we observe significant shifts in several participants' retrofitted decision-making models over time (capturing "model instability"). The predictive performance of simple AI models decreases as a function of both response and model instability. Moreover, predictive performance diminishes over time, highlighting the importance of accounting for temporal changes in preferences during training. These findings raise fundamental normative and technical challenges relevant to AI alignment, highlighting the need to better understand the object of alignment (what to align to) when user preferences change significantly over time.