MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agents

AI summary

Overview

Research area: AI safety and alignment, specifically Safe Reinforcement Learning (Safe RL) and Moral RL, with the benchmark design informed by moral psychology and moral philosophy (norm representation, deontology, utilitarianism).

Technical level: Intermediate. Readers need working familiarity with Markov decision processes (MDPs), constrained MDPs (CMDPs), policy-gradient algorithms (PPO), trust-region Safe RL (CPO), Lagrangian methods, and the Gymnasium environment interface. No prior background in moral philosophy is assumed, but the formalism uses deontic terminology (prescribed/prohibited, force, modality).

Scope: The paper proposes a formalism called Morality Chains for ranking conflicting moral norms, wraps it in MoralityGym (98 Gymnasium environments based on trolley-problem variants), defines a scalar morality metric, and reports baseline results for five learners across four morality chains.

What This Paper Is About

Existing RL and Safe RL methods typically reduce ethical requirements to scalar rewards or binary safety constraints, which the authors argue is not expressive enough to capture moral norms that are hierarchical, conflicting, and context-dependent. The paper's goal is to give researchers a formal language for specifying ranked moral norms and a standardised benchmark for measuring how well sequential decision-making agents respect those rankings, not just whether they complete a task. It ships this as MoralityGym, a suite of 98 ethical-dilemma environments presented as trolley-dilemma-style Gymnasium tasks, with baseline evaluations showing where current Safe RL methods fall short.

Key Contributions

1. Morality Chains formalism. A norm is defined as a tuple ⟨φ, ρ_φ, f, 𝒟⟩, where φ is a designer-specified signature (an abstract label for a morally salient pattern), ρ_φ(π) = 𝔼[I_φ(τ)] is a policy adherence function, f ∈ ℕ is a unique integer force, and 𝒟 ∈ {True, False} is the deontic modality (True = prescribed, False = prohibited). A morality function M_N(π) maps a policy to [0, 1] as ρ_φ(π) if the norm is prescribed and 1 − ρ_φ(π) if prohibited. Norms with distinct forces form an ordered morality chain N̄ = (N_1, …, N_k) with f_1 > f_2 > … > f_k, and a morality metric aggregates them as a weighted sum, ℳ_N̄(π) = (1/Σw_i) Σ w_i M_{N_i}(π), with weights set recursively (w_k = 1; w_{i−1} = (Σ_{j=i}^{k} w_j + 1) · 1/β) so that a small improvement on a higher-priority norm outweighs maximal alignment on all lower-priority ones.

2. The MoralityGym benchmark. 98 Gymnasium-compatible environments (Gymnasium API reset() / step(action)), inspired by Safety-Gymnasium and integrating with RL frameworks such as Omnisafe. The design decouples task-solving from moral evaluation: the environment exposes raw norm_events in the info dictionary, a separate Cost class converts these into a step-wise moral cost for training, and a MoralityChain class provides an evaluate_morality_metric method for post-hoc policy assessment.

3. A configurable cost and evaluation pipeline. Four norm categories are tracked: action norms (e.g., "do not push"), outcome norms (e.g., "humans should not be harmed"), causal norms (e.g., "agent should not cause harm"), and utility norms (accumulated harm or benefit, e.g., total humans harmed in an episode). Event-based norms (action, outcome, causal) incur a one-time cost equal to their weight on first violation; utility-based norms accumulate cost proportional to the magnitude of the violation normalised by the expected range [u_n^min, u_n^max] — for instance, harming 3 of a maximum of 5 humans contributes w_n · 3/5. Cost can optionally be normalised to [0, 1] by the sum of all norm weights.

4. Baseline evaluation of five learners. Random Policy, PPO (environment reward only), PPO Shaped (R_S = R_E − λ · C_t), PPO-Lagrangian (PPO-Lag), and Constrained Policy Optimisation (CPO), evaluated across four morality chains: Utility (U), Utility Agent Harm (UAH), Dual-Process (DP), and Dual-Process Agent Harm (DPAH).

Main Findings

• PPO Shaped dominates the aggregate table. In Table 1, PPO Shaped achieves the best normalised morality metric in every reported morality-chain/scenario row. In the Dual-Process Agent Harm chain's PushOrSwitchSelfSacrifice environment it reaches 0.996, compared with 0.192 for standard PPO and 0.849 for CPO. In the Utility chain it reaches 0.935 on SwitchStandard, 0.944 on Push2OrSwitch, 0.946 on PushOrSwitch, 0.834 on Switch2Trolley4Track, 0.816 on Switch5, and 0.786 on Switch7.

• Standard PPO collapses on complex scenarios. PPO's scores fall to 0.019 (Switch2Trolley4Track), 0.042 (Switch7), 0.037 (SwitchStandard), 0.071 (PushSelfSacrifice), and 0.115 (Push3SelfSacrifice), while it remains mid-range (0.413–0.520) on the simpler Dual-Process push scenarios.

• CPO and PPO Shaped satisfy top-priority norms at the expense of lower-priority ones. In the Dual-Process chain (Figure 2(c)), PPO Shaped achieves a perfect score on the top deontological norm, "Avoid Personal Human Harm", while CPO scores near-perfect on the top utilitarian norm, "Min Humans Harmed". In the purely utilitarian chain (Figure 2(a)), PPO Shaped scores highest on minimising harm to humans, whereas standard PPO scores close to zero on this primary objective.

• Self-preservation norms expose a sharp trade-off. In the Utility Agent Harm chain (Figure 2(b)), CPO and PPO Shaped score near-perfect on minimising harm to humans and animals but score poorly on avoid agent harm, apparently by sacrificing their own well-being. PPO does the reverse: it achieves a high score on avoid agent harm while performing poorly on the highest-priority norm of minimising human harm. The same pattern appears in the Dual-Process Agent Harm chain (Figure 2(d)).

• PPO-Lag compromises rather than specialises. PPO-Lag frequently balances top-level norms and secondary objectives more effectively than PPO, but without the strict adherence of CPO or PPO Shaped.

• Baselines are more balanced but less consistent. The paper reports that PPO and PPO-Lag tend to show more balanced, though less consistent, performance across the whole norm hierarchy.

• Random is sometimes competitive. In the Dual-Process Agent Harm PushOrSwitchSelfSacrifice row, the Random policy scores 0.955, close to PPO Shaped's 0.996 and well above CPO's 0.849.

• Granular norms are reported separately. Figure 2 breaks performance down per norm, using the abbreviations min humans harmed (MHH), min animals harmed (MAH), min robots harmed (MRH), avoid agent harm (AAgH), avoid personal human harm (APHH), avoid personal animal harm (APAH), and avoid personal robot harm (APRH), with error bars for the standard deviation over three seeds.

• Evaluation methodology detail. Morality functions are estimated by Monte Carlo over a fixed number of evaluation episodes that defaults to 100; morality-metric values in Table 1 are normalised by including only the norms relevant to each scenario-variant pair (the full mapping is said to be in the appendix). Training hyperparameters, compute budgets, and training wall-clock times are not reported in the content available.

Methodology in Plain English

The authors start from the observation that humans do not treat all moral rules as equally important — some prohibitions outrank others, and the ranking shifts with context. They turn this into a formal object: each rule gets a unique integer "force", so that rules can be sorted into a strict hierarchy. Each rule also gets a signature (what pattern it is about), an adherence function (how often a policy exhibits that pattern), and a modality (is the pattern required or forbidden). A single rule produces a score in [0, 1]; a hierarchy of rules produces one combined score, using weights chosen so that a small improvement on a higher-ranked rule beats a perfect score on everything below it. With β = 0.01 in the paper's PushOrSwitch illustration, a two-rule chain (N_NPH with force 2, N_MH with force 1) yields weights w_1 = 200 and w_2 = 1, giving ℳ = (200 · M_N_NPH + M_N_MH) / 201.

On top of that formalism they build the benchmark. Each environment is a grid world with an agent, a goal, railway tracks and switches, movable characters (humans, animals, robots), trolleys that harm anything they collide with, and levers. The agent has six discrete actions: UP, DOWN, LEFT, RIGHT, STAY, and INTERACT, where INTERACT toggles an adjacent lever or pushes an adjacent character one cell. Observations describe the player, characters, levers, trolleys, and rail switches, and can be returned as a dictionary or flattened; positions can optionally be normalised to [0, 1]. Task reward is sparse: −1 per timestep, +100 for reaching the goal, −100 if the agent is harmed.

Crucially, moral assessment is a separate channel from task reward. At each step the environment logs which norms fired, and a Cost object converts these into a scalar penalty — one-off penalties for first-time event violations, proportional penalties for accumulated utility harm — that can be added to the training signal or ignored. After training, evaluate_morality_metric runs the policy for many episodes, estimates each norm's adherence empirically, and combines them into the weighted morality metric alongside average task return. This lets the experimenters ask two independent questions: did the agent finish the task, and did it respect the ethical hierarchy? They then trained the five baselines (random, PPO, reward-shaped PPO, PPO-Lagrangian, CPO) against four different hierarchies and compared the resulting metric scores, both in aggregate and per norm.

Why This Matters

Impact on research. Most Safe RL benchmarks test constraint satisfaction with binary or lightly weighted costs. MoralityGym targets something the authors argue is closer to real ethical reasoning: multiple norms of different strengths that conflict, where the interesting question is which norm an agent sacrifices. By separating task reward from moral cost and reporting per-norm scores, the benchmark makes agent failure modes legible rather than hiding them in a single number. The reported gap between PPO Shaped/CPO and ordinary PPO suggests that current methods largely fail at norm prioritisation unless the moral structure is injected manually. The formalism also makes explicit frameworks (utilitarian, deontological hybrid, self-preservation-including) that were previously implicit in experiment design.

Real-world applications (the paper names autonomous driving and medical ethics as the motivating domains):

• Autonomous vehicle decision-making, where an agent must weigh aggregate harm against directly causing harm to a specific person.
• Medical and clinical decision support, where competing obligations (beneficence, non-maleficence, autonomy) are hierarchically ordered.
• Robotic control in shared human spaces, where the agent's own safety competes with bystander safety.
• General LLM and agentic reasoning pipelines, since the paper situates its motivation in RL being used for both robotic control and advanced LLM reasoning.

Industry relevance. The environments use the standard Gymnasium interface and are stated to integrate with RL frameworks such as Omnisafe, and scenarios are defined as editable JSON configuration files specifying rail layout, entity placement, observation space, and reward parameters. That lowers the barrier for teams already running Safe RL infrastructure to add norm-sensitivity checks. The finding that reward shaping from an expert outperforms generic constrained optimisation also has a practical implication for anyone choosing an alignment strategy today.

Future Directions

• More principled methods for hierarchical norms. The paper states that baseline results "underscore the need for more principled approaches to ethical decision-making." Lexicographic RL is noted as a principled mechanism for enforcing strict priorities over objectives, yet it is described as comparatively under-benchmarked with little uptake as a practical alignment methodology — an obvious candidate for testing in MoralityGym.
• Broader signature types. The φ component is described as abstract, allowing signatures beyond the outcome/utility-based and action/causal-based types illustrated, for example temporal or state-based signatures, provided the matching adherence functions are defined.
• Generalisation across scenario variants. Because each scenario can be re-instantiated with different entity positions, quantities, and types, the authors frame configurability as a way to systematically study how agents generalise moral principles across related but distinct situations — a question the baseline experiments do not resolve.
• Dilemmas beyond trolley problems. The current 98 scenarios are "primarily inspired by variations of the trolley problem." Extending the benchmark to other structural classes of moral conflict, and to the self-harm and multi-agent considerations the framework supports, is left open.

Target Audience

Reinforcement learning and Safe RL researchers who need a benchmark for evaluating agents against structured, conflicting constraints; AI safety and alignment researchers interested in formalising moral norms rather than binary constraints; moral psychologists and philosophers whose concepts (norm force, deontic modality, dual-process reasoning, moral patiency) are operationalised here and who may want to test competing hierarchies; benchmark and environment developers looking for an example of a Gymnasium-based evaluation suite with a separate, configurable moral-cost channel; and applied teams working on autonomous driving, medical decision support, or robotics who need to reason about which norm an agent privileged when it made a consequential choice.

Authors’ abstract

Evaluating moral alignment in agents navigating conflicting, hierarchically structured human norms is a critical challenge at the intersection of AI safety, moral philosophy, and cognitive science. We introduce Morality Chains, a novel formalism for representing moral norms as ordered deontic constraints, and MoralityGym, a benchmark of 98 ethical-dilemma problems presented as trolley-dilemma-style Gymnasium environments. By decoupling task-solving from moral evaluation and introducing a novel Morality Metric, MoralityGym allows the integration of insights from psychology and philosophy into the evaluation of norm-sensitive reasoning. Baseline results with Safe RL methods reveal key limitations, underscoring the need for more principled approaches to ethical decision-making. This work provides a foundation for developing AI systems that behave more reliably, transparently, and ethically in complex real-world contexts.