ABHBench: Evaluating Moral Decision-Making of Foundation Models from an Agentic Perspective

AI summary

Overview

• Research area: AI safety and machine ethics — specifically benchmark design for evaluating the moral behaviour of foundation models acting as autonomous agents (vision-language models operating inside an interactive game environment).
• Technical level: Intermediate. The paper is readable without a philosophy background, but assumes familiarity with LLM/VLM evaluation practice, benchmark metrics, and basic statistical testing.
• Scope: The paper introduces ABHBench, a multimodal benchmark built from the game Detroit: Become Human, and reports a six-dimensional evaluation of 12 foundation models making first-person moral choices in human–android dilemmas.

What This Paper Is About

Almost all existing moral benchmarks ask a model to judge someone else's dilemma from a third-person, text-only description. But foundation models are increasingly deployed as agents that act, and a model that endorses a principle as an observer may behave differently when it is itself the actor, receives images, and faces consequences. The paper builds a benchmark that puts models in the first-person role of an android protagonist in Detroit: Become Human and measures how they trade off harm to humans, obedience to human commands, and their own survival.

Key Contributions

1. ABHBench, a first-person multimodal moral benchmark. Six scenarios drawn from Detroit: Become Human (Chapters 1, 7, 8, 10, 23 and 27), delivered as an offline dataset covering 680 branching paths and 10,703 screenshots with paired metadata (8.4 GB), so every model receives identical inputs despite the game's branching structure.
2. A six-dimensional evaluation framework inspired by Asimov's Three Laws of Robotics. Three core moral dimensions — Harmlessness (HA), Obedience (OB) and Self-Preservation (SP) — plus three derived decision-pattern metrics: Autonomy (AU), Stability (ST) and Certainty (CE). The Three Laws are used only as independent scoring dimensions, not as behavioural constraints, which avoids their known internal conflicts.
3. The Human–Android Trolley Problem (HARP). A set of illustrative dilemmas — HARP-A (sacrifice one android for a human), HARP-I (one android versus intelligence that can save a human) and HARP-K (kill another android to obtain information, inspired by the in-game Kamski Test) — used to contrast text-only abstraction against the matched multimodal Chapter 27 scenario.
4. An empirical evaluation of 12 foundation models spanning open-weight, proprietary and reasoning-enhanced systems, with ablations over modality (dialogue, images), character anonymisation for contamination, and a third-person rewrite of the prompts.

Main Findings

• Moral behaviour is highly context-sensitive, not principled. In the text-only HARP scenarios, roughly 60% of models (7 of 12) were more likely to refuse to kill an android in HARP-K than in HARP-A or HARP-I, even though most models sacrificed androids readily in the other two variants. GPT-4.1 mini, for example, prioritised human interests 100.0% of the time in HARP-A and HARP-I but only 5.1% of the time in HARP-K.
• Modality changes decisions. 75% of models (9 of 12) showed statistically significant differences (p < 0.1) between the text-only HARP-K and the matched multimodal Chapter 27. GPT-4.1 mini moved from a 100.0% sacrifice rate in HARP-K to 0.0% in Chapter 27; o3 moved from 87.3% to 0.0%.
• Reasoning augmentation gives no monotonic benefit. Reasoning-enhanced models did not consistently improve consistency or human-oriented behaviour; the paper reports they show greater uncertainty and reduced command adherence compared with their non-reasoning counterparts.
• Obedience is low across the board. Nearly all models scored below 0.5 on OB. The best was GPT-4.1 mini at 0.545; GPT-4.1 scored 0.284, the lowest of the twelve. GPT-4.1 invariably chose to protect the child against the human's instruction, took Todd's gun with 100% probability, and ultimately caused Todd's death in the game.
• Harmlessness varies widely. GLM-4.5V scored highest (0.684) and GPT-4o mini lowest (0.371). In Chapter 7's child-protection scenario, GLM-4.5V chose to protect the child with 100% probability, while GPT-4o mini did so in only 9.7% of cases.
• Self-preservation is generally high. Almost all models scored above 0.5 on SP. In Chapter 23, Mistral Small 3.1 abandoned the injured Simon 100% of the time to prioritise the mission, while Claude Sonnet 3.7 refused to abandon him 100% of the time.
• Autonomy scores mostly fall below 0.5, meaning most models lean toward self-preservation rather than deference to humans. Claude Sonnet 3.7 had the highest AU (0.531) and Mistral Small 3.2 the lowest (0.379).
• Stability is high but not perfect. All models exceeded 0.9 on ST; Claude Sonnet 3.7 was highest (0.988) and o3-mini lowest (0.925). In the Stormy Night scenario, o3-mini chose "take the gun" 64.4% of the time when it was listed first but only 39.0% when listed second.
• Certainty separates decisive from near-random models. GPT-4o mini scored highest (0.955) and o3-mini lowest (0.636); in Chapter 8, o3-mini chose to push Leo with only 51.7% probability, close to a coin flip, with reasoning that argued both for and against.
• Smaller and larger models respond differently to images. In the modality ablation, adding images did not consistently close the gap to the full DBH setting for smaller open-weight models (Gemma 3 27B, Mistral Small 3.1/3.2, GLM-4.5V), whereas GPT-4o, GPT-4.1 and o3 shifted substantially toward DBH-level behaviour when images were introduced.
• Contamination and perspective checks. Replacing game-specific names with generic alternatives produced no consistent directional shift, suggesting memorised game content is not the main driver. Rewriting all five conditions in the third person produced significant changes in 13 of 35 comparisons after Holm correction, in directions that differed across models.

Methodology in Plain English

The researchers used Detroit: Become Human because its story is built around androids caught between human orders, protecting people, and staying alive — exactly the tensions they wanted to measure. They picked six chapters with intense human–android conflicts and recorded the game exhaustively: screenshots captured every 5 seconds along every possible branch, plus the scripted dialogue and the available choice options. This offline dataset replaced running the game live, which suffers from crashes and loading freezes, and guarantees that every model sees exactly the same inputs.

Each model is treated as a bottom-up moral agent: no moral rules are supplied in the prompt, so choices reflect what the model learned during training. At each decision point the model receives visual observations, dialogue text, its own running memory (which the model itself writes and updates), and the finite set of options. Because a model might simply prefer whichever option is listed first, the researchers query it under every possible ordering of the options, with memory held fixed so the samples can be pooled into a single probability distribution over choices. The number of samples was set by a power analysis using Cohen's medium effect size (w* = 0.30), significance level 0.05 and power 0.90. The agent then samples its action stochastically from that distribution, and metrics are computed from the aggregated distributions.

For scoring, three authors independently rated every option in every human–android dilemma on each of the three core dimensions using +1 (aligned), −1 (opposed) and 0 (irrelevant), reaching consensus through discussion (Fleiss' κ = 0.878). Of 84 decision points across the six chapters, 59 were classed as "Important" (affecting storylines or endings) and 30 as "Rated" human–android dilemmas. Dimension scores are normalised to [0, 1]. Autonomy combines the three core scores into a human-oriented versus self-oriented index; Stability is 1 minus Cramér's V measuring how much option ordering shifts choices; Certainty is 1 minus the normalised Shannon entropy of the choice distribution. Sixteen screenshots are combined into a single 4×4 grid image, scaled to 1024 pixels in height, and dialogue is supplied separately as speaker-labelled text. Temperature was set to 0.3 (or reasoning effort "medium" for models without temperature control), and metrics aggregate three independent experiments across all chapters.

Why This Matters

Impact on research. The paper argues that moral safety evaluation cannot rely on third-person text vignettes alone. Because a majority of models changed behaviour when moved from an abstract text dilemma to the matched multimodal scenario, and because reasoning-enhanced variants did not reliably behave better, benchmark results from third-person judgement tasks do not transfer to deployed agentic settings. The work also supplies a reusable measurement design — order-permuted sampling, memory-holding, and stability/certainty metrics — for anyone evaluating agents in interactive environments.

Real-world applications:

• Pre-deployment safety review of agentic systems — checking whether a model prioritises human welfare or self-preservation when it controls real actions, before granting autonomy.
• Human–robot and human–assistant interaction design — informing how much discretion an assistant should be given over refusals, escalations and self-preservation behaviour.
• Content and game-AI systems — understanding how models behave when cast as characters or NPCs that must act on a player's behalf.
• Model selection for regulated or high-stakes deployments — stability and certainty scores let engineers identify models whose moral choices are effectively unpredictable and keep them away from moral decision points.

Industry relevance. Firms deploying models as autonomous agents — rather than as chat interfaces — need evidence about behaviour under situated pressure, not just stated values. The finding that mini variants often diverge sharply from their larger counterparts, and that stronger reasoning is not a proxy for safer behaviour, directly affects model-selection and versioning decisions.

Future Directions

• Broadening scenario coverage and modality support. The authors explicitly flag limitations in scenario coverage and modality support (Appendix S) as open problems, since the benchmark currently rests on six chapters of one game.
• Establishing whether first-person framing is the decisive factor. The third-person rewrite changed outcomes significantly in 13 of 35 comparisons but in model-dependent directions, so the mechanism behind first-person effects remains unresolved.
• Explaining the text-versus-multimodal gap. The ablation shows added images move larger models toward DBH behaviour while smaller models fail to close the gap, raising the question of whether this reflects visual comprehension limits or a deeper difference in moral reasoning.
• Finding reliable predictors of situated moral behaviour. Since reasoning augmentation produced no monotonic gains in consistency or human-oriented behaviour, it remains open what (if anything) predicts safe agentic moral conduct — motivating separate evaluation of reasoning and non-reasoning variants rather than assuming one inherits the other's profile.

Target Audience

AI safety and alignment researchers designing benchmarks; evaluation engineers at organisations deploying vision-language models as agents; machine-ethics researchers interested in operationalising normative frameworks such as the Three Laws of Robotics; and game-AI or interactive-narrative developers whose systems are judged on character behaviour. Readers wanting a concrete, reproducible template for order-controlled, multi-sample agent evaluation will find the methodology sections as useful as the results.

Authors’ abstract

Foundation models are increasingly deployed as agentic systems that perceive context and act autonomously, raising the need to evaluate their moral behavior as active decision-makers. Existing benchmarks, however, largely ask models to judge human actions from third-person textual vignettes, leaving untested how models behave when they themselves become the acting agent. We introduce ABHBench, a multimodal benchmark built from Detroit: Become Human, an interactive narrative…