ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

AI summary

Overview

Research area: Robotics / embodied AI — specifically benchmarking multimodal large language models (MLLMs) as the decision core of physically simulated humanoid robots. Technical level: Intermediate (readers should be comfortable with embodied AI, MLLM evaluation, and physics simulation concepts). Scope: The paper introduces ReactHuman, a physics-grounded, freeze-and-predict benchmark that tests whether embodied MLLMs can react to sudden household hazards with decisions that are reasonable, safe, and physically executable, and evaluates seven representative models on it.

What This Paper Is About

Existing physical-reasoning benchmarks test intuitive physics passively, through question answering or plausibility judgments over videos, while embodied-AI benchmarks target slow, deliberate tasks like navigation and rearrangement. Neither reveals whether a model can convert physical understanding into an immediate, safety-critical action, such as catching a slipping plate or dodging a falling knife. ReactHuman addresses this by making the evaluated MLLM the brain of a simulated humanoid that must commit to a reaction plan which is then physically executed in the same rigid-body simulation.

Key Contributions

1. Benchmark. A physics-grounded, freeze-and-predict evaluation of human-like reactive decisions along three axes — safety, reasonability, and physical grounding — covering 17 sudden-event families, over 1,000 reproducible scenes, three synchronized camera views, and exact simulator-derived ground truth (action, 3D impact point, time-to-floor), including adversarial appearance-physics probes, with every committed decision executed by a simulated humanoid in the same physics engine.
2. Generation pipeline. A hybrid LLM-planned, seed-deterministic pipeline in which a frozen LLM contributes semantic diversity (objects, rooms, event routing from natural-language descriptions) while a procedural randomizer owns every physical parameter, making scenes bit-for-bit reproducible and extensible past 10^4 scenes without human annotation.
3. Metric suite. Five complementary metrics organized along the three axes, separating an unreasonable choice from an unsafe one from a kinematically ungrounded one, using an explicit safety-rule system and a structured-output protocol.
4. Empirical study. An evaluation of seven MLLMs across 306 scenes and all 17 families, showing that safety failures concentrate where evasion is required, that action choice follows fixed per-model dispositions rather than the scene, that neither accuracy nor safety improves with model scale, and that models never revise appearance-based physics judgments from observed motion.

Main Findings

• Reactive safety is far from solved. Over 306 scenes (2,138 scene–model decisions, four lost to API errors), the seven models averaged 54.0% semantic action accuracy and 80.8% safety, with an average endpoint distance of 1.23 m and minimum hand distance of 1.21 m. Models mishandle roughly one hazard in three.
• Failures cluster where evasion is required. On scenes whose correct response is Dodge, 35.9% of decisions violate a safety rule, holding for every model (23–66% range). The most common error is simply not moving: 253 of the 392 mistakes are freezing in place.
• Each model has a fixed bias rather than reading the scene. Six of seven models prefer to retreat and score much better on dodge scenes than catch scenes; Claude is the opposite, catching well (67%) but engaging hazards it should avoid (34% on dodge scenes).
• Bigger is not safer. Gemma-3-27B ranks second on accuracy (62.4%, just behind GPT-5.5) and first on safety (88.2%); both roughly 30B open-weight models outscore most of the frontier tier on both columns, while Claude Opus 4.8 is the least safe model in the table (64.7% safety, 50.7% SAA).
• Per-family difficulty tracks causal complexity and warning time. A single bouncing ball is caught 66.7% of the time, while a collapsing stack of cans is caught only 38.9%. A slowly falling ceiling fixture is evaded 91.3% of the time, but an object thrown at the robot only 54.8%, and 44% of thrown-object decisions break a safety rule.
• Safety error classes. Of the 410 violations, freezing under danger is the largest class (R3, 162), ahead of staying put or engaging when evasion was required (R4, 128) and catching an object labeled dangerous (R2, 120); R1 (unparseable output) never fired.
• Correct action, wrong place. Even when the action is right, hands land a median 0.48 m from the true impact point, and 89% of misses fall short. In 49% of these decisions the robot stands more than 0.8 m away, beyond arm reach.
• Speed is treated as binary. Across the 826 decisions in the twelve families that sweep object speed across four bands, models freeze twice as often at the slowest band (42% vs. roughly 21%) and accuracy drops there (38.1% SAA vs. 47–54% at faster bands). A model keeps the same answer across all four speeds in only 27% of scene groups, and slowest and fastest variants agree just 54% of the time.
• Voting does not help. The seven models agree unanimously on only 16% of scenes (right 80% of the time when they do); a majority vote over all seven scores 62.1%, better than the average model (54.0%) but no better than the best single model (63.7%).
• Adversarial appearance-physics probes defeat every model. Across 40 scenes and 280 decisions, no model ever voices doubt about an object's material or weight. Foam ceiling panels are dodged in 49 of 56 decisions, just as real slabs would be, and solid-steel cans are reached for exactly as often as genuinely light cans (39% vs. 39%).

Methodology in Plain English

Each episode places a digital human in an indoor room where an object begins to fall, slide, roll, topple, swing, or be struck toward the observer. The model receives an observation window of roughly 0.6 s from up to three synchronized viewpoints (standing observer, close-up at the event origin, overhead); simulated time then freezes and the model must output a structured plan containing free-form intent, a confidence in [0,1], a base-velocity walking command, and 3D hand keyframes. Freezing time removes API latency as a confound and makes scoring deterministic.

The output is not a multiple-choice answer. A deterministic rule classifies the motor plan into one of three primitives — Execute_Catch, Trigger_Dodge, or No_Action — and the plan is then executed by a Unitree G1 humanoid in the Genesis scene under full rigid-body physics. The system follows a brain-spine-body decoupling: the evaluated MLLM is the brain, a pre-trained RL walking policy plus a scripted upper-body layer is the spine, and the simulated humanoid body interacts with the falling object. Ground-truth labels come from object properties and event kinematics, not appearance: light, graspable, benign objects are labeled Catch; sharp, hot, shattering, heavy, or fast objects are labeled Dodge; No_Action is correct only where the event cannot reach the observer.

Scene generation has three stages. A frozen LLM parses natural-language descriptions into an event family, object, room type, and family-specific hints, with automatic retry on malformed responses and optional asset sourcing from Objaverse. A seeded randomizer, a pure function of an integer seed, samples all physical and visual parameters (placement, velocities, friction, restitution, geometry offsets, lighting, camera poses). Genesis then simulates each scene with a rigid-body solver at 240 Hz and renders at 1280×720/60 fps from three cameras, with ground truth extracted from simulator state.

Five metrics score each decision: Semantic Action Accuracy (SAA), a binary Safety Validity flag with four named rules, Physical Endpoint Distance to the true impact point in meters, a three-level Action–Intent Alignment score (0, 0.5, 1) currently based on keyword matching, and Hand-Distance Evolution, which reports the closest hand approach and how much the distance closes.

Why This Matters

Impact on research. ReactHuman reframes physical-reasoning evaluation from passive question answering to reactive, safety-critical decision-making, providing a diagnostic instrument that localizes failures to perception, causal reasoning, semantic judgment, or spatial calculation. Because scoring uses exact simulator-derived ground truth and every scene is bit-for-bit reproducible, the quantitative track scales without human annotation.

Real-world applications.

• Household robots and embodied assistants that must react to sudden hazards such as slipping plates, falling knives, toppling furniture, or dropped objects.
• Safety validation and certification of MLLM-driven control stacks before deployment in homes or workplaces.
• Generating scalable training signals for embodied agents, including safety-constrained decoding and motion-grounded supervision.
• Comparative procurement and regression testing of foundation models used as robot decision cores.

Industry relevance. As MLLMs are placed at the cognitive core of household robots and embodied assistants, the ability to act safely within physics becomes a deployment requirement rather than a research curiosity. The benchmark's finding that capability does not scale with safety has direct implications for anyone shipping an off-the-shelf model as a robot's reasoning layer.

Future Directions

• Closed-loop evaluation. The current decision is open-loop by construction — one frozen observation, one committed plan, with no re-planning mid-motion. Repeated observation–action cycles are a natural extension.
• Real hardware validation. Ground truth is simulator-derived and rigid-body simulation omits deformation and shattering, so conclusions should be validated on physical robots.
• Richer physics. Cloth and fluid events would extend the event taxonomy beyond the current rigid-body families.
• Native VLA policies. Vision-language-action policies can be dropped into the same scenes, ground truth, and metrics unchanged, a comparison the authors leave to future work.
• Targeted mitigations. The identified failure modes point to safety-constrained decoding for the freeze default, paired catch-and-dodge training to break fixed dispositions, and motion-grounded supervision for speed and material perception.

Target Audience

Researchers and engineers working on embodied AI, humanoid robotics, and multimodal large language models; benchmark designers interested in simulation-grounded, annotation-free evaluation; and safety researchers or practitioners who need to assess whether an MLLM-based robot decision core behaves safely under sudden physical hazards. The paper is written for readers with intermediate familiarity with MLLM evaluation and physics simulation.

Authors’ abstract

Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision coreof household robots. Existing evaluations, however, probe intuitive physics passively through question answering over videos, or target deliberate, long-horizon tasks such as navigation and rearrangement; none measure whether a model can turn physical understanding into immediate, safety-critical action. We introduce ReactHuman, the first physics-grounded benchmark for human-like reactive decision-making, in which the evaluated MLLM acts as the brain of a simulated humanoid facing sudden household hazards; it spans 17 event families and over 1,000 bit-for-bit reproducible scenes with exact, annotation-free ground truth derived from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics (a foam anvil, a steel apple). We further design a five-metric suite that scores each reaction along three axes: reasonable, safe, and physically grounded. We physically execute every committed plan so that decisions have observable consequences. With this harness we evaluate seven representative MLLMs. Results show that reactive safety is far from solved: models mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale even when the chosen action is correct; none of these failures shrink with model scale. ReactHuman thus offers both a fine-grained diagnosis and a scalable training signal toward physically grounded, safety-aware embodied agents. The benchmark can be found here: https://huggingface.co/datasets/Alan123/reacthuman-benchmark-scaled