AffectSim: A Controllable Interactive 3D Simulation Benchmark for Embodied Affective Perception

AI summary

Overview

• Research area: Embodied affective computing and human-computer interaction, at the intersection of 3D simulation, embodied agent navigation, and multimodal emotion recognition.
• Technical level: Intermediate. The paper is readable without deep simulation expertise, but assumes familiarity with emotion-recognition benchmarks, vision-language models, and embodied navigation metrics.
• Scope: AffectSim is a controllable, replayable 3D simulation benchmark of 27,647 emotion-labeled human-motion episodes in which the observation conditions (distance, orientation, occlusion, scene geometry, agent viewpoint) can be varied independently of the affective behavior being observed.

What This Paper Is About

Existing emotion-recognition benchmarks hand the model a recording that was already framed, positioned, and selected for it — the paper calls this the assumption of an "ideal photographer." AffectSim instead puts emotion-expressive human motion into an executable 3D world so the same affective performance can be replayed under systematically different observation conditions, or observed by an agent that moves to acquire better evidence.

The goal is to make observation itself an experimental variable, and to measure how much emotion recognition depends on what an embodied observer can see and at what cost, rather than only on the capability of the recognizer.

Key Contributions

1. A controllable interactive 3D simulation benchmark for embodied affective perception. AffectSim places affective human behavior inside executable 3D environments and exposes the observation process as part of evaluation. The authors state that to their knowledge it is the first affective benchmark to combine interactive 3D simulation, controlled re-observation of the same affective behavior, and agent-controlled observation.
2. Replayable and factorized affective episodes. The benchmark contains 27,647 interactive 3D episodes built by decoupling affective motion from scenes and observation conditions, spanning 57 scenes and supporting both single-person (26,207 episodes from 1,259 motion assets) and dyadic (1,440 episodes from 48 interpersonal motion assets) behaviors.
3. Large-scale evaluation of observation effects. 24 frozen emotion-perception model configurations (19 open-source, 5 closed-source) are benchmarked under matched observation protocols, revealing a substantial observation-quality gap; a proof-of-concept two-stage active-observation baseline improves 21 of 24 configurations.
4. Episode-level and path-aware diagnostics. Beyond aggregate Macro-F1, the paper defines Recovery Rate (how often active observation fixes an initially wrong prediction) and Emotion-SPL (E-SPL@r for r in {1, 3, 5} m) to characterize how efficiently a prediction is obtained through embodied movement.

Main Findings

• Observation quality is a large source of performance variation. With the underlying affective behavior and emotion label held fixed, P-Ref (a privileged reference trajectory using ground-truth body pose and scene geometry only during trajectory generation) substantially outperforms P-Init. For 19 open-source configurations, mean Macro-F1 rises from 9.89% to 15.56%; for five closed-source models it rises from 22.61% to 30.78%. P-Ref achieves the highest score for 17 of 19 open-source configurations and for 22 of 24 configurations overall.
• Active observation helps in most, but not all, configurations. A simple two-stage active-observation baseline improves 21 of 24 configurations over P-Init. Mean Macro-F1 increases from 9.89% to 11.70% for open-source models and from 22.61% to 24.26% for closed-source models, recovering 32.0% and 20.1% of their respective P-Ref–P-Init gaps.
• Gains are larger on episodes where new evidence is actually acquired. On the 1,881-episode new-observation subset (episodes where the active baseline records a valid post-handoff observation), open-source mean Macro-F1 goes from 10.95% under P-Init to 14.35% (+3.40 pp), and closed-source from 25.38% to 28.29% (+2.91 pp). On the remaining 2,060 episodes, A-Obs falls back to the paired P-Init input.
• An external monitoring camera is not equivalent to informative observation. P-Mon, a fixed room-level view, reaches 10.31% Macro-F1 for open-source models and 24.44% for closed-source models — only slightly above P-Init for open-source models (9.89%) and well below P-Ref (15.56%). The authors conclude that observation quality depends jointly on target visibility, framing, orientation, distance, and the discriminative motion cues exposed.
• Reference observation is a diagnostic, not an upper bound. P-Ref is generated without emotion labels, recognition predictions, or model confidence, and is described explicitly as neither an oracle nor an upper bound. OGR is therefore not clipped to [0, 1]; values above one indicate A-Obs surpasses P-Ref, and negative values indicate degradation relative to P-Init.
• Episode-level and efficiency results. Recovery Rate measures the fraction of initial P-Init errors corrected under A-Obs, and E-SPL@r (r in {1, 3, 5} m) combines final emotion-recognition success with path efficiency relative to the shortest collision-free NavMesh path to a distance-conditioned target region. Specific numerical values for these two diagnostics appear in the portion of the paper that is not included in the provided content.

Methodology in Plain English

The benchmark is built in three stages. First, emotion-labeled human performances from KDAE, Emilya, and a newly captured pilot dataset of 48 dyadic motion sequences are converted into reusable "affective motion assets." Because the source datasets use different emotion vocabularies, the authors define a shared five-class label space — angry, fearful, happy, neutral, and sad — and exclude categories without a clear cross-dataset correspondence rather than force-merging them. Motions are retained only if they can be faithfully instantiated, and they are retargeted to a common avatar (source-skeleton normalization, joint correspondence, scale alignment, body-pose transfer, root-motion conversion) and screened for skeletal distortions, root-motion discontinuities, foot sliding, and body penetration.

Second, an "episode composer" combines a motion with a 3D scene, a human placement, an agent initialization, and an observation challenge. The authors formalize an episode as e = C(m, S, p_h, x_0, c), where the emotion label comes from the motion and the difficulty comes from the embodied configuration. Construction splits into two modes: action-grounded composition, where semantics depend on the environment (seated behaviors aligned with support surfaces, knocking with door geometry, walking with navigable floor trajectories), and observation-controlled composition, which holds behavior fixed while applying one of four challenge families — far-visible, rear-view, partial-occlusion, and obstacle-detour. A Fetch mobile robot is placed on navigable floor space as the initial state, and motions are replayed cyclically to factor out event-timing uncertainty.

The implementation uses the Habitat simulation platform with indoor scenes from the Habitat Synthetic Scene Dataset (HSSD). Third, episodes pass automated validation (configuration integrity, reproducibility, physical validity, support alignment, hand–door geometry, collision-free trajectories, and verification that the intended challenge is realized) and human screening by 15 reviewers, who label each episode reasonable, unable to judge, or unreasonable; only reasonable episodes enter the inventory.

Evaluation compares four observation settings sharing the same scene, motion, label, rendering parameters, camera intrinsics, clip duration, and frame sampling: P-Init (stationary camera at the initial pose), P-Ref (privileged reference trajectory maintaining an informative relative camera pose), P-Mon (fixed room-level surveillance-style view), and A-Obs (the agent may move and decide when to stop acquiring). The A-Obs baseline combines ETPNav-based search with a handoff to a lightweight target-following controller, running at 4 Hz with 0.25 m translations and 15° rotations, a maximum budget of 480 primitive actions and 80 global planner calls, and no access to emotion labels, oracle trajectories, simulator masks, or recognizer outputs. The observation policy is deliberately decoupled from the recognizer so the same trajectories can be scored across all frozen models. Splits are group-disjoint at performer, source-performance, or interaction-sequence level; the same 57 scenes may appear across splits, so the protocol tests generalization to unseen performers and source motions within a shared scene inventory rather than to unseen environments.

Why This Matters

The paper argues that the field's collect–annotate–recognize paradigm conflates two different failure modes: failing to interpret affective evidence and failing to observe it in the first place. Because recorded observations are inseparable from their original distance, orientation, occlusion, and framing, simulation offers a way to run controlled, repeatable, counterfactual experiments on the observation process that prerecorded datasets cannot support.

Real-world applications:

• Care robotics: the paper's motivating example is a care robot that may need to notice sadness in a person before responding.
• Service robotics: a service robot may need to recognize frustration before offering help, which requires repositioning rather than relying on a fixed camera.
• Human–robot interaction evaluation: the Robot-centric Emotion Dataset and similar settings are cited as adjacent work, and AffectSim extends evaluation to agents that can physically change their own visual evidence.
• Embodied social intelligence and empathetic agents: systems such as EmpathyAgent that combine affective perception, memory, reasoning, and action generation currently take supplied evidence as given; AffectSim targets the upstream step of how that evidence is physically acquired.

Industry relevance: the benchmark provides a standardized way to test whether vision-language models and robot perception stacks degrade under realistic viewing conditions, and it separates the contribution of the recognizer from the contribution of the sensing policy — useful for anyone building robots or embodied assistants that must operate without a well-placed camera.

Future Directions

1. Stronger active-observation policies. The current two-stage baseline is described as proof-of-concept and improves 21 of 24 configurations while leaving most of the P-Ref–P-Init gap unrecovered; the reported Recovery Rate and E-SPL results are intended as reference numbers for future methods.
2. Temporally evolving and non-repeating behavior. Motions are currently replayed cyclically to isolate the spatial active-perception problem; the authors explicitly leave non-repeating and temporally evolving behaviors to future extensions.
3. Expansion of the asset and scene library. The modular episode abstraction is designed to accept new emotion-expressive motions, performers, dyadic behaviors, generated motions, avatars, scenes, and observation challenges without changing the evaluation protocol.
4. Richer social and affective settings. The dyadic pilot (1,440 episodes from 48 interpersonal motion assets) is a starting point; the authors frame AffectSim as a path toward richer studies of affective behavior and embodied social intelligence.

Target Audience

Researchers and engineers working on affective computing, embodied AI, human–robot interaction, and 3D simulation benchmarking; practitioners evaluating vision-language models or emotion-recognition systems under realistic viewing conditions; and roboticists building perception pipelines where the agent — not a dataset curator — decides what it gets to see. Readers looking for a state-of-the-art emotion classifier will not find one here, since no recognizer is fine-tuned on AffectSim.

Authors’ abstract

Existing affective benchmarks largely consist of fixed recordings whose observation conditions are determined before inference, making it difficult to systematically study how embodied sensing influences affective perception. We introduce AffectSim, a controllable interactive 3D simulation benchmark for embodied affective perception. Rather than treating affective samples as fixed recordings, AffectSim instantiates emotion-expressive human motions as replayable 3D episodes in which distance, orientation, occlusion, scene geometry, and agent viewpoint can be systematically varied while preserving the underlying behavior and emotion label. AffectSim contains 27{,}647 episodes across five emotion categories and 57 scenes. Its factorized design separates affective behavior from observation conditions, supporting controlled re-observation of the same behavior as well as agent-controlled sensing in an executable 3D environment. To demonstrate this capability, we instantiate embodied emotion perception under matched initial (P-Init), reference (P-Ref), and actively acquired (A-Obs) observations. Across 24 frozen perception-model configurations, P-Ref substantially outperforms P-Init, while a simple two-stage active-observation baseline improves 21 of 24 configurations. Mean Macro-F1 increases from 9.89% to 11.70% for open-source models and from 22.61% to 24.26% for closed-source models, recovering 32.0% and 20.1% of their respective P-Ref--P-Init gaps. Episode-level recovery and path-aware evaluation further characterize the current baseline beyond aggregate recognition performance. These results demonstrate the value of making affective observation controllable and establish AffectSim as an initial platform for studying embodied affective perception through interactive 3D simulation.