Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration

AI summary

Overview

Research area: Computer vision and robotics — specifically 3D human motion prediction (HMP) in settings where a robot is present as an active agent, covering both human-robot collaboration (HRC) and human-robot interaction (HRI).

Technical level: Advanced. The paper assumes familiarity with 3D pose forecasting, transformers, cross-attention, and frequency-domain (DCT) motion representations.

Scope: The paper proposes AINet, a dual-branch, attention-based model that jointly forecasts future human poses and future robot poses, and reports state-of-the-art MPJPE results on the CoMaD, CHICO, and HARPER benchmarks.

What This Paper Is About

Most human motion prediction research assumes a person moving alone, or several identical people interacting, or a person manipulating passive objects and static scenes. In human-robot collaboration none of these assumptions hold: the robot has a different body structure from the human, the two agents influence each other asymmetrically, and the robot's own autonomous movement constantly changes the context the human is reacting to. The paper's goal is "robot-aware" human motion prediction — explicitly modeling how a robot's motion shapes a human's future motion — and it introduces a dedicated network, AINet, to do so.

Key Contributions

1. Local-Global Spatial Interaction (LGSI) module. An attention-based module that decomposes human-robot interaction into fine-grained local cues (contact-sensitive, part-level geometry around the human hand/arm and robot end-effector) and coarse global cues (the overall interaction state such as approaching, separating, yielding, or handover), using an Align-Interact-Reconstruct pipeline that projects both agents into a shared latent space and then maps them back to their own domains.

2. Adaptive Weighted Aggregator (AWA). A module that fuses the local and global interaction features with context-dependent weights rather than fixed coefficients, so the model can shift emphasis between near-contact precision and broad coordination depending on the situation.

3. Traj-Pose Module with a learned pseudo-trajectory. A coarse-to-fine strategy in which an ST-Encoder and a Traj Module first produce a low-dimensional pseudo-trajectory aggregating multi-joint motion cues, and a Pose Module then refines detailed poses conditioned on it — avoiding the limitations of using a fixed hip-joint trajectory as a proxy for global motion.

4. State-of-the-art results on three benchmarks. The method is evaluated on CoMaD, CHICO (both HRC), and HARPER (HRI), and the authors report superior performance across all three.

Main Findings

• Best average MPJPE on CoMaD (Table II). With human and robot input ("HR"), AINet reaches an average MPJPE of 27.30, ahead of the next best result of 28.10 (GCNext, HR) and 28.17 (GCNext, H). With human-only input ("H"), AINet averages 27.88. The paper states the method achieves SOTA at all time steps (evaluated at 100 through 1000).

• Best average MPJPE on CHICO (Table III). With human and robot input, AINet averages 49.35, versus 50.37 for siMLPe (HR) and 51.28 for GCNext (HR). With human-only input it averages 50.49, versus 51.65 for siMLPe (H). SOTA is claimed at all time steps (80, 160, 320, 400, 560, 720, 880, 1000).

• Robot input helps some models and hurts others. On CoMaD, giving robot motion to siMLPe worsened its average from 28.42 (H) to 29.69 (HR), and IAFormer dropped from 30.94 (H) to 36.21 (HR). Methods such as GCNext and SeS-GCN improved when robot input was added. AINet improved in both datasets when robot input was added (CoMaD: 27.88 to 27.30; CHICO: 50.49 to 49.35).

• Robot-aware baselines were not the strongest competitors. SeS-GCN averaged 42.14 (HR) on CoMaD and 60.03 (HR) on CHICO; PoseTron averaged 38.12 (HR) on CoMaD and 64.83 (HR) on CHICO. ALIEN was the weakest on CoMaD with 53.17 (HR) and 41.98 (H).

• Fixed-joint trajectories are insufficient. Joint-wise motion pattern analysis (Fig. 3) shows HARPER's hip joints exhibit large displacements, while on CHICO and CoMaD motion concentrates in the upper limbs with nearly static hips — motivating the learned pseudo-trajectory instead of a fixed hip-joint proxy.

• HARPER quantitative results are not reported in the available content. The paper states that HARPER is used to test whether robot-aware modeling generalizes to interaction-rich HRI scenarios and that the method "has demonstrated superior performance on these datasets," but the numeric tables for HARPER lie beyond the truncated text.

• Ablation results are not reported in the available content. The paper describes evaluating a simplified version of AINet without the LGSI module and AWA using human-only input, but the corresponding numbers are not included in the excerpt.

• Training setup. Training used a single NVIDIA GeForce RTX 4090 GPU, batch size 256, roughly 6 GB of memory, and about 2 hours. On HARPER, training ran for 31,000 iterations with an initial learning rate of 0.0003 for the first 30,000 iterations, then reduced to 0.00001. The model takes 60 input frames and predicts 30 future frames.

• Evaluation metric. Mean Per Joint Position Error (MPJPE), the average Euclidean distance between predicted and ground-truth joint positions; lower is better.

Methodology in Plain English

AINet runs two symmetric branches side by side — one for the human, one for the robot — and trains them together, with human motion prediction as the primary task and robot trajectory estimation as an auxiliary task that forces the model to stay aware of both agents.

Each branch starts with a Motion Encoder that converts the input motion into the frequency domain using a Discrete Cosine Transform, which smooths the signal and suppresses high-frequency noise, followed by fully connected layers. Poses are kept in the world coordinate system rather than a local one, so the model can directly perceive spatial relationships between the human and the robot.

The two branches are then coupled by the LGSI module, which works in three steps. Align: MLPs project the human and robot skeletons into a shared latent space to bridge their structural differences; the local pathway uses a higher-dimensional space to retain rich geometry, while the global pathway uses a lower-dimensional space for abstract interaction states. Interact: multi-head cross-attention lets one agent attend to the other (robot features act as query, human features as key and value in the human branch), with skip connections preserving each agent's own motion. Reconstruct: fully connected layers and skip connections map the interaction features back to each agent's original skeleton space, so embodiment-specific semantics are retained.

The AWA module then decides how much to trust local versus global interaction features at each moment. It builds a context vector from the aligned global features, normalizes it, and uses small MLPs to produce two weights that are clamped to [0, 1] and normalized so they sum to 1. A regularization loss on these four weights (two per branch, for human and robot) keeps them in a meaningful range.

Finally, the Traj-Pose Module applies a coarse-to-fine strategy: a first ST-Encoder produces a spatio-temporal representation, a Traj Module maps it to a low-dimensional pseudo-trajectory, that trajectory is broadcast back across all joints and added to the features, and a second-stage encoder refines the detailed poses. A Motion Decoder then reconstructs the motion with fully connected layers and an inverse DCT, predicting the residual relative to the last observed pose. The total loss combines the human prediction loss, the robot prediction loss, and the weighted regularization term, where each prediction loss combines position error and velocity error (computed by temporal differencing).

Why This Matters

Impact on research. The paper reframes human motion prediction around an autonomous, heterogeneous partner rather than a passive object or a symmetric peer, and argues that single-person, multi-person, human-object interaction, and scene-aware methods each fail in HRC for a distinct reason: they ignore inter-agent coupling, assume homogeneous agents, or treat context as static. It also contributes a concrete architectural recipe (separate branches, shared-latent cross-attention, adaptive fusion, learned pseudo-trajectory) plus a fair-comparison protocol in which every baseline is evaluated both with human-only input and with human-plus-robot input.

Real-world applications.

• Collaborative manufacturing: predicting an operator's next poses alongside a robotic arm in factory assembly lines, as captured by the CHICO dataset (polishing workbenches in alternation, side-by-side concurrent tasks).
• Domestic service robotics: kitchen assistance such as handing over items, stirring food, and transferring objects from a cart, as captured by CoMaD with a robotic arm performing human-like assistive actions.
• Mobile/legged robot coexistence: interactions between a person and the Spot quadruped robot at various distances, including physical contact and collision-avoidance scenarios, as in HARPER.
• Safety-critical autonomy more broadly: the introduction situates human motion prediction in autonomous vehicles, pedestrian tracking, and robotics, where anticipating human movement supports responsive motion planning in shared workspaces.

Industry relevance. The paper notes that lower MPJPE reduces the uncertainty of predicted human motion, which can decrease conservative waiting and replanning during execution while enabling tighter safety margins in human-aware planning. That connects directly to deployment concerns in robotics pipelines, where collision checking and cost-aware forecasting are already used but typically without an explicit, coupled pose-level human-robot motion model. Code is released at https://github.com/LyTingHub/AINet.

Future Directions

• Robust quantitative reporting on HARPER. The HRI benchmark numbers are asserted but not shown in the available content; results that test whether robot-aware modeling generalizes beyond task-oriented collaboration would clarify the claim.
• Isolating the value of each component. The paper mentions a human-only, LGSI- and AWA-free variant but does not report its numbers here, leaving open how much each module contributes and whether robot input is essential or merely helpful.
• Generalizing across embodiments. The evaluated robots are two robotic arms and one quadruped, with 10, 9, and 23 joints respectively; whether the dual-branch, Align-Interact-Reconstruct design scales to other morphologies, multiple robots, or multi-human multi-robot teams remains an open question.
• Handling the cases where robot input hurts. Several baselines degrade when robot motion is added, suggesting that poorly modeled heterogeneity can inject noise; understanding and controlling that effect is a natural next step.
• Real-time and downstream integration. The training footprint (roughly 6 GB, about 2 hours on a single RTX 4090) is modest, but the paper does not report inference latency or closed-loop deployment with an actual planner.

Target Audience

Researchers and graduate students working on 3D human motion prediction, human pose forecasting, and human-robot collaboration/interaction; robotics engineers building human-aware planning and safety systems for shared workspaces; and computer vision practitioners interested in cross-agent attention architectures, adaptive feature fusion, and frequency-domain motion modeling. Readers should be comfortable with transformer attention mechanisms and pose-based evaluation metrics, since the paper is written at an advanced technical level.

Authors’ abstract

Human motion prediction during human-robot collaboration is critical for achieving safe and efficient interactions in shared environments. Unlike previous studies that have primarily focused on humans and objects, we focus on the robot-aware human motion prediction task, which explicitly models the influence of robots on human motion. This task presents new challenges, such as capturing human-robot heterogeneity and modeling complex spatial interactions. To address these issues, we develop an adaptive interaction network (AINet) model that consists of two branches: a primary branch that predicts future human motion and an auxiliary branch that estimates robot trajectories. The two branches are jointly optimized and coupled via a Local-Global Spatial Interaction (LGSI) Module, which effectively captures fine-grained and global contextual dependencies between human and robot motion sequences. Then, we introduce an Adaptive Weighted Aggregation (AWA) Module to dynamically fuse motion features using context-dependent weights, thereby increasing adaptability across diverse scenarios. Furthermore, we adopt a coarse-to-fine prediction strategy, in which a coarse pseudo-trajectory is first predicted, followed by the refinement of detailed human poses based on that trajectory. Extensive experiments on three challenging datasets reveal that our proposed method achieves superior performance, validating its effectiveness in human-robot collaboration scenarios. Our code is available at: https://github.com/LyTingHub/AINet.