TERRA: Terrain-Aware Reconstruction, Retargeting and Control for Musculoskeletal Locomotion

AI summary

Overview

Research area: Physics-based musculoskeletal simulation, reinforcement learning for motor control, motion retargeting, and motion-conditioned terrain reconstruction (arXiv:2609.38653v1 [cs.RO], 29 Sep 2026).

Technical level: Advanced. The paper assumes familiarity with Hill-type muscle models, sequential quadratic programming, motion-imitation RL, and biomechanics measurement (EMG, ground reaction forces).

Scope: An end-to-end pipeline that recovers collision terrain from scene-less motion-capture kinematics, retargets that motion onto a full-body muscle-actuated model under anatomical and contact constraints, and trains a single terrain-aware tracking policy on a 9.4-hour, five-source motion library.

What This Paper Is About

Musculoskeletal models driven by reinforcement learning can now reproduce complex human movement, but almost exclusively on flat ground. Two obstacles stand in the way: existing motion-capture datasets rarely ship with aligned terrain geometry, and transferring human motion onto a complex musculoskeletal body demands more than matching joint positions—contacts, penetrations, skating, floating, and anatomical limits must all be respected, and these errors get amplified on non-flat terrain. TERRA's goal is to close that gap by turning plain kinematic trajectories into motion–terrain pairs that a single muscle-actuated controller can track.

Key Contributions

1. Motion-only terrain reconstruction with structured priors. TERRA infers task-relevant support geometry from kinematics alone, combining positive surface evidence (stationary, low ankle/toe landmarks), negative free-space evidence (the rest of the body trajectory), foot-orientation cues, and swing-clearance profiles to distinguish flat supports, ramps, staircases, and seats.
2. Terrain-aware musculoskeletal retargeting. TERRA adapts OmniRetarget's interaction-mesh objective and adds target-specific anatomical, tendon-continuity, contact-anchor, clearance, collision, and seat signed-distance residuals, solved as a sequential quadratic program over 89 generalized coordinates.
3. A single terrain-aware muscle-actuated policy. Using motion–terrain pairs from five datasets, TERRA trains one reference-conditioned controller on 9.4 hours of diverse locomotion, spanning ramps, staircases, independent horizontal supports, and seats.
4. Benchmarks plus a physiological comparison. The paper evaluates reconstruction against ground-truth terrain and against paired meshes, evaluates retargeting and held-out tracking against three baselines and two ablations, and qualitatively compares policy-generated muscle activity and vertical ground-reaction forces with held-out human EMG and force-plate data.

Main Findings

• Reconstruction accuracy: TERRA reached 99.9% terrain-family accuracy over a pooled denominator of 5,474 labeled clips (3,476 Gait120 + 1,282 Darmstadt + 716 Vielemeyer). Removing the foot-orientation and swing-clearance cues dropped accuracy to 73.5% while leaving other metrics largely unchanged.
• Geometry fidelity: The subject-clustered 95% CI for pooled family accuracy was 99.853–99.982% (144 dataset–subject clusters). Across independent captures, ramp grade MAE was 0.544° and 0.529° at 7.5° and 10°, and stair riser MAE was 7.36, 4.34, and 3.61 mm at 100, 170, and 240 mm. Table II(a) lists TERRA grade MAE values of 0.223 ± 0.244 and 0.536 ± 0.353, riser MAE values of 8.08 ± 6.99 mm and 5.17 ± 4.36 mm, and stool MAE 45.62 ± 13.57 mm. Baselines on the same table include contact least squares with 490.00 ± 0.00 mm stool MAE and 150.49 ± 57.41 mm riser MAE, and Voronoi with 24.81 ± 36.39 mm stool MAE; the paper states TERRA outperformed the baselines on all terrain categories.
• PRISM mesh comparison: All three methods succeeded on 31/31 PRISM motions. TERRA reported foot height MAE 12.95 ± 6.56 mm, seat 11.16 ± 5.93 mm, observed 12.20 ± 6.36 mm, raised footprint coverage 90.8 ± 12.4%, flat coverage 98.0 ± 0.6%, and IoU 47.3 ± 5.5%. Voronoi reported 34.13 ± 10.02, 15.44 ± 8.38, 26.29 ± 13.15 mm, 98.4 ± 4.6%, 40.3 ± 28.1%, and 11.2 ± 17.9% respectively; contact least squares reported 102.41 ± 34.09, 204.21 ± 19.14, 145.10 ± 57.89 mm, 99.1 ± 5.0%, 37.1 ± 5.8%, and 3.5 ± 1.0%.
• Motion–terrain consistency on AMASS: TERRA reported 993/993 successful reconstructions, contact MAE 10.24 ± 9.21 mm, 97.1% of events within 50 mm, 0.7% penetrating and 2.2% floating, raised-support precision/recall/F1 of 98.6/89.7/93.9%, toe–ankle pair MAE 9.01 ± 7.46 mm, and free-space penetration 0.077 ± 0.444%. Voronoi succeeded on 966/993 with contact MAE 25.57 ± 25.57 mm and 0.000 ± 0.000% free-space penetration; contact least squares succeeded on 993/993 with contact MAE 47.46 ± 30.38 mm, 34.5% penetrating and 8.0% floating.
• Retargeting quality: TERRA converged on 6498/6498 retargeting runs and achieved the best score on all five physical violation metrics and contact-timing F1: D_ten 0.026 ± 0.185%, D_col 0.52 ± 1.34%, D_pen 0.16 ± 2.46%, D_skat 1.80 ± 1.87%, D_float 1.02 ± 3.75%, and contact F1 93.80 ± 4.06%. Its residual terms raised contact F1 from 88.49% to 93.80% and cut terrain penetration from 33.45% to 0.16%. For comparison, OmniRetarget reached 86.05 ± 11.79% contact F1 and 31.97 ± 29.22% penetration; GMR reached 74.19 ± 21.01%; MM-MoCap-Body reached 92.86 ± 7.47%.
• Held-out policy tracking: TERRA-retargeted references produced the highest completion rate at 87.59 ± 0.18% with MPJPE 74.47 ± 1.51 mm. OmniRetarget reached 80.29 ± 1.39% success and 87.23 ± 2.92 mm; GMR reached 56.43 ± 0.64% and 91.97 ± 1.73 mm; MM-MoCap-Body reached 47.34 ± 0.22% and 90.26 ± 0.60 mm. TERRA+Voronoi achieved better MPJPE (71.65 ± 1.00 mm) but lower success (82.16 ± 0.41%).
• Per-family tracking: TERRA obtained the highest success rate on flat ground (91.70 ± 0.46% on N=262), ramps (99.49 ± 0.18% on N=286), boxes (46.45 ± 4.45% on N=61), and seats (89.32 ± 0.82% on N=246), plus the lowest MPJPE on flat ground (70.01 ± 2.55 mm), ramps (66.74 ± 2.92 mm), and boxes (82.50 ± 0.77 mm). TERRA+Voronoi was best on stairs (85.01 ± 0.24%) and on seat MPJPE (83.09 ± 1.13 mm).
• Robustness: Under 2, 5, and 10 mm RMS landmark noise, overall accuracy was 99.91%, 99.51%, and 97.06%. Removing 10%, 20%, and 30% of support intervals yielded 97.30%, 94.40%, and 90.04% overall accuracy, with stools degrading most (89.46%, 79.84%, 70.36%). Perturbing thresholds to 80% and 120% gave 99.95% and 99.91%.
• Physiological comparison: Policy-generated and human EMG and vertical-GRF traces were independently min–max normalized and compared across flat ground, ramp descent, stair descent, and sit/stand, using three representative muscles (vastus lateralis, vastus medialis, and others listed as SOL, GAS, BF, TA) with between-subject SEM shading. The paper reports both similarities and visible mismatches in timing and shape, and describes the comparison as qualitative—no quantitative agreement metric is reported.

Methodology in Plain English

Terrain recovery from motion alone. The system watches which ankle and toe landmarks stay nearly still and low; those intervals are evidence that a surface exists there. The rest of the moving body acts as negative evidence—space it passes through cannot contain terrain. A joint-specific offset (from a subject-matched flat reference, self-calibration, or the lowest supported level) converts landmark height into surface height, and heights are clustered into distinct levels at 4 cm separation with recursive splitting of groups wider than 8 cm. Foot pitch during stance is compared against the pitch a flat or inclined surface would predict; when the two errors are within 2°, a swing-clearance profile decides between a near-linear ramp transition and the early-ascent/late-descent signature of discrete steps. Staircases are only chosen if their treads are within 5 cm of every raised stationary foot and no other landmark penetrates them by more than 5 cm. Seats are detected from slow pelvis motion (< 0.15 m/s for at least

Authors’ abstract

Recent advances in musculoskeletal modeling and reinforcement learning have enabled muscle-actuated agents to reproduce increasingly complex human motions. Yet these capabilities remain largely confined to flat ground, in part because motion datasets rarely include aligned terrain geometry and because retargeting terrain interactions to complex musculoskeletal bodies is challenging. We present TERRA, an end-to-end pipeline for terrain-aware retargeting and control of musculoskeletal locomotion. From kinematic trajectories alone, TERRA combines terrain priors, estimated contacts, and negative free-space evidence to recover task-relevant support geometry. TERRA further considers anatomical, tendon-continuity, and contact constraints during retargeting. Using the resulting motion-terrain pairs from five datasets, we successfully train a single muscle-actuated control policy on 9.4 hours of diverse locomotion. Across reconstruction, retargeting, and held-out tracking benchmarks, TERRA improves terrain accuracy, sharply reduces anatomical and interaction violations, and achieves the highest observed completion rate over supported terrain families. Overall, TERRA provides a practical route from scene-less motion data to muscle-actuated locomotion over diverse non-flat terrain. Project website: https://cnai.epfl.ch/terra/