HEIR: Learning Graph-Based Motion Hierarchies

AI summary

Overview

Research area: Computer Vision / Computer Graphics / Motion Representation Learning (with connections to robotics and physics simulation).

Technical level: Intermediate to Advanced. The core idea is intuitive, but the formulation relies on graph neural networks, directed acyclic graphs, Gumbel-Softmax sampling, Neumann series reconstruction, and as-rigid-as-possible deformation solvers.

Scope (1 sentence): The paper introduces HEIR, a data-driven framework that learns interpretable parent-child motion hierarchies as directed acyclic graphs and applies them to 1D trajectories, 2D rotational systems, and dynamic 3D Gaussian splat scenes.

What This Paper Is About

Complex motion—whether a human body, a robotic arm, or a deforming 3D scene—is usually built from simpler motion components that inherit and modify each other's movements. Existing methods model these "motion hierarchies" either by hand-crafting them (e.g., skeletons, kinematic trees) or by using opaque neural black boxes. HEIR instead learns the hierarchy itself from observed motion data, discovering which motion element is the "parent" of which, and decomposing each element's motion into what it inherits from its parent versus what it contributes locally.

Key Contributions

1. A differentiable graph-learning formulation of motion hierarchies. The paper casts hierarchy discovery as learning edge weights on a proximity graph, where vertices are motion elements and directed edges encode parent-child dependencies. A hierarchy matrix H (a binary adjacency matrix representing a directed acyclic graph) is sampled from these weights using the Gumbel-Softmax trick, keeping the whole pipeline trainable end-to-end.

2. A residual motion decomposition with a principled decoder. Absolute velocity Δ^t at each element is expressed as a parent-inherited component (HΔ^t) plus a local residual δ^t. The decoder reconstructs absolute motion by summing residual contributions along ancestor chains—formally, a truncated Neumann series Σ H^l δ^t—which remains stable even if sampling occasionally produces cycles.

3. An extension to rotational inheritance. By predicting relative motion in polar coordinates (radial and angular velocity components) rather than Cartesian coordinates, the method captures rotational parent-child relationships such as moons orbiting planets. A graph Laplacian connectivity prior (algebraic connectivity λ₂) keeps the learned hierarchy well-formed and non-fragmented.

4. A scalable application to dynamic 3D Gaussian splatting. Each Gaussian becomes a graph vertex, and the learned hierarchy drives scene deformation through an as-rigid-as-possible solver, enabling user-specified edits (e.g., lifting a shovel, posing a body) that preserve structural integrity.

Main Findings

• 1D hierarchy recovery succeeds far beyond chance (73%). On a synthetic 11-node, 200-frame dataset with a known three-layer hierarchy (root → low-frequency → high-frequency motion), the method recovered a valid hierarchy in 73% of 1,000 independent training runs, versus roughly 5×10⁻⁷ for a Monte-Carlo random baseline. The paper notes there are 10¹⁰ candidate hierarchies but only ~50 valid ones, making this a strong signal.

• Rotational hierarchies are learned perfectly without noise, robustly with noise. On a synthetic planetary system (a star, planets, and moons in circular orbits), HEIR reconstructed 100% of hierarchies in the noise-free case and 73.6% under Gaussian noise (σ = 0.05).

• Dynamic 3D scene deformation outperforms the SC-GS baseline across all four D-NeRF scenes. On Excavator, Hook, Jumpingjacks, and Warrior, HEIR achieved higher PSNR, SSIM, and CLIP-I, and lower LPIPS. For example, on Hook, PSNR rose from 15.7 (SC-GS) to 18.3, and LPIPS dropped from 0.0954 to 0.0617.

• Qualitative deformations are more physically plausible. The baseline SC-GS introduced unnatural bending at shovel-body connections, exaggerated body distortions, and unrealistic limb deformations. HEIR preserved rigid geometry (e.g., only repositioning a shovel to keep the excavator's structure intact) and produced smooth, anatomically consistent limb movement.

• The local-motion regularizer is essential, not cosmetic. Without the ℓ₁ penalty on residual velocities, the model trivially explains all motion locally (δ^t = Δ^t) and collapses into a useless star-shaped hierarchy, bypassing any meaningful structure.

Methodology in Plain English

The researchers start with a sequence of observed positions for N motion elements (points, keypoints, or Gaussians) across time. They compute each element's velocity as the frame-to-frame difference.

They then build a "proximity graph": each element is connected to its k nearest neighbors in space. These neighbors are the only candidates that could serve as that element's parent. A small graph neural network (specifically a graph attention layer) looks at each candidate edge and assigns a weight—essentially, "how likely is it that this neighbor is my parent?"

To turn these soft probabilities into a hard, discrete hierarchy (each element has exactly one parent), they use the Gumbel-Softmax trick, which allows sampling a discrete choice while still passing gradients through it during training. They sample several candidate hierarchies per step and train the model to pick good ones.

The key insight is a residual decomposition: an element's full motion equals its parent's motion plus a small local correction. So a decoder reconstructs each element's total motion by walking up the hierarchy, accumulating parent motions and adding local residuals—implemented as a truncated series rather than infinite recursion.

Training minimizes two terms: (1) how well the reconstructed motion matches the observed motion, and (2) how small the local residuals are, which pushes the model toward explaining motion through inheritance rather than assigning every element independent motion.

For rotations, they switch the encoder to polar coordinates, predicting how the distance and angle between parent and child change over time, and add a graph connectivity penalty based on the Laplacian's second eigenvalue to avoid fragmented hierarchies.

For 3D scenes, each Gaussian splat is a vertex. At edit time, the user picks a Gaussian and moves it; the hierarchy identifies all its descendants, which are treated as controlled "handles," and an as-rigid-as-possible solver computes how the remaining Gaussians should move to preserve local structure.

Why This Matters

Impact on research. The paper reframes hierarchy discovery as a learnable, differentiable graph problem rather than a hand-designed template. This offers a shared formalism that could unify work across computer vision, graphics, and robotics—where motion hierarchy is currently handled with domain-specific heuristics. It also provides interpretability: unlike black-box deformation networks, the learned DAG can be read off directly, with edge weights showing which motion element drives which.

Real-world applications:

• 3D content creation and editing. Artists could grab one part of a scanned or reconstructed object and have the rest deform plausibly, with the hierarchy automatically determining what should follow.

• Character animation and motion retargeting. Learned parent-child structures could transfer motion between skeletons or body proportions without manual rigging.

• Robot motion planning and control. Decomposing whole-body tasks into inherited and local components could yield more modular, verifiable controllers.

• Video analysis and action recognition. The unsupervised discovery of motion groups and their dependencies could provide structured features for understanding human activity.

Industry relevance. Dynamic 3D Gaussian splatting is a rapidly growing area for real-time rendering and scene capture. A method that makes deformations of these scenes editable, interpretable, and physically coherent is directly relevant to gaming, AR/VR, film VFX, and digital twin applications—where large Gaussian scenes with hundreds of thousands of elements are now standard.

Future Directions

• Scaling beyond k-nearest neighbors to global connectivity. The current proximity graph limits parent candidates to close spatial neighbors. The authors suggest sparse global attention, dilated-radius neighborhoods, or random long-range edges to capture long-range motion dependencies (e.g., a hand driven by a torso).

• Supporting multi-parent hierarchies. Each element currently has exactly one parent, which cannot express motion influenced by multiple sources simultaneously (e.g., a cloth pulled by two hands). Generalizing to DAGs with multiple incoming edges or weighted blends is an open challenge.

• Inferring latent motion not present in training data. HEIR cannot discover the hierarchy of a part that stays still during training—so editing unseen configurations (like a shovel that never moved in training) remains difficult. Incorporating semantic or task-level priors could address this.

• Tighter integration of hierarchy with deformation. The hierarchy currently feeds an external ARAP solver. The authors propose selectively adding local rigidity during deformation guided by the learned hierarchy, which could produce even cleaner edit results and cleaner handling of contact and occlusion.

Target Audience

This paper is best suited for researchers and graduate students in computer vision, computer graphics, and robotics who work on motion representation, dynamic scene reconstruction, or neural deformation models—particularly those familiar with Gaussian splatting, graph neural networks, or NeRF-based dynamic scene methods. Practitioners building editable 3D content pipelines or animation systems will find the practical results on D-NeRF directly relevant. Readers without a background in graph learning or the Neumann series may find Section 3 dense, but the high-level framing and experimental results are accessible with moderate effort.

Authors’ abstract

Hierarchical structures of motion exist across research fields, including computer vision, graphics, and robotics, where complex dynamics typically arise from coordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on manually-defined or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals. We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent-child dependencies through graph neural networks. We evaluate our hierarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D Gaussian splatting scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable to a broad range of motion-centric tasks. Project Page: https://light.princeton.edu/HEIR/