SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

AI summary

Overview

• Research area: Computer Vision — monocular 4D (3D + time) reconstruction, Geometry Foundation Models, and structured motion perception.
• Technical level: Advanced. The paper assumes familiarity with Lie groups, SE(3)/se(3) representations, twist parameterization, and large feed-forward transformer backbones such as DINOv2 and DepthAnythingV3.
• Scope: The paper proposes a single feed-forward transformer, SM4RT, that jointly predicts 3D scene geometry, world-coordinate dense motion, and a compact kinematic decomposition of scene motion from monocular RGB video, and evaluates it on depth, 3D reconstruction, 3D tracking, and a new rigid-body deformation benchmark.

What This Paper Is About

Most existing motion perception methods represent motion as independent per-point displacements — either sparse point tracking or dense point-wise flow — which ignores the fact that real objects obey rigid-body kinematics and therefore move collectively. The paper's goal is to replace this over-parameterized, unstructured displacement view with a representation in which motion itself has geometric structure: a small set of rigid-body motion bases shared across time, combined by per-pixel assignment weights. SM4RT is the resulting end-to-end transformer that recovers geometry and this structured motion in one forward pass from monocular RGB video, with no auxiliary inputs and no test-time optimization.

Key Contributions

1. Structure-of-Motion (SoM) representation. Scene motion is decomposed into a compact set of N motion bases, each a temporal sequence of 6D twists in se(3). Dense scene motion is recovered by sparse, time-shared per-pixel assignment weights over these bases, so points on the same object share a common rigid-body motion trajectory. The paper frames this as shifting from "geometry with motion" to the "geometry of motion."
2. A parameter-efficient architecture and training scheme. SM4RT introduces a parallel Motion Geometry Encoder that distills dynamic cues from intermediate geometry features, and a Motion Geometry Decoder with a Base Head (Dense Prediction Transformer producing per-pixel base weights, followed by SparseMax) and a Motion Head (MLP decoding motion tokens into twist sequences). Training combines geometry supervision, motion point supervision, and base regularization (entropy regularization, pseudo-background regularization, and twist singularity regularization).
3. A DoF reduction argument. Instead of predicting 3SHW degrees of freedom for point-wise locations, SM4RT predicts an assignment map plus temporal motion bases with NHW + 6SN DoF, which the authors describe as parsimonious, physically grounded, and only weakly scaling in the number of frames.
4. Structured temporal reasoning capability and a new benchmark. Operating in se(3) twist space enables structured motion interpolation and extrapolation that preserves object structure, unlike parameter-free linear tangent interpolation or cubic B-splines. The paper also proposes Kubric-Rigid-Eval, described as the first benchmark for 3D rigid-body motion reconstruction, containing 147 rigid-object videos with ground-truth segmentation and world-centric motion, evaluated with a von Mises deformation score (VM) alongside tracking metrics.

Main Findings

• Depth estimation is competitive, not uniformly best. On the MonST3R video depth protocol, SM4RT reports AbsRel / δ<1.25 of 0.054 / 97.49 on Bonn, 0.169 / 75.51 on Sintel, 0.056 / 97.20 on KITTI, 0.067 / 92.91 on 7Scenes, and 0.029 / 98.86 on ScanNetV2. DepthAnythingV3 reports 0.046 / 97.38, 0.186 / 70.66, 0.052 / 97.35, 0.066 / 92.74, and 0.032 / 98.04 on the same benchmarks.
• Best overall 3D reconstruction on DA3Bench. SM4RT reports Pose (Auc3 / Auc30) and Reconstruction of 86.16 / 98.36 and 89.89 on HiRoom, 43.88 / 91.56 and 74.86 on ETH3D, 93.30 / 99.32 and 1.153 CD on DTU, 27.91 / 86.76 and 54.31 on 7Scenes, and 81.45 / 97.88 and 78.50 on ScanNet++. The paper states SM4RT achieves the best overall performance across all five benchmarks.
• Best tracking on ADT and DriveTrack in TapVid3D minval. SM4RT reports AJ / APD of 32.5 / 40.6 on ADT and 24.3 / 30.1 on DriveTrack, versus 4RC at 30.2 / 38.5 and 23.1 / 28.3, and V-DPM at 28.3 / 35.3 and 23.1 / 28.6. On PStudio SM4RT reports 16.7 / 23.6 and the paper acknowledges the best performance there is achieved by other baselines.
• Strong world-coordinate tracking results. Under Global Alignment over 64 frames, SM4RT reports All/Dynamic APD and EPE of 88.56 / 80.68 and 0.1339 / 0.0935 on ADT, 82.61 / 71.62 and 0.1918 / 0.2912 on DS, 81.62 / 73.68 and 0.2578 / 0.3808 on PO, and 78.69 / 78.69 and 0.1790 / 0.1790 on PStudio. Since D4RT is not open-source, the authors adapt OpenD4RT to report its results under the same setting.
• Adaptive grouping trades tracking accuracy for structure preservation. On Kubric-Rigid (24 frames), raw SM4RT reaches AJ 84.37 and APD 90.62 with VMfg 0.1905 / 0.1876, VMbg 0.2452, and initial VMfg 0.0122. SM4RT (Adaptive), which applies hdbscan to group regions, reports AJ 82.54 and APD 89.56 with VMfg 0.0254 / 0.0191, VMbg 0.2499, and initial VMfg 0.0119 — the paper states this nearly eliminates deformation caused by dynamic motion while causing limited tracking degradation, possibly due to inconsistent base assignments for some objects.
• More coherent qualitative motion. Compared with 4RC, V-DPM, Any4D, and D4RT, SM4RT produces more coherent tracklines that better adhere to object structure, and more coherent projected motion with fewer scattered points, especially near motion boundaries. SM4RT is visualized with directly predicted motion, while SM4RT (Adaptive) applies adaptive grouped averaging per motion base assignment.
• Assignment maps expose skeleton-aware structure. The weight assignment output provides a structural segmentation that can be simplified into a dynamic mask, and the paper notes it clearly delineates articulated parts undergoing movement rather than treating the object as a monolithic whole. Existing entity-level motion segmentation evaluations are described as unsuitable because static parts of an entity would not be assigned as dynamic.
• Acknowledged limitation. Like existing parameter-free prediction methods, SM4RT cannot explicitly handle complex physical interactions such as collisions, rebounds, or contact dynamics.

Methodology in Plain English

The authors start from a gap in existing work: point-wise displacement models fit every pixel independently, even though a scene with N rigid objects really has only 6N motion degrees of freedom per target frame. Rather than predicting where each point goes, SM4RT predicts how groups of points move.

Concretely, the model predicts two things. First, a per-pixel assignment map that says which latent motion basis each source pixel belongs to — this map is shared across target frames, so it encodes which points move together. Second, N temporal motion bases, each a sequence of 6D twists (a translation component and a rotation component) that describes how one latent entity moves over time. A dense motion field is obtained by taking a weighted sum of these twists in the twist tangent space, then applying the exponential map to get a per-pixel rigid transform in SE(3), which is applied to source-frame points.

Architecturally, the input video is patchified and encoded with a pretrained DINOv2 backbone, then passed through a geometry-aware backbone (a pretrained DepthAnythingV3 with local-global attention) to produce geometry tokens. A Scene Geometry Decoder with a Camera Head and a Depth Head produces camera parameters and depth, unprojected to source-frame world coordinates. In parallel, a Motion Geometry Encoder uses N motion tokens initialized with per-frame time embeddings, cross-attending to intermediate geometry token layers and injected with those layers after each of four frame motion attention layers. A Motion Geometry Decoder then produces per-pixel base weights (Base Head, with SparseMax for sparsity) and twist sequences (Motion Head). An optional post-processing step applies hdbscan clustering to the assignment map and replaces each masked region's base with its regional average, yielding SM4RT (Adaptive).

Training uses a weighted sum of geometry, point, and base losses. Depth is supervised with a normalized ℓ1 loss over valid pixels, camera parameters with a standard ℓ1 loss, and scale with a log-difference loss pushing the mean Euclidean distance of source points toward 1. Motion point supervision differs between dense and sparse annotations — dense data decouples foreground and background via mask with a downweighting factor α on background, sparse data does not. Base regularization adds entropy regularization for sparse unambiguous assignments, a pseudo-background constraint that penalizes the dominant static base from bleeding into non-background regions, and a twist singularity regularization on translation components to prevent the dense SE(3) field collapsing into translation-only motion. No auxiliary inputs such as depth maps or segmentation masks and no iterative test-time optimization are used.

Why This Matters

Impact on research. The paper reframes motion perception from point-wise displacement estimation to structured kinematics, arguing that accurate point-wise displacement does not by itself imply meaningful dynamic understanding. It shows that structure and accuracy are mutually reinforcing rather than conflicting, and it introduces Kubric-Rigid-Eval and a von Mises deformation metric to measure rigid-entity structure preservation, which existing point-wise metrics cannot capture. It also extends Geometry Foundation Models from static 3D reconstruction into 4D, and unlike Shape of Motion it learns an amortized Structure-of-Motion inside a feed-forward model rather than through per-scene optimization with external priors.

Real-world applications (as framed by the paper and its citations):

• Embodied intelligence, where agents need to reason about how parts and objects move rather than only where points are.
• Autonomous driving, where world-coordinate tracking of moving objects is needed for scene understanding.
• Structured motion interpolation and future state prediction grounded in rigid-body kinematics — capabilities the authors argue point-wise displacement methods cannot provide.
• Dynamic mask and part-level motion structure extraction, useful for downstream scene decomposition tasks.

Industry relevance. Motion-specific modules account for 0.17 billion of the model's 1.38 billion parameters, and inference is a single forward pass with no test-time optimization, which matters for deployment settings where per-scene optimization is impractical. Because the base architecture follows Depth Anything 3 with pretrained DINOv2, DepthHead, and CameraHead weights, the approach is designed to sit on top of existing geometry foundation model infrastructure.

Future Directions

• Handling physical interactions. The authors explicitly note SM4RT cannot handle collisions, rebounds, or contact dynamics, leaving interaction-aware motion modeling as an open problem.
• Improving adaptive grouping consistency. The tracking drop in SM4RT (Adaptive) is attributed to potentially inconsistent base assignments for some objects, suggesting assignment refinement as a next step.
• Extending the rigid-body benchmark and metric. Kubric-Rigid-Eval and the von Mises deformation score are presented as the first benchmark for 3D rigid-body motion reconstruction; expanding coverage and metric derivation is a natural extension (the full derivation and protocol are deferred to Appendix A, which is not included in the provided content).
• Deformable object motion. The paper argues deformable motion can be expressed as a linear combination of neighboring skeletal rigid motions, but does not report dedicated deformable-object experiments in the provided content.

Target Audience

This paper is most valuable to researchers and engineers working on 4D reconstruction, dynamic scene understanding, and geometry foundation models, particularly those already familiar with SE(3)/se(3) representations and feed-forward 3D reconstruction systems. It will also interest practitioners in robotics and autonomous driving who need world-coordinate motion estimation from monocular video, and readers of prior work such as D4RT, 4RC, V-DPM, Shape of Motion, and DepthAnythingV3 who want to understand how structured kinematic decomposition compares against point-wise displacement formulations.

Authors’ abstract

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements. Building on this insight, we propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces Structure-of-Motion to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in SE(3). Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from monocular RGB video. SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.