MoRel: Long-Range Flicker-Free 4D Motion Modeling via Anchor Relay-based Bidirectional Blending with Hierarchical Densification

AI summary

Overview

Research area: Computer vision, specifically dynamic 3D scene reconstruction and novel view synthesis using Gaussian Splatting extended into the temporal (4D) domain.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), deformation fields, anchor-point representations, and differentiable rendering pipelines. The mathematics of the blending and densification schemes (learnable temporal opacity, quantile-based level assignment, gradient-modulated densification) is dense.

Scope in one sentence: The paper proposes MoRel, a 4D Gaussian Splatting framework that reconstructs long-range dynamic video content with bounded GPU memory and suppressed temporal flickering by chaining key-frame anchors through bidirectional deformations and learnable temporal blending, and it introduces the SelfCap_LR benchmark to measure this capability.

What This Paper Is About

Existing 4D Gaussian Splatting methods break down on long videos. Training on all frames at once ("all-at-once") causes GPU memory to grow with sequence length, while splitting a video into independently trained chunks ("chunk-based") produces visible flickering and appearance jumps where chunks meet, and fails to reconstruct regions that only become visible in later chunks.

MoRel's goal is to keep memory use bounded while still producing temporally coherent, flicker-free reconstruction of scenes containing long-range motion, including appearing and disappearing occlusions.

Key Contributions

1. Anchor Relay-based Bidirectional Blending (ARBB). A two-phase, four-stage training strategy (Global Canonical Anchor, Key-frame Anchor, Progressive Windowed Deformation, Intermediate Frame Blending) that places Key-frame Anchors (KfAs) periodically and fuses neighboring anchors through a learnable temporal opacity control, rather than interpolating linearly.

2. Feature-variance-guided Hierarchical Densification (FHD). A densification scheme that assigns each anchor-point a hierarchical level (0, 1, or 2) based on the variance of its learned features, using quantile thresholds, then modulates the densification gradient with level-specific weights that shift emphasis from low-frequency stabilization early in training to high-frequency refinement later.

3. Progressive Windowed Deformation (PWD) with on-demand loading. Each KfA is optimized independently inside its own Bidirectional Deformation Window (BDW), which the paper states avoids a failure mode it names backward contamination, where a KfA updated for a later chunk loses characteristics needed for the earlier chunk.

4. A new benchmark, SelfCap_LR. Five challenging dynamic sequences (Bike 1, Bike 2, Corgi, Yoga, and Dance) totaling over 3500 frames, described as having larger average dynamic motion magnitude and wider captured spaces than previous dynamic video datasets.

Main Findings

• Best average reconstruction quality on SelfCap_LR (Table 1). MoRel reports 21.00 PSNR / 0.664 SSIM / 0.355 LPIPS averaged over the five sequences, versus 18.95 / 0.648 / 0.402 for 4DGS, 19.61 / 0.643 / 0.391 for MoDec-GS, 20.64 / 0.652 / 0.371 for LocalDyGS, 19.02 / 0.653 / 0.405 for GIFStream, and 19.31 / 0.656 / 0.389 for the authors' own chunk-wise 4DGS adaptation.

• Lowest temporal flicker (Table 2). MoRel achieves the best tOF score of 0.203, compared with 0.222 (4DGS), 0.249 (MoDec-GS), 0.215 (LocalDyGS), 0.539 (GIFStream), and 0.680 (4DGS_chunk). Chunk-based methods score worst on this metric, which the paper attributes to flickering at chunk boundaries.

• Bounded, mid-range memory use. MoRel reports approximately 6,000 MB training memory and 126 MB rendering memory. All-at-once methods use far more training memory (~12,000 MB for LocalDyGS, ~18,000 MB for 4DGS, ~22,000 MB for MoDec-GS). The chunk-based methods use less (GIFStream ~9,000 MB / 93 MB; 4DGS_chunk ~4,500 MB / 65 MB) but at substantially worse quality and tOF.

• Strength on large-motion scenes. The paper states MoRel shows particularly strong advantages on Corgi, Yoga, and Dance, and that all-at-once approaches lose motion expressiveness and fine detail for frame ranges of 2k or more.

• Ablation results on a 300-frame SelfCap_LR subset (Table 3). Progression from a GCA-only unidirectional deformation baseline (19.71 PSNR / 0.654 SSIM / 0.386 LPIPS; ~12,000 MB training, 156 MB rendering) to the full model (21.20 / 0.672 / 0.348; ~6,000 MB, 126 MB). Adding KfAs (variant b) cuts memory sharply (to ~4,500 MB / 94 MB). Adding PWD with linear blending (variant c) and then IFB (variant d) improves fidelity (20.66 → 21.07 PSNR). Adding FHD (variant e) gives nearly identical or slightly improved performance while reducing rendering memory usage.

• Learnable blending beats linear blending. The paper attributes the gain from variant (c) to variant (d) to the learnable opacity control, which assigns each anchor-point its own temporal offset and decay speed so that irregular motion and occlusion are better modeled.

Methodology in Plain English

The approach divides a long video into time ranges governed by "key-frame anchors" — periodically placed canonical 3D models, one per time index, spaced by a Group-of-Pictures (GOP) interval. Rather than training one giant model over the whole video, the system trains a single global anchor first to establish global consistency, then derives each key-frame anchor from it, then teaches each anchor to deform forward and backward in time within its own local window, and finally learns how to blend two adjacent anchors together.

Blending is the key to avoiding flicker. Instead of hard-switching between anchors or linearly interpolating based on how far a frame is from each anchor, each anchor-point gets its own learned temporal offset and decay speed, so its opacity fades in and out in a way tuned to the actual motion. This is expressed as an exponential decay of opacity with distance in normalized relative time.

Memory is controlled by loading only the one or two anchors (and their deformation fields) needed at any moment, then unloading them. The same principle applies at render time. The densification scheme watches the variance of each anchor-point's learned feature as a proxy for how much high-frequency detail is nearby: unstable, high-variance points are handled more cautiously early and refined more aggressively later, so the model does not waste anchors on unstable regions.

Evaluation is on the authors' new SelfCap_LR dataset using PSNR, SSIM, LPIPS, and tOF (temporal optical flow error between consecutive frames), plus training and rendering memory.

Why This Matters

Impact on research: The paper frames long-range 4D modeling as a three-way trade-off between memory, temporal consistency, and system applicability (including random temporal access). It argues that neither all-at-once nor chunk-based nor sliding-window nor Temporal Gaussian Hierarchy approaches resolve all three, and positions ARBB as addressing them jointly. It also contributes a benchmark (SelfCap_LR) specifically designed to stress long-range motion, which may shift evaluation practice away from short-clip datasets.

Real-world applications:

• Streaming or on-demand viewing of volumetric/immersive video, where only a segment of a long capture needs to be decoded and transmitted — the paper explicitly notes that periodic key-frame anchors provide natural random access points.
• Reconstructing long real-world captures (minutes rather than seconds) for virtual production, telepresence, or immersive metaverse services, which the paper's funding acknowledgment references.
• Archival and replay of dynamic events from multi-camera rigs where the full sequence cannot fit in GPU memory at once.
• Content where objects appear and disappear over time (the "disocclusion" problem the paper emphasizes) — for example, scenes with moving occluders.

Industry relevance: Bounded memory and random access are practical deployment constraints, not just academic metrics. The paper's motivation for avoiding external optical flow and hierarchical CPU–GPU streaming is explicitly about reducing system complexity relative to prior long-range methods.

Future Directions

• Reducing the remaining gap between MoRel's training memory (~6,000 MB) and that of chunk-based methods (~4,500 MB for 4DGS_chunk), though the latter's tOF of 0.680 suggests that savings come at a large quality cost.
• Determining how GOP and the temporal tolerance are best chosen for different motion speeds; the paper defines these as inputs but does not report a systematic study in the provided content.
• Extending evaluation beyond SelfCap_LR: the paper mentions benchmark results on DyCheck-iPhone, HyperNeRF, DyNeRF, and PanopticSports in the supplementary material, but those results are not reported in the content available here.
• Since FHD's level count, thresholds, and the initial importance factor for higher levels are hyperparameters, a natural next question — the paper says analysis is in the supplementary material, not in the main text — is how sensitive the memory/quality balance is to these settings.
• Training time and inference frame rate are not reported in this content, leaving the throughput-side of the efficiency claim open.

Target Audience

Researchers and engineers working on dynamic novel view synthesis, 4D Gaussian Splatting, and volumetric video who already understand 3DGS and deformation-field methods. It is also relevant to practitioners building streaming or random-access volumetric video systems, since the paper's design goals (bounded memory, on-demand loading, no external optical flow) are systems-oriented. Readers without a background in Gaussian Splatting or differentiable rendering will find the four-stage training design and the densification equations hard to follow without supplementary reading.

Authors’ abstract

Recent advances in 4D Gaussian Splatting (4DGS) have extended the high-speed rendering capability of 3D Gaussian Splatting (3DGS) into the temporal domain, enabling real-time rendering of dynamic scenes. However, one of the major remaining challenges lies in modeling long-range motion-contained dynamic videos, where a naive extension of existing methods leads to severe memory explosion, temporal flickering, and failure to handle appearing or disappearing occlusions over time. To address these challenges, we propose a novel 4DGS framework characterized by an Anchor Relay-based Bidirectional Blending (ARBB) mechanism, named MoRel, which enables temporally consistent and memory-efficient modeling of long-range dynamic scenes. Our method progressively constructs locally canonical anchor spaces at key-frame time index and models inter-frame deformations at the anchor level, enhancing temporal coherence. By learning bidirectional deformations between KfA and adaptively blending them through learnable opacity control, our approach mitigates temporal discontinuities and flickering artifacts. We further introduce a Feature-variance-guided Hierarchical Densification (FHD) scheme that effectively densifies KfA's while keeping rendering quality, based on an assigned level of feature-variance. To effectively evaluate our model's capability to handle real-world long-range 4D motion, we newly compose long-range 4D motion-contained dataset, called SelfCap$_{\text{LR}}$. It has larger average dynamic motion magnitude, captured at spatially wider spaces, compared to previous dynamic video datasets. Overall, our MoRel achieves temporally coherent and flicker-free long-range 4D reconstruction while maintaining bounded memory usage, demonstrating both scalability and efficiency in dynamic Gaussian-based representations.