Neu-PiG: Neural Preconditioned Grids for Fast Dynamic Surface Reconstruction on Long Sequences

AI summary

Overview

Research area: Computer vision and computer graphics — specifically dynamic (4D) surface reconstruction, deformation modeling, and neural field representations from point cloud sequences.

Technical level: Intermediate. Readers should be comfortable with 3D geometry basics (meshes, normals, quaternions), gradient-based optimization, and standard neural network concepts; no prior knowledge of Sobolev methods is required.

Scope: The paper introduces a fast, category-agnostic optimization method that fits a single shared latent deformation field to an entire dynamic point cloud sequence, achieving state-of-the-art accuracy at roughly an order of magnitude lower runtime than prior training-free approaches.

What This Paper Is About

Reconstructing a moving 3D surface from a sequence of unstructured point clouds is difficult because the surface must stay geometrically accurate, temporally consistent, and drift-free across many frames. Existing training-free methods compute deformations frame by frame, which is slow and accumulates error over long sequences, while learned methods are fast but require category-specific training data (e.g., only human bodies or only faces). Neu-PiG aims to combine both strengths: a general optimization approach that encodes the deformations for all time steps into one shared multi-resolution latent grid, trained with preconditioning so it converges in seconds rather than minutes.

Key Contributions

1. A fast optimization-based deformation method for arbitrary subjects (humans, animals) that estimates temporally coherent deformations from sequential point clouds without pretraining or category-specific priors.

2. A preconditioned surface encoding parameterized by the positions and normal directions of a reference mesh, storing deformations across all time steps in a unified latent space.

3. A multi-scale latent grid representation with Sobolev-preconditioned gradient updates, enforcing spatial smoothness at global and local scales and enabling fast decoding through a lightweight MLP.

4. Strong empirical validation showing best-in-class accuracy on DFAUST, AMA, and DT4D, running at least 60x faster than prior training-free methods and scaling to long sequences (up to 120 frames) where baselines degrade or fail.

Main Findings

• Best accuracy across all benchmarks: Neu-PiG achieves the lowest Chamfer Distance and correspondence error and the highest normal consistency and F-score on DFAUST, AMA, and DT4D, outperforming both training-free methods (DynoSurf, PDG) and pretrained models (CaDeX, M2V).
• Large speed advantage: On DFAUST, the short schedule finishes in 8 seconds and the long schedule in 32 seconds, versus 7 minutes for PDG and 30 minutes for DynoSurf.
• Stable scaling to long sequences: On AMA sequences of 40 to 120 frames, Neu-PiG reconstructs complete sequences in under two minutes with stable correspondences, while PDG fails on 120 frames and DynoSurf degrades sharply.
• Every architectural component matters: Ablations show removing the normal-direction encoding, preconditioning, the isometry loss, or the multi-resolution hierarchy all hurt accuracy; multi-resolution hash encoding is more memory-efficient but sacrifices spatial smoothness.
• Fourier time encoding is best: Against polynomial, Gaussian Fourier, and learned time embeddings, the proposed frequency-based encoding gives the best balance of accuracy and temporal coherence.
• Confidence weighting design matters: The cumulative temporal weighting and the catch-up schedule for the confidence term both beat constant, linear, exponential, and interpolated alternatives.
• Modest returns from longer training: The 1000-epoch configuration improves metrics only marginally over the 250-epoch version, and convergence plots show high-quality results within seconds.

Methodology in Plain English

The method starts by picking one representative frame (a keyframe) from the sequence and building a reference mesh from that frame's point cloud using Poisson surface reconstruction. This mesh defines the topology for the whole sequence. The keyframe choice favors frames near the temporal midpoint with a large spatial extent.

Rather than predicting a fresh deformation for every frame independently, the method learns a single set of deformable features stored in voxel grids anchored to the reference surface. There are two grids: a position grid, which is coarse-to-fine multi-resolution (eight levels) and stores richer 30-dimensional features, and a smaller normal-direction grid (single 4x4x4 resolution) that stores 2-dimensional features. Each surface point looks up both grids, with trilinear interpolation averaging the levels together. The idea is that the position captures where a point is and the normal direction captures how its local surface faces, letting nearby regions with different orientations move differently.

For a given frame, the looked-up features are concatenated with a sinusoidal encoding of the timestamp, and a small three-layer MLP (512 hidden units) outputs a rotation quaternion plus a displacement vector. The rotation is made identity-preserving by offsetting the scalar quaternion component by one and normalizing, and the displacement is bounded with a tanh so points cannot fly away. Applying this transformation to every reference vertex gives the deformed surface for that frame.

Training uses two losses: a Chamfer distance between the deformed surface and the target point cloud (with a confidence weight that favors frames following well-reconstructed ones), and an isometry loss that penalizes changes in mesh edge lengths to keep the surface from stretching or tearing. The distinguishing trick is Sobolev preconditioning: instead of using raw gradients to update the latent grid features, the gradients are filtered through an inverse Laplacian-like operator. This low-pass filter couples neighboring cells so that updates stay spatially smooth and temporally stable, which is what prevents drift over long sequences.

Why This Matters

This work shows that a compact, preconditioned latent grid plus a tiny MLP can replace both slow per-frame optimization and heavy pretrained category models, making high-fidelity 4D reconstruction practical without any training data. It also demonstrates that Sobolev preconditioning, previously applied to raw deformation fields, transfers effectively to learned latent representations, which is a useful insight for other optimization-heavy neural field problems.

Real-world applications:

• Motion capture and animation: Fast reconstruction of actor or animal performances from markerless point cloud captures, feeding downstream rigging and retargeting pipelines.
• AR/VR and telepresence: Reconstructing and transmitting dynamic human geometry in real time for immersive communication and virtual avatars.
• Robotics and autonomous systems: Modeling how objects and bodies deform in the environment to support manipulation planning and human-robot interaction.
• Medical and biomechanical analysis: Tracking surface deformation of body parts or soft tissue over long capture sessions without requiring subject-specific templates.

Industry relevance: The method's speed (seconds rather than minutes) and lack of category-specific training make it attractive for production pipelines where capture throughput matters, and for content creators who need to handle diverse subjects (people, animals, generic objects) without maintaining separate specialized models. The public code release lowers the barrier to adoption.

Future Directions

• Handling incorrect or incomplete keyframes: The method inherits topology from a single keyframe mesh, so recovering from a bad initial surface (missing parts, wrong connectivity) remains unsolved.
• Extending the capacity bound on sequence length: The latent grids and MLP have finite representational capacity, and how far this scales for arbitrarily long captures is an open question.
• Improving robustness to large motions and occlusions: Correspondence is inferred only through Chamfer distance, so extreme motion or missing data still degrades results, and severe local shrinkage can cause face flipping.
• Unifying with learned priors: Combining the training-free latent grid with lightweight category priors could push accuracy further without losing generality, and the authors frame real-time 4D reconstruction as a natural next target.

Target Audience

Researchers and graduate students in computer vision, computer graphics, and 3D reconstruction who work on dynamic shape analysis, neural fields, or deformation modeling. It is also relevant to practitioners building motion capture, AR/VR, or robotics pipelines who need fast, template-free 4D surface tracking, and to anyone interested in how preconditioning techniques can stabilize optimization in high-dimensional latent spaces.

Authors’ abstract

Temporally consistent surface reconstruction of dynamic 3D objects from unstructured point cloud data remains challenging, especially for very long sequences. Existing methods either optimize deformations incrementally, risking drift and requiring long runtimes, or rely on complex learned models that demand category-specific training. We present Neu-PiG, a fast deformation optimization method based on a novel preconditioned latent-grid encoding that distributes spatial features parameterized on the position and normal direction of a keyframe surface. Our method encodes entire deformations across all time steps at various spatial scales into a multi-resolution latent grid, parameterized by the position and normal direction of a reference surface from a single keyframe. This latent representation is then augmented for time modulation and decoded into per-frame 6-DoF deformations via a lightweight multilayer perceptron (MLP). To achieve high-fidelity, drift-free surface reconstructions in seconds, we employ Sobolev preconditioning during gradient-based training of the latent space, completely avoiding the need for any explicit correspondences or further priors. Experiments across diverse human and animal datasets demonstrate that Neu-PiG outperforms state-the-art approaches, offering both superior accuracy and scalability to long sequences while running at least 60x faster than existing training-free methods and achieving inference speeds on the same order as heavy pretrained models.