Learning Compact Latent Space for Representing Neural Signed Distance Functions with High-fidelity Geometry Details

AI summary

Overview

• Research area: 3D computer vision and neural implicit shape representation, specifically neural signed distance functions (SDFs) and neural fields.
• Technical level: Advanced (assumes familiarity with SDFs, marching cubes, latent codes, and volume/hash grids).
• Scope: The paper proposes a dual-branch method — one generalization-based network and one overfitting-based network sharing a latent code and a spatial feature grid — that learns a compact shared latent space for representing multiple SDFs while recovering high-fidelity geometry details, evaluated on Stanford models, ShapeNetCore.v2, and D-FAUST.

What This Paper Is About

Neural SDFs can represent a single shape well, but they struggle when many SDFs must be analyzed together: the latent space encodes limited information and high-frequency geometry details are lost. The goal is to learn a common, compact latent space in which many SDFs can be represented at once, while preserving fine surface geometry such as sharp edges, folds, and small holes.

Key Contributions

1. A method for representing multiple neural SDFs with high-fidelity geometry details by learning a compact latent space.
2. A dual-branch architecture that combines a generalization-based network (used far from the surface) with an overfitting-based volume-grid network (used near the surface), both sharing the same latent code for a shape.
3. A novel sampling strategy with a balance constraint: dense query sampling in a bandwidth around each shape's surface and sparse sampling outside it, which improves training efficiency and removes artifacts caused by other SDFs sharing the grid.
4. State-of-the-art results on widely used benchmarks, reported in terms of both representative ability and compactness.

Main Findings

• Single complex objects (Stanford models): The method samples 500K points from reconstructed surfaces for evaluation, with reconstruction via marching cubes at a uniform resolution of 512. It reaches CD 6.56e-05, F-Score 0.983, precision 0.969, and recall 0.998, versus ACORN 6.76e-05 / 0.982 / 0.967 / 0.998, NGLOD 6.77e-05 / 0.980 / 0.968 / 0.994, Instant-NGP 7.37e-05 / 0.976 / 0.962 / 0.990, Mosaic-SDF (M-SDF) 1.30e-03 / 0.902 / 0.846 / 0.972, and HyperDiffusion 1.20e-04 / 0.835 / 0.847 / 0.823.
• Per-model Stanford results (Table 6): Our method is reported as best across all metrics on all six models, for example Dragon CD 2.28e-6 and Armadillo CD 9.14e-6, with the Thai Statue being the hardest model for every method (our CD 3.59e-4).
• Multiple 3D objects (ShapeNetCore.v2): Average CD (×10^-4) is 0.607 for our method, compared with IF-NET 0.890, Instant-NGP 1.128, 3DILG 1.683, HyperDiffusion 2.788, DeepSDF 7.030, and M-SDF 12.316; per-category our results include Bench 0.463, Chair 0.898, Plane 0.223, Table 0.790, Lamp 0.517, and Sofa 0.751.
• Shape completion (D-FAUST): Average CD (×10^-4) is 0.124, against Instant-NGP 0.403, DeepSDF 3.58, and IF-NET 474.25; per-subject values are 0.134, 0.124, and 0.115.
• Shape interpolation: Our method, DeepSDF, and Instant-NGP all use 256-dimensional latent shape codes, while HyperDiffusion uses a 3-layer 128-dimensional MLP. Our method produces more pronounced shape changes during interpolation; HyperDiffusion produces unrealistic geometries or structural artifacts when interpolating directly in its latent space, and is described as less compact because it represents a shape with the parameters of a neural network rather than a one-column latent vector.
• Branch ablation: On 100 randomly selected chair models, the figure reports CD (×10^-4) of 4.01 for the dual model and 4.05 for the generalization-only branch, while the overfitting-only branch is far worse at 11.6, with the same figure reporting 4.29 and 14.6 for the bandwidth study over n = 1, 3, and 6. The generalization branch recovers low-frequency geometry; the overfitting branch recovers sharp surfaces but shows artifacts from imbalanced sampling outside the bandwidth; fusing the two removes both problems.
• Bandwidth sensitivity: With trained networks, bandwidths of 1, 3, and 6 are tested in signed distance fusion; artifacts inherited from the overfitting branch appear when the bandwidth is too large.
• Latent code dimension: Reconstruction CD (×10^-4) improves as the dimension grows — 0.443 (d = 8), 0.433 (d = 16), 0.410 (d = 32), 0.397 (d = 64), and 0.409 (d = 128) — so performance improves until d = 64 and is maintained afterwards. This experiment uses only 100 shapes, and the paper notes higher dimensions may be needed for larger datasets.
• Sampling strategy: Compared with uniform grid-vertex sampling, our sampling reaches CD 0.401 with roughly 220^3 samples, better than 256^3 (0.417), 128^3 (0.890), 64^3 (13.100), and 32^3 (35.900) uniform sampling. At resolution 32 the sampling density is insufficient to supervise the 128^3 spatial feature grid, producing incomplete surfaces.
• Instant-NGP in the multi-object setting: Despite strong single-object performance, Instant-NGP degrades in multi-object reconstruction even when trained with the proposed sampled queries, which the paper attributes to hash collisions and the finite resolution of the hash table undermining feature fidelity.

Methodology in Plain English

The method splits the job of representing a signed distance field between two networks that share one latent code per shape:

1. Generalization branch: a network that takes a positionally encoded 3D point plus the shape's latent code and predicts a signed distance. Because it is trained as a data-driven prior over many shapes, it produces sensible distances even where training queries are sparse — that is, far from the surface — and it suppresses artifacts caused by other shapes.
2. Overfitting branch: a network that also takes a shared spatial feature grid. A query point is trilinearly interpolated to get a local cube feature, and the network maps the point, that feature, and the shape code to a signed distance. Because grid features are not constrained to be consistent with neighbors, this branch can capture high-frequency detail.
3. Fusion at inference: The generalization branch is first used to reconstruct a coarse shape with marching cubes. The occupied voxels are expanded by n layers on both sides to define a surface bandwidth, and marching cubes is rerun so that points inside the bandwidth use the overfitting branch and all other points use the generalization branch.

The sampling strategy addresses an imbalance problem: if some shapes contribute many more queries to a voxel than others, that voxel's feature becomes biased toward those shapes. Balancing queries in every voxel of a cubic grid would be prohibitively expensive, so the method only enforces dense coverage in the bandwidth near each surface (sampled at 512^3) and uses sparse uniform sampling elsewhere (128^3), letting the generalization branch handle the far field. Training optimizes both branches jointly with a composite loss: MSE between predicted and ground-truth signed distances for each branch, plus regularization on the shape code and the grid codes. The grid code uses a 128^3 grid with a 128-dimensional feature per grid and a learning rate of 1.0 × 10^-1, while the shape code (256-dimensional) uses 1.0 × 10^-3; both networks use 256-dim codes and eight hidden layers of 512 units, the learning rate decays by 0.5 every 1000 epochs, and training runs for 4000 epochs. Meshes were scaled into [-0.9, 0.9]^3 and made watertight with Manifold before sampling.

Why This Matters

• Research impact: The work shows that generalization-based and overfitting-based SDF learning are complementary rather than competing, and that a shared latent space can represent many high-detail SDFs without losing the fine geometry that volume grids capture and MLP-based priors tend to smooth away.
• Real-world applications (areas the paper's framing and experiments connect to):
  • 3D perception in computer vision and robotics, which the paper names as a primary use of SDFs.
  • Reconstructing detailed 3D assets from scans, as demonstrated on Stanford models such as Lucy, Happy Buddha, Armadillo, Dragon, Asian Dragon, and Thai Statue.
  • Shape completion from partial observations, demonstrated on human bodies from D-FAUST.
  • Shape interpolation and latent-space manipulation of 3D models, relevant to generative 3D modeling.
• Industry relevance: Compact latent codes matter for storing and transmitting large libraries of 3D shapes, for asset pipelines in gaming, simulation, and content creation, and for any system that needs to query many shapes through one shared model instead of overfitting a separate grid per shape. The authors released code at https://github.com/eoozbq/Compact-SDF, and the work was supported by the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62206106).

Future Directions

• Removing the supervision requirement: The paper notes as a limitation that the method needs large-scale signed distance ground truth and therefore cannot infer geometry without signed distance supervision (the unsupervised setting).
• Robust hyperparameter and bandwidth selection: Different bandwidth layers lead to residual noise or discontinuity in the shape, so the choice currently affects reconstruction quality.
• Scaling latent dimensionality to larger datasets: The dimension study used only 100 shapes and suggests higher dimensions may be needed for bigger datasets; how the compactness/quality trade-off behaves at scale is open.
• Overcoming grid-capacity limits: The paper diagnoses hash collisions and finite hash-table resolution as the cause of Instant-NGP's degradation in multi-object reconstruction, raising the question of how to retain fine detail in a shared latent space without such collisions.

Target Audience

Researchers and graduate students working on neural implicit representations, neural fields, and 3D reconstruction or generation; practitioners who need compact multi-shape latent spaces with high geometric fidelity; and readers already familiar with SDFs, marching cubes, and grid- or hash-based feature encodings who want to understand how to combine data-driven priors with per-shape overfitting.

Authors’ abstract

Neural signed distance functions (SDFs) have been a vital representation to represent 3D shapes or scenes with neural networks. An SDF is an implicit function that can query signed distances at specific coordinates for recovering a 3D surface. Although implicit functions work well on a single shape or scene, they pose obstacles when analyzing multiple SDFs with high-fidelity geometry details, due to the limited information encoded in the latent space for SDFs and the loss of geometry details. To overcome these obstacles, we introduce a method to represent multiple SDFs in a common space, aiming to recover more high-fidelity geometry details with more compact latent representations. Our key idea is to take full advantage of the benefits of generalization-based and overfitting-based learning strategies, which manage to preserve high-fidelity geometry details with compact latent codes. Based on this framework, we also introduce a novel sampling strategy to sample training queries. The sampling can improve the training efficiency and eliminate artifacts caused by the influence of other SDFs. We report numerical and visual evaluations on widely used benchmarks to validate our designs and show advantages over the latest methods in terms of the representative ability and compactness.