FUSE: A Flow-based Mapping Between Shapes

AI summary

Overview

Research area: Computer Vision / 3D geometry processing — specifically shape matching and neural representations of 3D geometry.

Technical level: Advanced. The paper assumes familiarity with flow matching, conditional flow matching, optimal transport, functional maps, geodesic distances, and signed distance fields. The core idea is intuitive, but the machinery around it is dense.

Scope (one sentence): The paper proposes FUSE, a flow-matching-based neural representation that expresses the map between two 3D shapes as the composition of two invertible flows through a shared Gaussian anchor, enabling accurate, coverage-rich correspondences across meshes, point clouds, SDFs, and volumetric data without per-pair optimization or large-scale training.

Authors and affiliations: Lorenzo Olearo, Giulio Viganò, Daniele Baieri, and Simone Melzi (University of Milano-Bicocca, Milan, Italy); Filippo Maggioli (Pegaso University, Naples, Italy); Daniele Baieri also with the University of Bonn (Germany) and the Lamarr Institute (Germany). Code is released at https://github.com/LorenzoOlearo/FUSE-Flow-based-Mapping-Between-Shapes.

What This Paper Is About

Estimating point-to-point correspondences between two 3D shapes with a common semantic meaning is a long-standing problem in geometry processing, and most existing methods are built around triangle meshes or depend on learned priors and heavy per-pair optimization. FUSE reframes the problem: instead of directly searching for a map between surfaces, it represents each shape as a probability distribution of per-point feature embeddings and learns an invertible flow that transports a fixed Gaussian anchor distribution onto that shape's embedding distribution. The map between a source and a target shape is then simply the composition of the source's inverse flow with the target's forward flow, which requires no additional training once both flows exist.

Key Contributions

1. A flow-matching representation for 3D shape maps. FUSE is presented as the first flow-matching-based representation for maps between 3D shapes that naturally supports correspondence between shapes expressed in different modalities, including meshes, point clouds, and SDFs.
2. Competitive or superior matching accuracy, with a landmark-based prior. The authors quantitatively show that FUSE achieves competitive or superior matching accuracy over state-of-the-art methods, particularly in landmark-based shape matching, using a Geodesic Feature Embedding built from a small number of landmarks.
3. Versatility as a backbone for downstream tasks. FUSE is demonstrated as a building block for shape parametrization (UV mapping), human scan fitting and skinning, and volumetric matching, in addition to pure correspondence.
4. A new SDF-based benchmark. Since no dataset existed to quantitatively evaluate matching between SDFs, the authors introduce a benchmark derived from FAUST, using neural implicit reconstructions of FAUST shapes with mesh vertices projected onto the SDF surfaces for evaluation.

Main Findings

• Lowest error under a lightweight prior. With landmark-based GFE and no test-time refinement, FUSE reaches Euclidean error of 0.028 on FAUST and 0.059 on SMAL, compared to 0.089 / 0.189 for the vision model Diff3F and 0.043 / 0.063 for the training-data-driven ULRSSM. This is the paper's headline comparison of prior assumptions.
• Accuracy versus smoothness trade-off. On FAUST, FUSE attains Euclidean error 0.0289, geodesic 0.0274, Dirichlet energy 0.0066, and coverage 0.5320. Functional-maps style baselines produce smoother maps (e.g., FMap ZO at Dirichlet 0.0009) but lower accuracy in Euclidean terms (0.0207 on FAUST, though with much higher coverage of 0.6969). The paper attributes FUSE's lower smoothness to the flow inversion from the surface back to the anchor, which is a non-smooth operation.
• Refinement stacks well on top of FUSE. Initializing ZoomOut and NAM with FUSE improves results — FUSE NAM reaches Euclidean 0.0179 / geodesic 0.0168 / coverage 0.7027 on FAUST, and on SHREC20 (strongly non-isometric animals) FUSE NAM reaches Euclidean 0.0658 and coverage 0.5342, outperforming the corresponding FMaps initializations.
• Data-driven methods do not dominate everywhere. ULRSSM is the best performer on FAUST (0.0092 Euclidean) and SMAL (0.0370), but degrades substantially on SHREC20 (0.3286 Euclidean, 0.3496 geodesic) and is reported as N/A on the Kinect point-cloud benchmark. It requires extensive training on a large dataset and is restricted to meshes.
• Point clouds are the weak spot. Kinect results show FUSE at Euclidean 0.0969 / geodesic 0.0914 / coverage 0.4931, still ahead of unrefined geodesic methods (KNN at 0.1119 / 0.1095; OT at 0.1327 / 0.1273; FMap at 0.0810 / 0.0779 Euclidean — note FMap is lower here) but behind the extrinsic NDP, which bypasses geodesics entirely (Euclidean 0.0457, geodesic 0.0438, coverage 0.4190) at the cost of failing under strong non-rigid deformations such as FAUST. FUSE's difficulty stems from unreliable geodesic distances and incomplete surface coverage on unstructured data.
• Flow matching beats diffusion for anchoring. Comparing FUSE against a DDIM-based variant on FAUST: mean KL divergence is 0.0375 for FUSE versus 0.3519 for FUSE DDIM (and 0.2528 for raw features); mean JS divergence is 0.0829 versus 0.2128 (0.1928 raw). Matching error also improves — Euclidean 0.028 and geodesic 0.027 for FUSE versus 0.037 and 0.035 for FUSE DDIM. The authors attribute this to flow matching's explicit, deterministic relationship between the Gaussian and the data distribution, whereas DDIM only asymptotically approaches the Gaussian.
• Smoothness of the embedding is a critical factor ("curse of frequency"). Geodesic Feature Embeddings have very low Dirichlet energy and benefit greatly from the flow mapping, while WKS and learned embeddings (ULRSSM) have substantially higher Dirichlet energy and perform poorly under FUSE. Smoothing WKS via projection onto a small Laplace-Beltrami eigenbasis (WKS*, WKS-L*) consistently improves FUSE's accuracy over raw features.
• Robust to landmark count and noise. Increasing the number of landmarks from 3 to 30 yields progressively better results that consistently outperform KNN, and perturbing landmark positions by a random fraction of the shape diameter degrades performance gracefully. The paper notes FUSE's advantage over other methods tends to increase as isometry decreases or landmark placement becomes noisy.
• SDF matching results. On the new SDF-based FAUST benchmark (excluding shapes whose implicit reconstruction altered topology), FUSE achieves Euclidean 0.0375, geodesic 0.0385, Dirichlet 0.0069, coverage 0.4653, versus KNN at 0.0562 / 0.0569, NDP at 0.0525 / 0.0643, OT at 0.0921 / 0.0881, and FUSE ANCHOR at 0.0472 / 0.0469. ZO and NAM refinements cannot be applied on SDFs, where the paper says FUSE's added value is most significant.
• Inter-representation matching works, with caveats. Using FAUST-derived meshes, point clouds (vertices extracted from the meshes), and SDFs, FUSE achieves results comparable to mesh-to-mesh matching when at least one shape is a mesh or SDF, but struggles when only point clouds are involved. Geodesics are computed per representation: heat geodesics on point clouds, Dijkstra on mesh or voxel grid.
• Volumetric matching is competitive. On tetrahedral meshes, average accuracy across four shapes (Octopus, Cat, Dino_skel, Airplane) is 0.0501 for FUSE versus 0.0578 for Volumetric FMaps, with the improvement most pronounced on the most non-isometric shape (Airplane: 0.0942 versus 0.1273).
• Not strictly bijective. Although the flows are invertible by design, the final KNN retrieval step assigns each source point its nearest target independently, so the resulting matching is not strictly bijective — a standard property of point-based correspondence methods.

Methodology in Plain English

The pipeline has four moving parts:

1. Sample points and compute features. FUSE only requires two things of a shape: that you can sample points from its surface, and that you can attach a task-specific feature vector to each sample. Shapes are treated as continuous 2-manifolds embedded in ℝ³, regardless of whether they arrive as meshes, point clouds, SDFs, or volumes.

2. Build a geodesic embedding. The default choice, Geodesic Feature Embedding (GFE), represents each surface point by its vector of geodesic distances to a small set of landmarks — in this work, 5 landmarks for human datasets and 6 for animals, manually selected on one shape and propagated to the rest of the dataset via ground-truth correspondences. Landmarks are chosen because they break intrinsic symmetries that cannot be resolved from geometry alone. Alternative embeddings (raw x, y, z coordinates, WKS, learned ULRSSM features) are tested for comparison.

3. Train one flow per shape. For each shape independently, a neural network learns a velocity field via Conditional Flow Matching that transports a fixed shared Gaussian anchor, 𝒩(0,1)^d, onto that shape's embedding distribution. The learned flow is invertible and diffeomorphic: integrating forward takes you from the Gaussian to the shape, and integrating backward takes you from the shape back to the Gaussian.

4. Compose the flows to get a map. The map from shape 1 to shape 2 is literally Φ¹² = ψ² ∘ (ψ¹)⁻¹ — push shape 1's embedding points back through its inverse flow into the shared Gaussian, then forward through shape 2's flow into shape 2's embedding space. Discrete correspondences are then recovered with a nearest-neighbor search in the shared embedding space.

The theoretical justification: work by Lipman et al. (Theorem 2, cited as [27]) shows that marginalizing the conditional velocity fields over target samples recovers a unique marginal field determined only by the source and target distributions, so the induced flow does not depend on the random couplings used during training. If two shapes share the same embedding distribution, their flows coincide; if the distributions are similar, the composition aligns them. This is also where FUSE can fail — if the shapes differ by a transformation the embedding is not invariant to, the distributions become dissimilar and unalignable.

For volume matching, FUSE extends naturally because volumetric shapes are 3-manifolds with well-defined geodesic distances. For UV mapping, 3D coordinates are sufficient as the embedding since no semantic consistency is required; because FUSE maps can lack smoothness, the transferred UVs are used to supervise an MLP that predicts continuous UV coordinates.

Why This Matters

The paper's central claim is a representational one: correspondences can be expressed probabilistically, through distribution alignment, rather than through linear operators on spectral bases or through explicit 3D deformation fields. That reframing matters because it decouples matching from the triangle-mesh assumption that dominates the field. FUSE requires no learned prior, no per-pair optimization inside the core pipeline, and no representation-specific machinery — the same construction works on the same anchors for a mesh, a point cloud, or a neural SDF.

Real-world applications the paper demonstrates or points to:

• Character and creature animation. The human scan fitting and skinning demo fits SMPL-X body model parameters (β, θ) to an input point-cloud scan using FUSE correspondences, then re-poses the scan by applying point-wise offsets from the neutral pose, preserving the scan's individual defects on the right leg, back, and chest.
• Texture mapping and UV parametrization. Training flows from a Gaussian to a genus-zero surface and to a reference sphere yields a bijection that transfers the sphere's UV parametrization to the shape.
• Pose transfer and shape interpolation. The introduction cites these as classic downstream tasks unlocked by accurate correspondences.
• Volumetric and simulation-oriented geometry. The volume-matching experiments on tetrahedral meshes suggest the framework applies where 3D solid representations are used, mirroring the surface results.

Industry relevance: the modality-agnostic property speaks to production pipelines where assets exist in mixed formats — scanned point clouds, authored meshes, and neural fields — and where a matching method has to bridge them rather than force a conversion first. The absence of large-scale training also lowers the barrier to adopting FUSE for one-off pairs or proprietary shapes, which is relevant for studios and medical/industrial geometry workflows where no large annotated corpus exists.

Future Directions

1. Extending to partial matching. The authors explicitly state that FUSE is unsuitable for partial matching in its current form because it relies on consistent geodesic distances across the full shape.
2. Enforcing full bijectivity. The final KNN step breaks bijection. Since both the GFE and the flow are defined continuously over the surface, the authors argue the construction already supports a continuous, fully bijective map and leave realizing it to future work.
3. Integrating learned smooth features. The finding that smoothing WKS features (WKS*, WKS-L*) improves FUSE suggests combining smoothly learned feature extractors with flow-based alignment as a promising direction.
4. Improving performance on point clouds. Because reliable geodesics and adequate surface coverage are hard to guarantee on unstructured data, making the flow mechanism robust to poor geodesic estimates remains open.

Target Audience

Researchers and graduate students in geometry processing, computer vision, and geometric deep learning who are working on shape correspondence, neural shape representations, or generative/probabilistic modeling of 3D geometry. It is also relevant to practitioners who need correspondences across mixed asset types — meshes, scans, implicit surfaces, volumes — and to readers interested in how flow matching and diffusion models compare as tools for representing geometry itself, not just generating it. A reader should arrive with prior exposure to flow matching, functional maps, and geodesic distances to get full value from the results tables.

Authors’ abstract

We introduce a novel neural representation for maps between 3D shapes based on flow-matching models, which is computationally efficient and supports cross-representation shape matching without large-scale training or data-driven procedures. 3D shapes are represented as the probability distribution induced by a continuous and invertible flow mapping from a fixed anchor distribution. Given a source and a target shape, the composition of the inverse flow (source to anchor) with the forward flow (anchor to target), we map points between the two surfaces. By encoding the shapes with a pointwise task-tailored embedding, this construction provides an invertible and modality-agnostic representation of maps between shapes across point clouds, meshes, signed distance fields (SDFs), and volumetric data. The resulting representation consistently achieves high coverage and accuracy across diverse benchmarks and challenging settings in shape matching. Beyond shape matching, our framework shows promising results in other tasks, including UV mapping and registration of raw point cloud scans of human bodies.