HaNDF: Object-Conditioned Geometric Neural Hand Distance Fields

AI summary

Overview

Research area: Computer Vision — 3D hand–object interaction, generative priors, and Riemannian/geometric deep learning.

Technical level: Advanced. The paper assumes familiarity with MANO hand models, neural implicit/distance fields, transformer architectures, and Riemannian geometry on the unit-quaternion manifold (S³).

Scope (one sentence): The paper introduces HaNDF, an object-conditioned neural distance field defined on the product manifold of 15 MANO finger-joint quaternions, whose zero level set represents the plausible hand grasps for a given object and whose gradient supports projection, denoising, completion, generation, and object-latent inversion.

What This Paper Is About

Recovering or generating realistic hand poses is hard because a valid grasp depends on the object being held, and hand articulations do not live in flat Euclidean space — small pose changes can create interpenetration or break contact. Existing priors are either geometric but unconditional (PoseNDF, NRDF, which model bodies or hands without any object) or object-aware but built in Euclidean latent spaces with no meaningful distance to the valid-pose set.

HaNDF closes that gap by learning a scalar field that predicts the geodesic distance from any hand pose to the nearest plausible pose given a specific object, so that the zero level set of the field is an object-specific grasp manifold that can be optimized against with Riemannian gradient descent.

Key Contributions

1. An object-conditioned Riemannian neural distance field for hands. HaNDF defines a family of object-dependent plausibility manifolds as the zero level set of a conditional field f_Φ(θ | z), where θ is a set of K = 15 local joint quaternions on the power manifold ℍ₁^K and z is an object shape code.

2. A training procedure and architecture for object-specific distance fields. The model uses per-object latent fitting (a pretrained DeepSDF auto-decoder with latent dimension c_o = 64), nearest-pose supervision on paired hand–object data, and a transformer with L = 4 self-attention layers that jointly processes hand-joint tokens, an object token, and a learnable [DIST] aggregator token.

3. Object-conditioned projection, generation, and latent inversion. Building on NRDF's RDF-Grad, the paper derives an adaptive conditioned Riemannian descent step (HaNDFGrad) for projection and denoising, a sampling scheme from an f-induced Boltzmann density p(θ|z) ∝ exp(−β f_Φ(θ|z)²), and an inversion procedure that retrieves and refines compatible object conditions by optimizing or retrieving the object latent.

4. A plug-and-play prior validated across three benchmarks. HaNDF is inserted into existing reconstruction pipelines (EasyHOI, AlignSDF) without retraining task-specific models and evaluated on HOI4D, OakInk, and DexYCB for generation, denoising, completion, and in-the-wild recovery.

Main Findings

• Grasp generation: On HOI4D, HaNDF achieves the best dNN of 4.77° and a FID of 0.54, dominating both unconditional priors (H-NRDF, H-PoseNDF) and conditional generative baselines in the FID–dNN plane. H-NRDF also reports a low FID (0.77) but a much higher dNN, indicating limited per-sample realism. Against LatentHOI, HaNDF reports substantially higher diversity (APD 12.73° vs. 6.01°) while also improving FID and dNN.

• Denoising robustness: On HOI4D with n = 3000 noisy samples per noise level and σ ∈ [0.2, 1.5] rad, HaNDF gives the best Δ_q and m2m at every noise level, with margins that widen as σ grows. Its dNN stays nearly constant across the full range (4.54° → 5.20°), and it dominates on dNN from σ ≥ 0.8 onward. At the two smallest σ, H-PoseNDF sits slightly closer to the manifold but with much higher Δ_q and m2m — a mode-collapse signature the authors identify.

• Hand completion: On HOI4D with n = 3000 samples per task, HaNDF is best on Δ_q, dNN, and FID in both masking protocols. Missing Fingers: Δ_q 12.57°, dNN 5.44°, APD 20.79°, FID 0.0596 (vs. H-NRDF 16.60° / 8.01° / 24.48° / 0.1558; H-PoseNDF 20.80° / 16.40° / 32.57° / 1.0318; HMP 24.37° / 18.77° / 36.86° / 0.7407). Missing Finger Parts: Δ_q 6.60°, dNN 3.57°, APD 12.23°, FID 0.0106 (vs. H-NRDF 8.47° / 4.52° / 12.17° / 0.0453). The paper reports this as cutting Δ_q by roughly 25% and FID by 3–4× over the strongest baseline.

• In-the-wild hand recovery improves existing pipelines: With ground-truth object geometry given to all methods, adding HaNDF improves both baselines on all three datasets. On HOI4D, EasyHOI goes from MPVPE 16.64 / MPJPE 17.06 / I.V. 13.28 to 13.98 / 14.33 / 11.40, and AlignSDF from 10.40 / 10.60 / 5.39 to 10.22 / 10.43 / 4.39. On OakInk, AlignSDF improves from 10.34 / 10.50 / 0.82 to 9.98 / 10.17 / 0.87 (intersection volume slightly higher), and on DexYCB from 16.82 / 17.19 / 0.99 to 16.59 / 16.97 / 0.98. IHoI is reported at 19.42 / 20.12 / 41.44 on HOI4D, 14.75 / 15.17 / 1.66 on OakInk, and 29.85 / 30.73 / 1.26 on DexYCB.

• Object refinement closes the loop with hand refinement: Starting from AlignSDF on HOI4D, applying HaNDF as a denoiser reduces Δ_q from 14.77 to 9.60, MPJPE from 12.08 to 11.74, and MPVPE from 11.90 to 11.52. Retrieving the nearest neighbor in the DeepSDF latent space (with PCA, K = 1) improves object reconstruction (obj-F@2% from 0.1187 to 0.1568) and further boosts hand quality (MPJPE 11.74 → 11.50, MPVPE 11.52 → 11.30). Additional object recovery with K = 8 gives better hand numbers (9.57 / 11.45 / 11.25) but slightly worse object metrics (obj-CD 0.1083, obj-F@1% 0.0387, obj-F@2% 0.1360).

• Both geometry and conditioning matter, and they interact: In the HOI4D ablation, a Transformer with object conditioning and a Riemannian field reaches Δ_q 5.52°, dNN 4.96°, m2m 6.58 mm. Removing object conditioning (Transformer, Riemannian) degrades to 6.10° / 5.75° / 7.40 mm; using a Euclidean field with object conditioning degrades to 10.27° / 8.09° / 9.92 mm. An MLP with object conditioning and a Riemannian field is comparable (5.98° / 4.57° / 7.14 mm), and the authors note Euclidean geometry loses more when combined with the transformer backbone.

• The object condition is genuinely used: At noise level σ = 0.8 rad on HOI4D, the noisy input scores 11.74 Δ_q / 12.43 dNN / 11.13 m2m; HaNDF with a wrong object condition reaches 5.80 / 5.25 / 7.05; HaNDF without the object token reaches 6.10 / 5.75 / 7.40; full HaNDF reaches 5.52 / 4.96 / 6.58. (The provided text is truncated at the description of this ablation.)

Methodology in Plain English

The core idea is to learn a single number for any (hand pose, object) pair: how far that hand pose is from being a valid grasp of that object. That number is zero exactly when the pose is a plausible grasp, so the set of all valid grasps is the "zero level set" of the learned function.

Representing the hand. The hand follows the MANO model, parameterized by shape, translation, and pose. The authors drop the global wrist rotation and keep only the K = 15 local joint rotations, each expressed as a unit quaternion. Quaternions live on a sphere (S³), so the full hand lives on a product of 15 spheres. Distances and updates therefore have to respect that curved geometry: the paper uses the exponential/logarithmic maps and geodesic distance between quaternions, and applies the spherical Riemannian gradient (the Euclidean gradient projected onto the tangent space) rather than plain Euclidean steps.

Representing the object. Each object is compressed into a short latent vector by a pretrained DeepSDF auto-decoder, which regresses signed distance from a latent code and query point. The authors choose a latent rather than raw geometry for compactness, resolution invariance, and compatibility with settings where the object is only partially observed. To recover an object at test time, they invert the auto-decoder multiple times from different initializations, retrieve the nearest training latents, and use PCA to shrink the search.

The network. Hand joint quaternions become hand tokens via a linear projection; the object latent becomes a single object token; a learnable [DIST] token is prepended as an aggregator. After positional embeddings, four self-attention layers let every joint token attend to the object token and vice versa. The final [DIST] representation is mapped to a nonnegative scalar with a linear head and a softplus.

Training. The field is trained with a regression loss on paired hand–object data with synthetic noise injected into poses. Because inference spends most iterations in a narrow band near the manifold, the training distribution emphasizes near-manifold (hard-negative) samples using the exponential noise sampling schedule inherited from NRDF. Objects are sampled across epochs in proportion to the number of plausible grasps each object provides in the dataset.

Using the field. To clean or complete a pose, the method minimizes the predicted distance with an adaptive Riemannian descent step whose length is proportional to the predicted distance itself, terminating when the distance falls below a threshold δ or a maximum number of iterations is reached. Every update stays on the quaternion manifold by construction. For completion, only masked joints are updated while observed joints are frozen. To generate novel grasps, it draws random quaternions (uniform on the sphere, or normalized Gaussians), projects them onto the field, and decodes through MANO; different initializations can land in different basins, which produces diversity. In downstream reconstruction, the prior appears as an additive penalty alongside a task-specific data term, optimized with Riemannian Adam over the hand and ordinary Euclidean optimization over the object latent.

Why This Matters

Impact on research. The paper reframes object-conditioned hand synthesis as learning a distance to a conditioned manifold rather than as sampling from a learned distribution. That gives a differentiable, geodesically meaningful objective that supports optimization, projection, and inversion within one trained model, and it extends the NDF line of work (PoseNDF, NRDF, NRMF) from unconditional body priors to object-conditioned hands.

Real-world applications:

• Robotics: generating and refining physically plausible grasps for a known tool or object before executing a manipulation.
• AR/VR: fitting a user's hand to a tracked virtual or real object without visible markers, and repairing hand poses when fingers are occluded.
• 3D content creation and animation: producing diverse grasp poses for the same object, and completing partially authored hand animations.
• Perception pipelines: acting as a drop-in post-process that improves hand reconstruction from single images, particularly under hand–object occlusion.

Industry relevance. Because HaNDF is a prior rather than a full pipeline, it can be bolted onto existing hand–object reconstruction systems — the paper demonstrates exactly this by plugging it into EasyHOI and AlignSDF without retraining those models — which lowers the cost of adopting it in production stacks for teleoperation, digital humans, and embodied AI.

Future Directions

• Beyond a single global object code. The object condition is one latent vector for the whole shape; extending to spatially local or part-level conditioning could sharpen grasps for large or articulated objects, and the paper notes compatibility with alternative encoders as an open design axis.
• Extending to hand motion. The current field models static configurations; the related NRMF work extends NRDF to temporal motion, which suggests a natural next step of object-conditioned hand motion distance fields.
• Better joint object–hand inversion. The results show that better object reconstruction translates into better hand reconstruction, but recovery with K = 8 optimizations improved hands while slightly worsening object metrics — how to jointly and stably optimize both modalities remains unresolved.
• Scalability and generalization to unseen objects. The training depends on per-object latent fitting and object-level splits (30 unseen HOI4D objects, 14 unseen OakInk objects); how well the prior transfers to categories or object geometries far outside the training distribution is an open question the paper only partially probes.

Target Audience

Researchers and practitioners in 3D vision, graphics, and robotics who work on hand–object interaction, generative pose priors, or neural implicit representations. It is most useful to readers already comfortable with MANO, distance fields, and manifold optimization; those new to the area should treat the background section and appendix (on SO(3), the quaternion manifold, and the group exponential/log) as prerequisites. Engineers integrating a hand prior into an existing reconstruction or grasp-synthesis pipeline will find the plug-and-play results and the ablation on object conditioning the most immediately actionable parts.

Authors’ abstract

We propose, HaNDF, object-conditional neural distance fields as data-driven priors for modeling the plausible hand-object configurations, in the product manifold of hand articulations. While recent unconditional generative priors showed success in modeling human pose, hands are versatile articulated bodies often found in interaction with other objects. This makes it crucial to develop a generative prior which can model the plausible hand poses conditioned on the object under…