InstantHand: Feed-Forward 3D Gaussian Splatting for Animatable Hand Avatar Reconstruction from a Single Image

AI summary

Overview

Research area: Computer vision and neural rendering — specifically feed-forward, animatable 3D hand avatar reconstruction using 3D Gaussian Splatting (3DGS) with MANO parametric hand models.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, MANO hand parameterization, transformer architectures, DINOv3 features, and standard novel-view synthesis metrics (PSNR, SSIM, LPIPS).

Scope: InstantHand reconstructs an animatable, photorealistic 3D Gaussian hand avatar from a single RGB image in one forward pass — no per-subject optimization — and renders it under novel poses and viewpoints.

What This Paper Is About

Building a personalized, animatable hand avatar normally requires multi-view capture rigs, monocular video, or slow per-subject optimization at test time, which makes the technology hard to scale. Hands are thin, highly articulated, and often self-occluded, so reconstructing them from one image is severely under-constrained. The authors propose InstantHand, a feed-forward network that takes a single image plus off-the-shelf MANO, camera, and mask estimates and directly predicts a canonical 3D Gaussian hand that can be posed and rendered from new angles.

Key Contributions

1. InstantHand framework: A feed-forward pipeline that produces an animatable canonical 3D Gaussian hand avatar from a single image, with no per-subject or test-time optimization, using off-the-shelf MANO, camera, and mask estimates as input.

2. Soft Alpha Band (SAB) regularizer: A task-specific loss that concentrates alpha supervision on an outer band (from dilation) and inner region (from erosion) of the silhouette, penalizing alpha leakage outside finger boundaries and insufficient alpha inside the silhouette. This prevents thin finger contours from thickening.

3. Reliability-gated Appearance Transfer: A module that refines only the Gaussian colors using a closed-form, contribution-aware reliability score per Gaussian (combining accumulated rendering contribution with foreground mask support within its projected footprint), preserving observed appearance such as skin tone and fine texture, while completing unobserved regions from a learned prior conditioned on a global source appearance identity token.

4. Surface-anchored constrained parameterization: Gaussians are placed as offsets from MANO-surface anchors (49,281 anchors from a 3-level subdivided mesh, k = 3) with a learned isotropic scale, while opacity and canonical rotation are fixed — removing unstable interactions among opacity, scale, and orientation.

Main Findings

• Matches per-subject-optimized baselines on InterHand2.6M: InstantHand reaches 26.21 dB PSNR, 0.866 SSIM, and 0.133 LPIPS, versus the strongest optimized baseline OHTA at 26.11 dB PSNR, 0.864 SSIM, 0.129 LPIPS — without any test-time fitting. InstantHand attains the second-best LPIPS.

• Competitive zero-shot on HanCo: Trained without HanCo subjects, it achieves 20.18 PSNR, 0.836 SSIM, 0.188 LPIPS, staying within 0.14 dB PSNR, 0.007 SSIM, and 0.002 LPIPS of the best fitted baselines at over 400× lower avatar-acquisition cost.

• Large efficiency advantage: Avatar construction takes 2.84 s and rendering runs at 161 FPS (160.71 FPS in the efficiency table) with 1.79 GB memory at 256×256. For comparison, HandAvatar requires 5579 s and runs at 0.32 FPS with 14.80 GB, and OHTA requires 3360 s at 0.26 FPS with 47.80 GB.

• Both appearance gates are necessary: Removing reliability gating (γ) drops PSNR to 23.18, and removing the learned appearance gate (g_i) drops it to 24.52, while removing appearance transfer entirely yields 26.01 dB — outperforming both single-gate variants. The authors note the two gates are not interchangeable despite appearing as a product in the composition equation.

• SAB regularizer is critical for thin structures: Removing it drops PSNR from 26.21 to 23.89, SSIM from 0.866 to 0.825, and LPIPS from 0.133 to 0.166.

• Head and body avatar recipes do not transfer to hands: Porting a head-avatar recipe (with FLAME swapped for MANO) under matched training loses 2.61 dB PSNR, drops boundary F-score from 0.698 to 0.299, and raises alpha leakage from 0.234 to 0.608, while interior coverage stays at 1.000 — localizing the failure to thin, articulated contours.

• Robust to estimated source cues: With fully estimated inputs (SAM 3 masks, WiLoR MANO and camera), PSNR falls by only 0.22 dB to 25.99, with SSIM and LPIPS essentially unchanged.

• Appearance module measurably improves source fidelity: Rendering each avatar in its source view with all non-color attributes fixed, the module improves source-view PSNR by 1.96 dB on InterHand2.6M and 3.41 dB on zero-shot HanCo.

• Degradation tracks finger articulation more than viewpoint: PSNR drops 2.69 dB (InterHand2.6M) and 1.93 dB (HanCo) between the mildest and strongest articulation thirds, while viewing-direction changes cost at most 0.97 dB.

• Temporal stability: Under continuous pose driving on zero-shot HanCo, frame-to-frame changes in renderings exceed ground truth by 4.7% on average and at most 8.0% per camera.

Methodology in Plain English

The pipeline has two stages.

Stage 1 — build a canonical hand. A frozen DINOv3 image encoder turns the source image into visual feature tokens. Separately, the system takes the MANO shape parameter and mean pose, subdivides the hand mesh three levels (giving 49,281 surface points), and embeds those points as query tokens. A conditional transformer with two sequential streams (one for geometry, one for appearance) cross-attends from these surface queries to the image features. Three small heads then decode each token: an offset head nudges the point away from its MANO anchor, a scale head predicts a single positive isotropic radius, and a color head predicts a base RGB value. Opacity and canonical rotation are held fixed. Training uses photometric and perceptual losses, a silhouette loss, geometry regularizers on offsets/smoothness/scale, and the Soft Alpha Band loss that watches the boundary bands of the silhouette.

Stage 2 — transfer the actual appearance from the photo. The Stage-1 predictor, including the base color head, is frozen. The canonical Gaussians are posed to the source hand pose and rendered from the source camera, which yields two things per Gaussian: its accumulated rendering contribution and its projected footprint. For each Gaussian, the method aggregates source RGB, rendered RGB, the residual between them, and local image features over that footprint. A closed-form reliability score combines rendering contribution with how much of the footprint lies inside the source foreground mask (with constants κ = 0.5, τ_m = 0.5, ζ = 95 fixed a priori and receiving no gradient). A summary "source identity" token captures global properties like skin tone and illumination bias. These feed a small network that predicts an observed color, a learned gate, and a displacement; displacements are pooled into a single source-level shift applied to all Gaussians. The final color is the base color plus the gate-weighted, reliability-weighted correction from local evidence, plus the global shift. Only colors change — geometry stays MANO-anchored, so novel poses are produced by the same MANO-driven deformation and renderer.

Why This Matters

Impact on research: The paper shows that avatar architectures designed for heads and bodies do not simply port to hands, and it isolates why: thin articulated contours suffer alpha leakage and boundary failure rather than interior failure. It offers a diagnostic vocabulary (boundary F-score, leakage, interior coverage) and a design pattern — surface-anchored constrained Gaussians plus explicit source-evidence transfer — for other thin or highly articulated structures.

Real-world applications:

• VR/AR and telepresence: a user's hands captured in a single photo become an animatable avatar for immersive interaction and dexterous manipulation.
• Gaming and social communication: fast, per-user hand avatar creation in 2.84 s at 161 FPS fits inside an interactive content pipeline rather than an offline one.
• Sign language and gesture-driven interfaces: animatable avatars that can be driven by new poses while retaining a specific person's appearance.
• Digital content creation and e-commerce: virtual try-on of rings, gloves, or watches on a reconstructed personalized hand.

Industry relevance: Eliminating test-time optimization and multi-view capture removes the main cost barrier to deploying hand avatars at consumer scale. The ~400× lower acquisition cost relative to fitted baselines, combined with 1.79 GB render memory, makes the approach plausible on commodity hardware.

Future Directions

• View- and pose-dependent appearance: predicted colors are independent of viewing direction and driving pose, so effects such as self-shadowing are not captured; the authors flag extensions toward view-dependent appearance modeling as an open limitation.
• Reducing articulation-dependent degradation: the 2.69 dB / 1.93 dB drops between mild and strong articulation suggest that heavily posed fingers remain the weak point, motivating better deformation-aware supervision.
• Richer input cues beyond MANO, camera, and mask: the 0.22 dB drop from fully estimated inputs suggests headroom for jointly improving upstream pose/shape estimators or folding them into the feed-forward model.
• Extensions to interacting hands and objects: the paper focuses on a single hand; two-hand interaction, hand-object contact, and the occlusions they create are natural but unreported territory.

Target Audience

Researchers and graduate students in computer vision, neural rendering, and digital humans who work on avatar reconstruction, 3D Gaussian Splatting, or parametric hand models. It is most useful to readers already comfortable with MANO and 3DGS who want to understand why feed-forward avatar recipes from heads and bodies fail on hands, and what design choices fix it. Practitioners building AR/VR or telepresence products will find the efficiency and robustness numbers directly relevant, though the method sections assume an advanced background.

Authors’ abstract

Hand avatar reconstruction typically relies on multi-view capture or per-subject optimization, limiting its scalability. We introduce InstantHand, a feed-forward framework that constructs an animatable 3D Gaussian hand avatar from a single image using off-the-shelf MANO, camera, and mask estimates, without test-time optimization. Image-averaged losses provide insufficient supervision for alpha leakage, which thickens thin finger contours. To address this issue, we propose a Soft Alpha Band regularizer that concentrates supervision on silhouette boundaries. Furthermore, colors regressed from image features can drift toward a learned prior even in observed regions. Our Reliability-gated Appearance Transfer preserves observed appearance by transferring source colors according to each Gaussian's rendering contribution, while completing unobserved regions using the learned prior conditioned on global source identity. InstantHand constructs an avatar in 2.84 seconds and renders at 161 FPS, matching per-subject-optimized baselines on InterHand2.6M and remaining competitive in zero-shot evaluation on HanCo.