OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting

AI summary

Overview

Research area: Computer vision — 3D human/avatar reconstruction, specifically single-image hand avatar reconstruction using 3D Gaussian Splatting (3DGS).

Technical level: Advanced. The paper assumes familiarity with NeRF-style implicit fields, 3D Gaussian Splatting, MANO-family parametric hand models, attention/transformer architectures, and linear blend skinning.

Scope: The paper introduces OASIS, a 3DGS-based framework that reconstructs an animatable, high-fidelity hand avatar from a single RGB image by transferring image evidence to 3D hand geometry in an occlusion-aware way.

What This Paper Is About

Reconstructing a full 3D hand avatar from one photograph is an ill-posed problem: a single view leaves most of the hand unseen, hands self-occlude heavily, and the fingers deform non-rigidly as they articulate. Existing hand avatar methods mostly use implicit NeRF-style volumetric representations, which are expensive to fit and render and tend to lose fine detail such as fingernails.

OASIS instead builds an explicit 3D Gaussian Splatting pipeline tailored to hands. It aligns image features with 3D hand geometry, uses an estimated per-point visibility score to decide whether a point should trust local image evidence or fall back on global appearance context, and represents Gaussian features on mesh faces rather than only at vertices to better capture local surface stretching.

Key Contributions

1. OASIS itself: a 3DGS-based one-shot hand avatar reconstruction model that produces animatable, high-fidelity hand avatars from a single image.
2. Geometry-aligned visual evidence tokens (VETs) plus visibility-conditioned evidence transfer: the paper constructs geometry-aligned visual evidence tokens and performs visibility-conditioned point-image attention to produce occlusion-aware Gaussian features, enabling faithful and robust single-view reconstruction.
3. Feature-on-Mesh (FoM) representation: features are organized over mesh faces so that Gaussian deformation is guided by local surface stretching under non-rigid hand articulation.
4. Empirical validation and versatility: experiments on InterHand2.6M and in-the-wild images show consistent gains in fidelity, robustness and rendering efficiency over baselines, and the pipeline also supports text-to-avatar generation and texture editing.

Main Findings

• Best one-shot accuracy on InterHand2.6M: OASIS with one-shot adaptation reaches PSNR 27.38, SSIM 0.956 and LPIPS* 11.45 (LPIPS* = 100 × LPIPS), outperforming all single-image baselines across all three metrics.
• The prior network alone is already stronger than baselines: the no-finetuning variant ("Ours §") scores PSNR 26.48, SSIM 0.950, LPIPS* 11.74, exceeding OHTA (26.11 / 0.864 / 12.93), Handy (25.56 / 0.794 / 14.98), HandAvatar with 1 training image (23.79 / 0.820 / 17.78), HARP (19.82 / 0.761 / 22.49) and the no-finetuning LHM § (24.45 / 0.901 / 13.96).
• Competitive with multi-image methods despite far less supervision: methods trained on 11,757 images report PSNR/SSIM/LPIPS* of 26.38 / 0.879 / 14.21 (SelfRecon), 27.64 / 0.884 / 11.45 (HumanNeRF) and 28.23 / 0.894 / 10.35 (HandAvatar); OASIS trained on 1 image is reported as the most competitive against them.
• Large speed advantage: rendering runs at approximately 390 FPS, while OHTA and HandAvatar render at roughly 0.16 FPS — about 2500× more time per single image. FPS is measured on an NVIDIA A100 GPU.
• Fast personalization: one-shot fitting takes approximately 5 minutes, compared with roughly 1 hour (10× more time to converge) for OHTA.
• OHTA loses high-frequency detail: the paper notes OHTA struggles with high-frequency texture details, "particularly evident in nails," while OASIS reconstructs them faithfully.
• Color calibration matters: the authors report that the no-finetuning model shows color discrepancies because the test subject was unseen during training, which they cite as validating the one-shot reconstruction pipeline.
• Ablation results (Table 2, one-shot setting): full model 27.38 / 11.45 / 0.956; w/o VETs 26.97 / 12.11 / 0.955; w/o Aggr. 27.08 / 11.87 / 0.955; w/o LTEP 27.12 / 11.98 / 0.955; w/o VPIA 27.10 / 12.09 / 0.955; w/o bias 27.07 / 12.08 / 0.955; w/o FoM 27.13 / 12.35 / 0.955. Removing any component degrades at least one metric relative to the full model.
• Downstream versatility: the pipeline is shown to support text-to-avatar generation and texture editing, and accepts human-image inputs, with demonstrated robustness to challenging poses, complex textures and extreme side views.
• Additional datasets: more quantitative comparisons are reported on the HanCo dataset in the appendix (C.3), with additional qualitative comparisons in appendix C.4; those numbers are not included in the provided content. Appendix B holds further implementation details, and appendix 4 holds additional ablations on the one-shot reconstruction strategy.

Methodology in Plain English

Starting point. Given one RGB hand image, the system predicts MANO hand shape and pose parameters and camera pose using an off-the-shelf estimator, then uses the MANO-HD template (a super-resolution version of MANO) to keep the hand topology. The goal is a set of 3D Gaussians in a canonical space that can be posed later with linear blend skinning.

Aligning image evidence to geometry. Rather than dumping raw backbone features (from a DINOv2 network) into the model, the authors project every posed 3D hand point onto the image plane and sample the feature map in a small local neighborhood around that projection (K = 5 samples with K = 5). Neighboring samples are combined with distance-based weights, so each 3D point gets a "point-aligned" image feature that reflects the local appearance around it.

Compressing to tokens. Because many points receive nearly identical evidence, these features are overcomplete. A set of learnable query tokens uses cross-attention to aggregate the point-aligned features into a compact set of visual evidence tokens, followed by self-attention so tokens can exchange information and a feed-forward network for refinement.

Handling occlusion. The key idea is that a self-occluded point projects onto visible hand surface that is not its own, so similarity-based attention would wrongly copy that appearance. The system computes a soft visibility score per point by comparing its depth with the z-buffer depth of the posed mesh at its projected location, pushed through a sigmoid. That score becomes an additive bias in the point-to-image cross-attention: visible points are biased toward the local visual evidence tokens, occluded points are biased toward a global appearance token. The result is an occlusion-aware Gaussian feature per point. Self-attention then refines these features so they share structural and visual context.

Modeling non-rigid deformation. Vertex-only features miss stretching and shearing that occur across mesh faces. So features are also lifted onto triangles using learnable barycentric coordinates (softmax-normalized), producing face-level features. Each face feature decodes to Gaussian attributes — opacity, color, scaling, rotation, and barycentric coordinates for positioning the Gaussian inside the triangle. At animation time the mesh is deformed with linear blend skinning and Gaussian positions follow barycentric interpolation of the posed triangle vertices; face-level rotations derived from mesh deformation guide the Gaussian transforms.

One-shot personalization. The model is first pretrained on multi-identity data to learn a shared hand prior. To fit a new image, the authors apply color calibration to correct global appearance mismatch between the input and the prior, then fine-tune the Gaussian decoder with LoRA to capture target-specific textures — 100 steps for color calibration and 400 steps for texture details.

Training setup. Pretraining uses 21 subjects from the InterHand2.6M training set, randomly sampling one source-view image and four target-view images per subject. Training runs on two NVIDIA A100 40GB GPUs with the Adam optimizer at a learning rate of 4×10⁻⁴. Loss weights are λ₁ = 10, λ₂ = λ₃ = 1, λ₄ = 5, λ_p = 1, λ_s = 0.5. In the one-shot stage, λ_μ = 20 and λ_σ = 1.

Why This Matters

Impact on research. The work argues that the dominant implicit NeRF-style approach to hand avatars is a practical bottleneck: dense ray sampling and repeated field evaluations limit interactivity and Deployment. By moving hand avatar reconstruction to 3DGS and explicitly confronting hand-specific problems — severe self-occlusion, sparse single-view evidence, and non-rigid articulation — it reframes single-image hand reconstruction around explicit Gaussian representations. It also introduces the Feature-on-Mesh idea of carrying Gaussian features on mesh faces rather than vertices, which is a generalizable representational trick.

Real-world applications (as supported by the paper):

• Virtual and augmented reality gesture-based interaction, since hands are the primary medium for grasping, manipulation and gesture communication in virtual environments.
• Digital human and avatar creation for entertainment and telepresence from a single photograph rather than a multi-camera rig or monocular video capture.
• Text-to-avatar generation, demonstrated as a downstream application.
• Texture editing on reconstructed hand avatars, also demonstrated downstream.

Industry relevance. The reported figures — roughly 5-minute adaptation and approximately 390 FPS rendering versus roughly 0.16 FPS for OHTA and HandAvatar — matter for deployment where interactive framerates and short capture-to-avatar turnaround are requirements. Single-image input removes the need for expensive multi-view capture setups. The authors state the code will be released on the project page.

Future Directions

1. Closing the remaining gap to multi-image supervision. Multi-image methods trained on 11,757 images still lead on PSNR (HandAvatar at 28.23 versus OASIS at 27.38), and the paper frames OASIS as merely "the most competitive" against them. Further prior learning or stronger generative priors are a natural next step.
2. Reducing reliance on the one-shot finetuning stage. The no-finetuning variant shows color discrepancies for unseen subjects, which the authors use to justify personalization. Whether this gap can be narrowed — making adaptation even shorter than the reported 100 + 400 steps — is open.
3. Broader dataset validation. Additional comparisons on the HanCo dataset are promised in the appendix rather than the main body; extending quantitative evaluation to more in-the-wild and cross-dataset settings would test generalization more thoroughly.
4. Extending beyond hands. The paper motivates OASIS by contrast with full-body and head avatars, which exhibit less articulation and self-occlusion. The visibility-conditioned attention and Feature-on-Mesh designs are natural candidates for other highly articulated or heavily occluded body parts.

Target Audience

Researchers and graduate students working on 3D avatar reconstruction, 3D Gaussian Splatting, neural rendering, and digital human modeling; practitioners building VR/AR hand interaction or avatar creation systems who need fast rendering and short adaptation times; and readers already comfortable with NeRF, parametric hand models such as MANO, and attention-based multimodal architectures who want to see how occlusion reasoning can be injected into a Gaussian splatting pipeline.

Authors’ abstract

Single-image 3D hand avatar reconstruction is fundamentally ill-posed and particularly challenging due to limited visual evidence under severe self-occlusion and the complex pose-dependent deformation of highly articulated hands. Existing methods predominantly rely on implicit NeRF-style representations, whose volumetric fitting is computationally expensive and often struggles to preserve fine-grained hand details. In this work, we present OASIS, a tailored 3D Gaussian Splatting framework for single-image hand avatar reconstruction. To faithfully encode sparse image-specific appearance cues in single-view reconstruction, we construct geometry-aligned visual evidence tokens by explicitly aligning input image observations with 3D hand geometry and context-adaptively tokenizing the resulting visual evidence. Since severe self-occlusion makes the reliability of image evidence inherently visibility-dependent, we introduce a visibility-conditioned point-image attention to reliably transfer visual evidence to geometric tokens, yielding occlusion-aware Gaussian features for faithful and robust reconstruction. To further capture non-rigid deformation of articulated hands, we introduce a Feature-on-Mesh representation to enable Gaussian deformation to be guided by local surface stretching. Under this framework, we adopt a one-shot adaptation scheme that learns a shared hand prior from multi-identity training data and then fits it to a target image for target-specific reconstruction. Extensive experiments show that OASIS outperforms existing baselines in both visual fidelity and efficiency across challenging poses and in-the-wild scenarios, and further demonstrates strong versatility in downstream applications such as text-to-avatar generation and texture editing.