Capturing Head Avatar with Hand Contacts from a Monocular Video

AI summary

Overview

Research area: Computer vision — monocular 3D head avatar reconstruction and hand-face interaction modeling.

Technical level: Advanced. The paper assumes familiarity with 3D morphable models (FLAME, MANO), neural implicit surface reconstruction, signed/occupancy fields, and deformation blendshapes.

Scope: The paper presents a two-stage framework that jointly tracks and reconstructs a detailed, textured 3D head avatar together with the non-rigid facial deformations caused by hand contact, using only a monocular iPhone video (arXiv:2510.17181v1, 20 Oct 2025).

What This Paper Is About

Most 3D head avatar methods reconstruct only the face or head and ignore the hand, even though people touch their faces roughly 50 times per hour (a figure the paper cites from prior work) and such contact conveys states like pondering. The core problem is that hand and face are usually tracked separately, so their relative pose and contact are wrong, and no public prior exists for how the face deforms when a hand presses into it. The goal is a single framework that aligns hand and face correctly, reconstructs high-fidelity geometry and texture, and models physically plausible contact-induced deformation from monocular video alone.

Key Contributions

1. A framework that jointly reconstructs detailed 3D head avatars with realistic textures and person-specific geometry while capturing physically plausible non-rigid deformations induced by hand-face interaction, all from a monocular iPhone video.
2. A joint tracking strategy combining a depth order loss (using depth from off-the-shelf estimators) with contact regularization, so that the hand and face meshes keep correct spatial relationships and contact dynamics.
3. A PCA basis for hand-induced facial deformation learned from a face-hand interaction dataset, which reduces deformation estimation to a compact set of PCA parameters instead of a full spatial deformation field, plus a physics-inspired contact loss to reduce interpenetration.
4. Extensive evaluation on both real RGB(D) videos and a newly built synthetic dataset, reporting better appearance fidelity and geometric accuracy than state-of-the-art surface reconstruction methods.

Main Findings

• Joint tracking works without depth hardware at test time: Depth order loss plus contact regularization in preprocessing correctly positions the hand relative to the face; the paper reports that independent tracking (DECA for head, HaMeR for hand) fails to capture their spatial relationship.
• PCA prior makes deformation learnable: Because non-rigid blendshapes and contact parameters are both unknown, the problem is highly under-constrained. Supervising the non-rigid blendshapes with a PCA basis extracted from per-frame non-rigid 3D displacements of FLAME vertices constrains optimization to a compact set of parameters and produces more natural deformations than direct spatial offset prediction.
• Contact loss improves physical plausibility: The contact loss samples points on the hand surface and penalizes face occupancy there, which reduces interpenetration artifacts. The paper states that RGB and mask losses alone are insufficient to learn accurate, plausible facial deformation.
• Qualitative advantage on real videos: On captured iPhone 15 Pro sequences, the method aligns hand and face in a unified coordinate system and produces high-quality avatars from RGB only, whereas the compared baselines NDR and Morpheus require LiDAR-derived depth maps. The paper reports the baselines still fail to reconstruct accurate surface topologies under large head rotations and diverse hand articulation.
• Quantitative results on the synthetic dataset (Table 1): The method reports NC 75.06, CD 2.74, F5 10.22, F10 33.20, versus NDR at NC 52.35, CD 19.14, F5 0.54, F10 1.93 and Morpheus at NC 53.53, CD 18.245, F5 0.31, F10 1.14. F5 and F10 are F-scores at 5 mm and 10 mm thresholds; NC is Normal Consistency; CD is Chamfer Distance over meshes extracted via marching cubes. The table marks NC, F5 and F10 with upward arrows and CD with a downward arrow.
• Baseline behavior on synthetic data: The paper states NDR fails to produce valid hand shapes and facial expressions, while Morpheus improves facial reconstruction but struggles with large hand motion variation and articulated hand shapes.
• Ablation evidence: Contact loss is shown to drive the hand mesh toward the facial surface (penetration visualized as red pixels), and depth order loss prevents excessive interpenetration under extreme head poses. The final column of the preprocessing ablation figure compares against Pixie, which the paper reports as inferior in hand-face positioning.

Methodology in Plain English

The framework runs in two stages.

Stage 1 — preprocessing (getting the hand and face in the right place). The system first estimates FLAME head parameters with DECA and MANO hand parameters with HaMeR for every frame, then refines scale, shift, and pose using a 2D landmark loss. Because separate tracking does not preserve the relationship between hand and face, the authors add two terms. The depth order loss randomly samples pixels in the hand and face regions, reads depth from both the rendered 3DMM depth maps and a pretrained depth estimator, and penalizes cases where the predicted front/back ordering of a hand pixel and a face pixel disagrees with the estimated ordering. The contact regularization pulls fingertip vertices toward the nearest facial vertices in contact-prone areas such as the cheeks, chin, and nose. These are optimized together with the projected landmark loss and temporal smoothness.

Stage 2 — avatar reconstruction. The face is represented as deformable neural implicit fields: a canonical geometry network, a canonical rendering network, and a deformation network, with rays sampled following the IDR approach. Points along a ray are mapped to a canonical space by undoing linear blend skinning and subtracting expression and pose offsets predicted by the deformation network, as well as contact offsets. The geometry network predicts occupancy; the ray-surface intersection is located where occupancy equals 0.5; a rendering MLP produces RGB from the canonical point, the deformed surface normal, and the expression and pose parameters.

The hand is simpler because hand geometry is similar across subjects: the tracked MANO mesh is converted to an occupancy field and a texture MLP maps canonical points and interpolated MANO normals to color, which makes joint rendering of face and hand straightforward.

Contact-induced deformation. A separate non-rigid deformation network takes a canonical point and a per-frame optimizable 30-dimensional latent code and outputs contact-related blendshapes; the same latent code also predicts per-frame contact parameters that scale those blendshapes. To regularize this under-constrained problem, the authors extract per-frame non-rigid 3D displacements of FLAME vertices from a face-hand interaction dataset, build a vertex deformation matrix, run PCA, and keep the top components as a basis used to supervise the non-rigid blendshapes. The contact loss samples points on the hand surface and penalizes face occupancy at those locations, and a regularization term minimizes non-rigid deformation away from penetration regions. Both the contact and regularization losses only update the non-rigid deformation network and contact parameters, so head geometry and expression-related deformation are not disturbed.

Training. The total objective combines a photometric RGB loss on foreground face and hand pixels, a cross-entropy mask loss on predicted occupancy (excluding pixels inside the hand mask to avoid supervising occluded regions), and a FLAME loss aligning predicted blendshapes, pose correctives and skinning weights with pseudo ground truth, including the non-rigid blendshapes against the PCA basis. Reported weights are λe = 1000, λp = 1000, λw = 0.1, λn = 10000, λM = 2, λlbs = 1, λcontact = 1000, and λreg = 10.

Data used for evaluation. A synthetic dataset of 3 subjects performing 4 hand-face interaction sequences, built with Unreal Engine 5's MetaHuman Creator (Face ID driving facial animation, manually designed hand interactions) and non-rigid deformation simulated with Position-Based Dynamics through Blender Geometry Nodes; it provides rendered video, segmentation masks, depth maps, surface normal maps, and ground-truth mesh tracking for face and hand. A real-video dataset of four sequences captured with the iPhone 15 Pro LiDAR sensor, about 1,000 frames each, single subject, four interaction tasks, with masks from off-the-shelf video segmentation, keypoints from a face keypoint detector and Sapiens, and MANO mesh landmark projections substituted where Sapiens fails on occluded thumb regions.

Why This Matters

The work moves head-avatar reconstruction past the face-only assumption and treats the hand as a first-class cause of deformation, which matters because hand-face contact is both frequent and communicative. It also shows that this can be done from an ordinary monocular phone video, rather than requiring a depth sensor at inference time.

Real-world applications:

• Telepresence and VR/AR avatars where a user's hands touch their face and the avatar must deform plausibly, not clip through the head.
• Gaming and digital humans, where characters need convincing skin response to self-contact.
• Behavioral and HCI analysis, since hand-face touch is a nonverbal cue tied to cognitive states such as pondering.
• Content creation and film/VFX pipelines, providing a way to capture a person-specific head rig with contact behavior from consumer footage.

Industry relevance: the method targets consumer capture hardware (an iPhone) and uses widely adopted components — FLAME, MANO, DECA, HaMeR — which lowers the barrier to integration into existing avatar pipelines and telepresence products. The paper notes the optimization is slow, so real-time deployment is not yet achieved.

Future Directions

• The physics-inspired contact loss is limited: the authors note it cannot model effects such as skin pulling or friction.
• Material properties of skin, muscle, and fat are not explicitly modeled, which the authors flag as an area for further exploration.
• The optimization process is slow and does not support real-time applications, motivating work on accelerating interaction modeling.
• The paper raises an implicit question about evaluation: real-video results are qualitative and use RGB only, while quantitative mesh metrics come from the synthetic dataset, so broadening ground-truth-based evaluation on real captures is an open problem.

Target Audience

Researchers and graduate students working on neural implicit surface reconstruction, 3D morphable models, and monocular human performance capture; practitioners building avatar or telepresence systems who need detailed face geometry plus physically coherent hand interactions; and technically fluent readers interested in how contact constraints and statistical deformation priors can regularize severely under-constrained reconstruction problems.

Authors’ abstract

Photorealistic 3D head avatars are vital for telepresence, gaming, and VR. However, most methods focus solely on facial regions, ignoring natural hand-face interactions, such as a hand resting on the chin or fingers gently touching the cheek, which convey cognitive states like pondering. In this work, we present a novel framework that jointly learns detailed head avatars and the non-rigid deformations induced by hand-face interactions. There are two principal challenges in this task. First, naively tracking hand and face separately fails to capture their relative poses. To overcome this, we propose to combine depth order loss with contact regularization during pose tracking, ensuring correct spatial relationships between the face and hand. Second, no publicly available priors exist for hand-induced deformations, making them non-trivial to learn from monocular videos. To address this, we learn a PCA basis specific to hand-induced facial deformations from a face-hand interaction dataset. This reduces the problem to estimating a compact set of PCA parameters rather than a full spatial deformation field. Furthermore, inspired by physics-based simulation, we incorporate a contact loss that provides additional supervision, significantly reducing interpenetration artifacts and enhancing the physical plausibility of the results. We evaluate our approach on RGB(D) videos captured by an iPhone. Additionally, to better evaluate the reconstructed geometry, we construct a synthetic dataset of avatars with various types of hand interactions. We show that our method can capture better appearance and more accurate deforming geometry of the face than SOTA surface reconstruction methods.