FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility

AI summary

Overview

Research area: Computer vision, specifically monocular 3D human avatar reconstruction using 3D Gaussian Splatting and parametric body models (SMPL-X).

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, SMPL-X parameterization, triplane feature fields, linear blend skinning, and diffusion-based video generation.

Scope: The paper proposes and evaluates a single pipeline, FlexiAvatar, that reconstructs animatable 3D Gaussian human avatars from monocular video across full-body, upper-body, and head-only visibility settings.

What This Paper Is About

Most 3D Gaussian avatar methods assume the entire body is visible during training, which is violated by video calls, talk-show clips, and social media footage where only the face, head, or upper body ever appears. Supervising Gaussians for limbs that are never observed produces hallucinated geometry and texture drift that corrupts even the visible regions. FlexiAvatar addresses this by optimizing only the body regions that are actually observed, while filling genuinely unseen areas (such as the back) with diffusion-generated views.

Key Contributions

1. Visibility-aware Gaussian optimization. A framework that restricts optimization to visually observed body regions, updating geometry and appearance only where image evidence exists, which the authors state eliminates hallucinated artifacts and texture drift and enables full-body, upper-body, or head-only animation from one pipeline.

2. Occlusion-robust SMPL-X registration. A registration pipeline using a binary per-joint visibility mask (threshold τ = 0.4) derived from detector confidence scores, plus joint and facial offsets supervised by a FLAME-aligned facial loss, producing a personalized canonical template stable under partial visibility.

3. Part-specific residual refinement. Three lightweight MLPs specialized for the face, left hand, and right hand that predict color residuals from positional encodings of Gaussian center, time, and pose, activated only when the corresponding part is visible.

4. Diffusion-based generative texture completion. A part-specific motion video generation strategy using MimicMotion that synthesizes an auxiliary video of the subject performing a full 360° rotation, revealing views such as the back to supervise consistently unobserved regions.

Main Findings

• NeuMan full-body results: FlexiAvatar reports PSNR 35.77, SSIM 0.987, LPIPS 0.83, compared with ExAvatar (34.80, 0.984, 0.90), Vid2AvatarPro (32.71, 0.983, 1.19), Vid2Avatar (30.70, 0.980, 1.40), GaussianAvatar (29.94, 0.980, 1.20), 3DGS-Avatar (28.99, 0.974, 1.60), NeuMan (29.32, 0.972, 1.40), InstantAvatar (28.47, 0.972, 2.80), and HumanNeRF (27.06, 0.967, 1.90).

• ZJU-MoCap full-body results: Across subjects 377, 386, 387, 392, 393, and 394, FlexiAvatar reports PSNR of 34.98, 39.50, 36.11, 36.15, 34.66, and 36.63, versus ToMiE (33.63, 37.29, 34.42, 34.46, 32.73, 34.26) and GauHuman (32.63, 36.47, 32.99, 32.47, 32.31, 33.81). The largest gap is in LPIPS, where FlexiAvatar reports 0.66, 0.85, 0.99, 0.80, 1.15, 1.01 against ToMiE's 1.80, 2.91, 2.50, 2.88, 3.02, 2.67.

• TalkShow upper-body results: On Oliver, FlexiAvatar reports PSNR 29.80, SSIM 0.952, LPIPS 1.81 versus ExAvatar (29.13, 0.938, 2.29), GUAVA (26.76, 0.890, 12.17), and GART (24.14, 0.927, 7.76). On Conan it reports 36.73, 0.984, 2.82 versus ExAvatar (35.66, 0.980, 3.61), GUAVA (29.70, 0.928, 7.69), and GART (26.89, 0.974, 5.56). On Chemistry it reports 27.80, 0.935, 9.67 versus GUAVA (26.85, 0.920, 7.15), ExAvatar (25.67, 0.922, 10.29), and GART (22.28, 0.914, 11.89).

• INSTA head-only results: FlexiAvatar reports PSNR 33.04, SSIM 0.953, LPIPS 5.63, compared with RGBAvatar (32.72, 0.953, 6.04), GaussianBlendShapes (30.01, 0.947, 9.17), GaussianAvatars (29.10, 0.945, 8.61), SplattingAvatar (29.03, 0.932, 10.35), MonoGaussianAvatar (28.91, 0.945, 7.43), FATE (27.85, 0.942, 5.68), and FlashAvatar (27.44, 0.912, 11.05).

• WildAvatar in-the-wild results: FlexiAvatar reports PSNR 30.98, SSIM 0.973, LPIPS 2.93 against ToMiE (28.65, 0.957, 3.81) and GauHuman (28.31, 0.957, 3.83).

• Aggregate improvement: The abstract states an average PSNR improvement of approximately 3% over state-of-the-art methods across datasets.

• Efficiency gain: The authors report a nearly 50% reduction in memory footprint for head-only avatars, attributed to visibility-based pruning of low-evidence Gaussians; runtime is described as reduced in partial-visibility scenarios but no specific speed figure is reported in the provided content.

• Visibility-aware optimization ablation: Disabling visibility-aware optimization on the INSTA dataset produces a consistent drop across all metrics, which the authors attribute to the importance of the component for head-only reconstruction.

• Gradient propagation evidence: On the Oliver upper-body sequence, where lower limbs are never observed, the authors show that conventional optimization propagates gradients into the invisible lower body and degrades the reconstructed torso, while visibility-aware optimization confines gradient propagation to the observed upper body.

• Not reported: The quantitative ablation table on the effect of refinement heads and generated views on NeuMan is truncated in the supplied content, so its numbers cannot be stated. Per-region face and hand evaluation numbers are referenced to Supplementary Material Sec. E.6, Tab. A7, which is not included in the supplied content.

Methodology in Plain English

The system takes a monocular video and builds a canonical mesh from SMPL-X, up-sampled to N vertices while preserving the original triangular connectivity, and attaches a learnable 3D Gaussian to each vertex. Identity and animation information are stored in a learnable triplane field: canonical vertex positions are projected onto the three planes, bilinear interpolation yields a per-vertex feature, and MLPs decode that feature into pose-independent Gaussian parameters (offset, scale, color logit). A second pose-dependent branch maps the feature plus body pose to pose-dependent offsets and scales, and a color MLP maps pose and normal to a pose-dependent color term.

Registration uses off-the-shelf estimators for SMPL-X parameters (pose θ ∈ ℝ^(55×3), shape β ∈ ℝ^100, expression ψ ∈ ℝ^50) and 2D keypoints with confidence scores. A binary per-joint mask with threshold τ = 0.4 excludes low-confidence or occluded joints from the 2D reprojection loss. Additional losses penalize deviation from the initialization, align the SMPL-X face to a DECA-initialized FLAME mesh (vertex L1, Laplacian L2, edge-length L1), and regularize shape, joint offsets, and bilateral symmetry.

For unseen regions, MimicMotion generates an auxiliary video of the subject performing a full 360° rotation, providing appearance cues for areas such as the back. Training uses captured and synthetic frames sampled with equal probability, with the synthetic frames' reconstruction losses down-weighted.

Visibility-aware optimization works by computing a per-Gaussian visibility rate: the average, over F frames, of a binary flag indicating whether that Gaussian contributed to a rendered pixel, taken directly from the rasterizer. All Gaussians are optimized without filtering for the first 2,000 iterations as a warm-up, after which the threshold is chosen automatically with Otsu's method over the distribution of visibility rates rather than by manual tuning. Gaussians below the threshold τ* are excluded from forward rasterization and from the Laplacian regularization terms applied to color, scale, and position. The authors argue this matters because all Gaussians share one triplane, so gradients from observed pixels would otherwise leak into unobserved regions and indirectly affect neighboring visible Gaussians.

Part-specific residual refinement activates after the coarse avatar has trained for 2,000 iterations: three small MLPs (face, left hand, right hand) predict color residuals from positional encodings of Gaussian center and time plus pose, and the final color is the sum of the triplane color, the pose-dependent color term, and the residual.

Animation applies SMPL-X pose and expression codes, combining identity and pose-dependent offsets, adopting SMPL-X deformations directly for face and hand regions, adding expression offsets to face vertices, and skinning with linear blend skinning using SMPL-X weights. Rendering uses 3D Gaussian Splatting. The training objective combines L1, SSIM, and LPIPS terms computed on a cropped human region, a facial consistency loss that compares a mesh-rendered facial image against ground truth, and a visibility-aware Laplacian regularization term.

Implementation details: PyTorch, Adam optimizer, a single NVIDIA RTX A6000 GPU, 30k iterations, batch size 1, base learning rate 1×10⁻³, and SAM for body masks.

Why This Matters

Impact on research. The paper reframes partial-body visibility as an optimization-scope problem rather than an architecture problem. Instead of building separate models for head-only, upper-body, and full-body inputs, it argues that a single formulation can cover the whole visibility spectrum by supervising only what is observed. It also provides a diagnostic observation — that shared triplane representations leak gradients from visible into invisible regions — which is relevant to other triplane- or volume-based avatar methods.

Real-world applications:

• Video calls and remote collaboration, where participants are typically captured from the chest or head upward.
• Talk-show, interview, and broadcast footage, where framing rarely includes the full body.
• Social media and user-generated content, where cropping and occlusion are the norm rather than the exception.
• AR/VR and digital content creation pipelines that need animatable avatars from whatever footage end users actually have.

Industry relevance. The claim that a single pipeline replaces per-setting, purpose-built pipelines lowers the engineering and maintenance cost of avatar production, which the authors identify as a significant practical barrier to deployment. The reported nearly 50% memory reduction for head-only avatars, and reduced runtime and memory overhead in partial-visibility scenarios, matter for deployment on constrained hardware.

Future Directions

• Dependence on generative priors. Unseen-region texture relies on MimicMotion-generated 360° rotations; how much residual hallucination these synthetic views introduce, and how robust the approach is when generation fails, is not fully quantified in the provided content.

• Automatic thresholding scope. Otsu's method assumes a bimodal visibility distribution. Whether this holds for sequences with ambiguous or gradually changing visibility, and how sensitive results are to that assumption, is an open question.

• Extension beyond a single subject. All reported experiments use monocular single-subject sequences; multi-person scenes and interaction are not addressed.

• Text and appearance editing. The related-work discussion notes that some head-only methods (FATE) support texture editing. Whether FlexiAvatar's triplane and residual structure supports controllable editing or re-posing to unseen expressions is not reported.

Target Audience

Researchers and graduate students working on 3D human avatar reconstruction, Gaussian Splatting, and monocular human performance capture, particularly those interested in in-the-wild and partial-visibility settings. It is also relevant to applied engineers building avatar pipelines for AR/VR, telepresence, and content creation, who need a single system that operates across different camera framings. Readers without a background in SMPL-X, 3D Gaussian Splatting, or diffusion models will find the methodology section demanding.

Authors’ abstract

Reconstructing animatable 3D human avatars from monocular video is a fundamental problem in computer vision with broad applications in AR/VR and digital content creation. Existing approaches typically couple parametric body models with neural rendering or 3D Gaussian splatting and optimize all body regions jointly from short videos, which often degrades fidelity in the visible areas. To overcome this limitation, we introduce FlexiAvatar, a unified framework that explicitly optimizes only the visible body regions, effectively eliminating artifacts arising from unobserved limbs. Our method integrates occlusion-robust SMPL-X tracking with part-specific residual refinement to capture high-frequency geometric and appearance details. To complete entirely unseen regions (e.g., back views), we leverage a diffusion-based approach to generate texture consistent with the observed appearance. Experiments on full-body (NeuMan, ZJU-MoCap, WildAvatar), upper/half-body (talk-show clips), and head-only (INSTA) inputs show that FlexiAvatar delivers consistently higher reconstruction quality, outperforming state-of-the-art methods by an average PSNR improvement of approximately 3% across datasets. Finally, by restricting optimization to observed regions, our method reduces the effective number of Gaussians that must be optimized and rendered, leading to reduced runtime and memory overhead in partial-visibility scenarios.