FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision

AI summary

Overview

Research area: Computer vision / 3D head avatar reconstruction and animation (3D Gaussian Splatting, transformers, neural rendering).

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, transformers, cross-attention, 3D morphable models, and standard image-quality metrics.

Scope: FlexAvatar is a transformer-based encoder-decoder pipeline that creates complete, animatable 3D head avatars from a single portrait image by combining monocular and multi-view training data through learnable "bias sink" tokens.

What This Paper Is About

Creating a full 3D head avatar from one photo is underconstrained: much of the head is unobserved, and the model must invent plausible facial animation for a person it has never seen moving. Training on internet-scraped monocular portrait videos gives broad identity coverage but produces incomplete heads, because the model can cheat — it learns to guess the target camera viewpoint from the expression signal. FlexAvatar's goal is to get the generalization benefits of monocular data and the 3D completeness of multi-view data in one model, then use only the multi-view behavior at inference time.

Key Contributions

1. A pipeline for high-quality 3D head avatars from a single image, described by the authors as novel and efficient, able to produce an avatar in minutes, with head animation learned directly from data rather than rigged to a restrictive 3DMM.
2. Learnable bias sinks — two tokens, z_2D and z_3D, concatenated to the expression code sequence — that absorb dataset-type-specific biases and let a model trained on mixed monocular and multi-view data behave like a purely multi-view-supervised model at inference while retaining monocular generalization.
3. An efficient upsampler architecture combining StyleGAN2-style CNN blocks with PixelShuffle, which the paper reports improves visual quality, especially in sensitive facial regions such as the eyes and mouth interior.
4. A smooth latent avatar space that emerges from the encoder-decoder design, enabling interpolation between identities and flexible fitting to an arbitrary number of input observations (few-shot and monocular video avatar creation).

Main Findings

• Monocular training alone fails on novel views. The paper identifies "entanglement of driving signal and target viewpoint" as the root cause: in monocular datasets I_drive = I_target, so the expression code can leak the target viewpoint. The ablation "only 2D" yields partial 3D heads (PSNR 13.7, SSIM 0.736, LPIPS 0.358, AKD 6.59, CSIM 0.593 on Ava256).
• Multi-view training alone fixes completeness but destroys generalization. The "only 3D" ablation gives complete high-quality avatars but poor generalization to unseen identities, "reflected in low CSIM scores" (CSIM 0.119 versus 0.593 for only 2D and 0.621 for the reference model).
• Mixing data without bias sinks does not work. The "w/o bias sinks" ablation reaches PSNR 14.5 and does not produce complete 3D heads for unseen images; the paper states the model mainly learns to identify the dataset rather than resolve viewpoint-expression entanglement.
• Bias sinks work as intended. In a separate analysis, fine-tuning a 2D-only model on NeRSemble with one bias sink per each of the dataset's 16 cameras shows each sink captures the viewpoint bias of its training subset: the "left cam" sink yields a head complete only from the left, the "right cam" sink the opposite.
• 3D portrait animation. On the VFHQ test split, FlexAvatar improves over the previous state of the art in all metrics except AKD. Self-reenactment: PSNR 23.47, SSIM 0.837, LPIPS 0.099, CSIM 0.830, AED 0.075, APD 0.010, AKD 2.965. Cross-reenactment: CSIM 0.663, AED 0.223, APD 0.026. (Comparators include LAM at PSNR 22.65 and AKD 2.059, GAGAvatar at 21.83, Portrait4D-v2 at 21.34, Real3DPortrait at 20.88, Portrait4D at 20.35, GPAvatar at 21.04.)
• Single-image avatar creation on Ava256. FlexAvatar reaches PSNR 16.9, SSIM 0.762, LPIPS 0.265, AKD 5.52, CSIM 0.695, versus LAM (13.1 / 0.702 / 0.399 / 11.2 / 0.411), GAGAvatar (12.7 / 0.709 / 0.371 / 7.45 / 0.555) and Portrait4D-v2 (11.9 / 0.671 / 0.404 / 7.77 / 0.578). The entire Ava256 dataset was held out during training, and the released LAM used in the comparison was itself trained on both monocular (VFHQ) and multi-view (NeRSemble) data, so the gains are not solely attributable to multi-view supervision.
• Few-shot avatar creation. With 4 input images and 1000 optimization steps (approximately 7 minutes per avatar), FlexAvatar scores PSNR 21.1, SSIM 0.733, LPIPS 0.218, AKD 5.39, CSIM 0.755, outperforming Avat3r (20.8 / 0.715 / 0.310 / 5.66 / 0.616), GPAvatar (20.0 / 0.700 / 0.291 / 5.72 / 0.341) and InvertAvatar (13.0 / 0.288 / 0.590 / 52.3 / 0.296), with particular gains in sharpness (LPIPS) and identity preservation (CSIM).
• Monocular avatar creation on the NeRSemble benchmark. From 900 frames and 2000 fitting iterations (approximately 10 minutes per avatar), FlexAvatar reaches PSNR 20.9, SSIM 0.830, LPIPS 0.156, JOD 6.08, AKD 3.80, CSIM 0.827, beating RGBAvatar (20.6 / 0.829 / 0.181 / 6.03 / 3.41 / 0.824) and CAP4D (19.8 / 0.821 / 0.185 / 5.79 / 4.19 / 0.793) with faster fitting (10 minutes versus 4 hours for CAP4D) and fewer frames.
• Upsampler ablation. Replacing the StyleGAN-PixelShuffle block with standard PixelShuffle drops performance to PSNR 17.1, SSIM 0.765, LPIPS 0.287, AKD 7.03, CSIM 0.614, versus 17.2 / 0.768 / 0.285 / 6.34 / 0.621 for the reference configuration (all ablation models trained for 500k iterations).
• Fitting helps. A single forward pass already produces accurate avatars, but fitting the avatar code further improves identity, sharpness and expression fidelity: PSNR 16.9, SSIM 0.771, LPIPS 0.280, AKD 5.59, CSIM 0.682. Fitting is fast (approximately 1 minute) because only the avatar code is optimized while the network stays frozen.
• Comparison with FaceLift. On Ava256, using 4 frontal and 4 back cameras for 5 subjects, FlexAvatar scores front-view PSNR 17.2, SSIM 0.786, LPIPS 0.265, AKD 4.72, CSIM 0.771 versus FaceLift's 12.8 / 0.715 / 0.357 / 6.32 / 0.658; back-view PSNR 15.2, SSIM 0.709, LPIPS 0.408 versus FaceLift's 13.2 / 0.687 / 0.411. FaceLift does not support head animation.

Methodology in Plain English

The system has three parts. An encoder takes one input image, extracts features with a pre-trained DINOv2 model plus a small learnable ViT (also feeding in the camera's Plücker embeddings), and then uses cross-attention from queries anchored in the UV space of a template head mesh to compress the image into a 2D latent avatar code. A decoder takes that code and, via cross-attention to a sequence of expression tokens, injects facial motion; a PixelShuffle plus StyleGAN2-style CNN upsampler then increases the resolution (two upsampling steps plus another 2x in grid sampling, for a total 8x upsampling rate) so enough 3D Gaussians can be decoded. Finally, the Gaussians are rendered from a chosen camera with the gsplat rasterizer.

The key trick is the bias sinks. During training on both monocular and multi-view data, a learnable token is appended to the expression sequence: z_2D for monocular samples and z_3D for multi-view samples. The decoder becomes explicitly aware of where a sample came from, so it can put the monocular data's bias (guess-the-viewpoint behavior) into z_2D and learn genuinely complete geometry under z_3D. At inference the model is always fed z_3D, so it inherits multi-view behavior and generalization at once. The network is trained with L1, SSIM, and perceptual losses based on DINOv2 and SAM features. Because the encoder-decoder design yields a smooth avatar latent space, the avatar code can also be initialized by the encoder from any single observation and then optimized against many observations while the decoder stays frozen — this is what enables few-shot and monocular video results.

Why This Matters

Impact on research. The paper reframes a known failure mode — incomplete 3D heads from monocular training — as a dataset-provenance bias problem, and offers a minimal, general mechanism (dataset-level learnable tokens) for suppressing that bias at inference. That idea is not specific to heads and may transfer to other domains where scarce 3D or multi-view data must be combined with abundant partial supervision. The result also challenges the common reliance on rigging predictions to a 3DMM such as FLAME for animation, by learning motion directly from data.

Real-world applications (as listed or implied by the paper):

• Immersive teleconferencing with personalized, freely viewable avatars.
• Virtual try-on for glasses, caps and other accessories, which the paper explicitly tests on in-the-wild images.
• Personalized video games and education, where avatars can be generated from text-to-image outputs.
• Avatar creation from everyday capture such as a phone scan or a monocular video, in minutes rather than the hours required by some baselines.

Industry relevance. The quality-versus-cost tradeoff matters commercially: FlexAvatar claims avatar creation within minutes, roughly 10 minutes fitting versus 4 hours for CAP4D on the NeRSemble benchmark, while matching or exceeding prior quality. It also removes the need for expensive multi-view capture rigs at deployment time, since only multi-view data is needed at training time, not for the end user.

Future Directions

• More expressive driving signals. All experiments use FLAME expression codes, which limits fine details such as the tongue; the model-agnostic design could instead use expression codes from implicit morphable models or features from generalized expression encoders.
• Explicit lighting control. Lighting is currently baked from the input image, which can look unnatural when the avatar is placed in a different virtual environment.
• Extending bias sinks to other domains, such as human bodies or generalized dynamic novel view synthesis, and more broadly to any setting where scarce 3D data must be combined with monocular partial supervision.
• Interpolation and editing workflows. The smooth latent space already supports convex interpolation between identities and flexible fitting to arbitrary numbers of observations; the paper treats further applications of this space (and of latent editing of A, z_exp and camera) as open ground.

Target Audience

Researchers and graduate students working on 3D head avatars, neural rendering, monocular-to-3D reconstruction, and Gaussian Splatting will benefit most, since the paper's core insight applies directly to their training pipelines. Practitioners building avatar products for teleconferencing, gaming, education or virtual try-on should also read it for the accuracy-versus-fitting-time tradeoffs, and anyone training on mixed-quality datasets will find the bias-sink concept portable. Readers will need a working understanding of transformers and neural rendering to follow the architecture details, though the central idea — train with a dataset-type token, infer only with the good one — is understandable without the math.

Authors’ abstract

We introduce FlexAvatar, a method for creating high-quality and complete 3D head avatars from a single image. A core challenge lies in the limited availability of multi-view data and the tendency of monocular training to yield incomplete 3D head reconstructions. We identify the root cause of this issue as the entanglement between driving signal and target viewpoint when learning from monocular videos. To address this, we propose a transformer-based 3D portrait animation model with learnable data source tokens, so-called bias sinks, which enables unified training across monocular and multi-view datasets. This design leverages the strengths of both data sources during inference: strong generalization from monocular data and full 3D completeness from multi-view supervision. Furthermore, our training procedure yields a smooth latent avatar space that facilitates identity interpolation and flexible fitting to an arbitrary number of input observations. In extensive evaluations on single-view, few-shot, and monocular avatar creation tasks, we verify the efficacy of FlexAvatar. Many existing methods struggle with view extrapolation while FlexAvatar generates complete 3D head avatars with realistic facial animations. Website: https://tobias-kirschstein.github.io/flexavatar/