EDGAR: Efficient Diffusion-Driven Gaussian Avatar Refinement via Synthetic Failure

AI summary

Overview

Research area: 3D computer graphics and computer vision — specifically generative refinement of animatable 3D Gaussian head avatars, combining diffusion models, Gaussian splatting, and human-specific feature extractors.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, diffusion models (Stable Diffusion / SD-Turbo), LoRA adaptation, and Mixture-of-Experts routing.

Scope: EDGAR is a one-step diffusion-based refinement framework that corrects artifacts in renderings of Gaussian head avatars and distills those corrections back into the underlying 3D avatar, trained on a purpose-built 400K-image triplet dataset of simulated avatar failures.

What This Paper Is About

Gaussian-splatting head avatars built from a single video or a few images are cheap to capture but severely under-constrained, so their renderings show blurring, stretching, floating blobs, and temporal instability in poorly observed regions. Existing fixes based on SDEdit-style noise-and-denoise refinement guess appearance from a generic prior, which introduces randomness, temporal jitter, and identity drift. The authors ask whether a diffusion refiner can be trained specifically on the failure modes of Gaussian avatars — with a subject reference image and expression supervision — so that artifacts are removed while identity and facial motion are preserved, and whether those 2D corrections can be baked back into the 3D avatar so no diffusion model is needed at render time.

Key Contributions

1. EDGAR-SET, a hybrid triplet dataset of failure image → ground-truth image → reference image pairs built from real captures (NeRSemble, reconstructed via RGBAvatar, 99 identities, 1–4 viewpoints) and synthetic data (CAFCA reconstructed with 3DGS, 1,500 identities, at least 5 viewpoints), totaling 400K high-resolution images across thousands of identities after filtering out roughly half the raw pairs.

2. EDGAR, a conditional one-step diffusion refiner built on SD-Turbo in which a fusion block injects self-supervised Sapiens features from a reference image into the cross-attention layers, and a Mixture-of-Experts module (six expert MLPs) specializes across degradation types and severities.

3. Avatar distillation, a stage that uses EDGAR-refined multi-view renderings as pseudo-targets to optimize the single underlying Gaussian avatar with L1 and SSIM losses, so improvements are absorbed into 3D and the final avatar renders without per-frame diffusion inference — and without splitting or adding Gaussians.

4. Identity- and motion-preserving supervision, combining perceptual (LPIPS), pixel (ℓ2), Gram-matrix style, POSTER V2 feature-based expression consistency, and MoE load-balancing losses, with zero-initialized added layers so training starts from exactly the pretrained SD-Turbo behavior.

Main Findings

• Consistent gains on the more degraded baselines: On RGBAvatar and GaussianAvatars, EDGAR leads all four metrics (CLIP, L1, LPIPS, PSNR) at every input count. For RGBAvatar with 8 videos, CLIP rises from 0.9210 to 0.9379, L1 falls from 0.0502 to 0.0332, LPIPS from 0.2100 to 0.1894, and PSNR from 20.5128 to 22.9937.

• Partial gains on CAP4D: EDGAR achieves the highest CLIP and lowest L1 at every input count (e.g., 8 views: CLIP 0.9462 → 0.9531, L1 0.0518 → 0.0513), but the unrefined baseline retains the lowest LPIPS and the SDEdit baseline the highest PSNR in that comparison.

• Identity preservation versus SDEdit: In cross-identity reenactment measured by CSIM, EDGAR lands close to the input avatar (0.5169 / 0.5326 / 0.5419 for 1 / 4 / 8 views versus base 0.5510 / 0.5642 / 0.5651), whereas SDEdit collapses to 0.3178 / 0.3432 / 0.3380.

• Qualitative artifact removal: EDGAR restores eyes, wrinkles, sharper hair, and skin texture and removes floating blobs around the ear, neck, and hair, while SDEdit refinement oversmooths texture and shifts identity.

• Ablations matter most for the reference image: Removing the reference image I_r causes the largest drop in cross-reenactment CSIM (0.4508 at 1 view, versus 0.5169 for the full model) and clearly degrades self-reenactment CLIP and LPIPS. Removing the synthetic split, the expression loss, or the MoE (or reducing to two experts) each produce smaller but measurable regressions.

• Robustness claims: Cross-view evaluation shows improved calibrated consistency, and tests under large yaw and open-mouth expressions verify robustness in extreme settings — which the authors frame as "a useful operating regime for recoverable appearance artifacts."

• Known limitation: Severe structural failures are not reliably represented by the available image/reference pairs; the more severely degraded GaussianAvatars example is improved but remains only partially corrected.

Methodology in Plain English

The authors first need training data that does not exist: pairs of "bad avatar rendering" and "what it should look like." They manufacture it. They take high-quality multi-view datasets of real people, build avatars from only some of the views so the avatars are deliberately under-constrained, render them from new angles and expressions to get the failure images, and keep the high-quality renders as ground truth. A separate frame of the same person (different view or expression) becomes the reference. They discard roughly half the pairs using LPIPS similarity, because if the failure image is too far from the target, the recovery task is ambiguous — for example when the expression itself is wrong.

They then train a refiner. The starting point is SD-Turbo, a distilled diffusion model that denoises in a single step. Most of it is frozen; only the self-attention layers (which control pixel relationships) and a LoRA-augmented decoder are trained. The reference image is passed through Sapiens, a large self-supervised human model, and its features are injected as keys and values into cross-attention while the diffusion features act as queries — so the network knows who it is fixing. Because the new layers are zero-initialized, the model starts out identical to SD-Turbo and gradually learns to refine.

A small router inspects intermediate features and sends each sample to one of six expert networks, so different artifact types and severities are handled by different specialists; a load-balancing term prevents all samples from collapsing onto one expert. Training combines pixel, perceptual, and texture-style losses with a facial-expression loss computed from POSTER V2's internal features.

Finally, refinement is not just post-processing. The authors render the avatar from 50 views, run EDGAR on each, and then optimize the original Gaussian parameters for 500 steps to match those refined images with L1 (weight 0.6) and SSIM (weight 0.4) losses. One reference image is chosen at random and held fixed throughout. Because a single shared 3D representation is optimized across many views, the corrections stay 3D-consistent, and the finished avatar renders at its original speed.

Why This Matters

Research impact: The paper reframes avatar refinement as a supervised failure-correction problem rather than blind restoration, and it shows that synthetic degradation of real reconstructable subjects is a viable way to generate supervision that no existing dataset provides. The distillation step — turning a 2D refiner into an improved 3D asset — is a template for improving representations without paying generative cost at inference.

Real-world applications:

• Consumer VR telepresence and social gaming avatars captured from a phone video, where side profiles and open-mouth expressions currently break down.
• Virtual production and film previz, where an actor's avatar must hold up under extreme yaw angles and expressive dialogue.
• Live streaming and video-conferencing avatars that need real-time rendering without a diffusion model in the loop.
• Asset cleanup pipelines that ingest avatars from third-party capture tools and upgrade them without re-shooting.

Industry relevance: Because EDGAR applies to arbitrary Gaussian avatars (demonstrated on CAP4D, RGBAvatar, and GaussianAvatars) and preserves the original Gaussian count, it can be dropped into existing production pipelines as a post-hoc quality step rather than requiring a new reconstruction stack.

Future Directions

• Handling severe structural failure: The authors explicitly note that irreparable cases were filtered out and that heavier degradation (e.g., the GaussianAvatars example) is only partially corrected. Extending the approach to geometry-level failures — not just recoverable appearance artifacts — is the clear next question.
• Better cost accounting and throughput: The paper references a per-avatar refinement time table and separate rendering-throughput benchmark, but those numbers are not included in the provided content, leaving the practical efficiency trade-off of 50-view rendering plus 500 optimization steps unquantified here.
• Routing interpretability: With six experts trained by a self-supervised router, it remains open what each expert specializes in — degradation type, severity, reference-view distance, or something else — and whether routing could be made controllable.
• Beyond heads: The triplet-construction recipe (degrade a reconstruction by restricting its inputs, keep the clean render as target) is domain-agnostic and could plausibly extend to hands, full bodies, or object-level Gaussian assets.

Target Audience

Researchers and graduate students in 3D vision and graphics working on avatar reconstruction, Gaussian splatting, or diffusion-based restoration; engineers building real-time avatar pipelines for VR/telepresence who need to know how far monocular-capture quality can be pushed without new hardware; and practitioners interested in dataset synthesis strategies — specifically how to manufacture supervised pairs for a task where no ground-truth corpus exists.

Authors’ abstract

Recent advances in Gaussian splatting have enabled efficient and photorealistic head avatar reconstruction, yet monocular video-based or image-based methods still suffer from under-constrained geometry and texture, leading to artifacts such as blurring, stretching, and temporal instability. We present EDGAR (Efficient Diffusion-Driven Gaussian Avatar Refinement), a conditional diffusion framework designed to enhance the quality of Gaussian avatars by correcting artifacts while preserving identity fidelity. To train EDGAR, we construct EDGAR-SET, a hybrid triplet dataset composed of real-captured and synthetic-rendered samples, which simulates realistic Gaussian avatar failure cases by pairing degraded renderings with ground-truth and reference images. EDGAR refines degraded frames through a fusion block that injects identity features extracted by the self-supervised Sapiens model into a one-step diffusion backbone, enabling effective artifact removal without sacrificing efficiency. Our EDGAR model incorporates a Mixture-of-Experts (MoE) architecture to address diverse degradation types and severity levels. The refined multi-view renderings are subsequently used as pseudo-targets to optimize a single underlying Gaussian avatar, so that the final avatar can be rendered without applying EDGAR to every output frame. Experiments on three different Gaussian avatar reconstruction methods, each evaluated under three different settings, show improvements in both perceptual and reconstruction measures over the raw outputs and an SDEdit-style refinement baseline. In addition, cross-view evaluation demonstrates improved calibrated consistency, while tests under large yaw and open-mouth expressions verify robustness under extreme settings. These results collectively indicate a useful operating regime for recoverable appearance artifacts. The code is available at https://kwanyun.github.io/edgar/