Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture

AI summary

Overview

Research area: Computer vision and computer graphics, specifically neural rendering and 3D facial avatar reconstruction (Gaussian Splatting, relightable neural models, texture de-lighting).

Technical level: Advanced.

Scope: The paper presents a pipeline that turns a small set of uncalibrated self-captured face images into a de-lit, textured triangulated surface mesh plus a view-dependent Gaussian neural texture, both usable in a standard graphics pipeline.

What This Paper Is About

Most high-fidelity face reconstruction needs light stages, many calibrated cameras, or long videos, and the resulting neural models (such as NeRFs) usually cannot be dropped directly into a conventional real-time graphics pipeline. This paper asks whether Gaussian Splatting, being explicit rather than implicit, can be constrained so that it produces both an accurate triangulated surface and an albedo texture separated from lighting, using only commodity hardware and a handful of images. The goal is a face asset that is identity-accurate, relightable, and immediately compatible with existing geometry, lighting, and renderer infrastructure.

Key Contributions

1. Two modifications to Gaussian Splatting for surface reconstruction. Soft constraints pull Gaussians tightly toward the underlying triangulated surface so that Gaussian perturbations can drive mesh deformation, and semantic segmentation annotations supervise the Gaussians so they do not explain image regions they should not be associated with.

2. A method for disentangling albedo texture from lighting and normals. PCA coefficients of a textured mesh (from a PCA representation of a mesh-based facial texture derived from the Metahuman dataset) are optimized to capture as much albedo color as possible, while the contribution of a relightable Gaussian model — which captures residual differences between the synthetic mesh rendering and the target image — is minimized.

3. Avoidance of controlled capture setups. The approach needs only commodity hardware and a limited number of views, and the flexibility of the system allows joint training on data from different capture setups, for example combining images from the paper's capture method with flashlight capture images.

4. A view-dependent neural texture built from Gaussians. The Gaussians are moved out of world space and into texture space, so high-fidelity Gaussian Splatting can be applied to any asset in a scene without modifying any other asset or any other aspect of the graphics pipeline (geometry, lighting, renderer, and so on).

Main Findings

• Reconstruction from 11 images: A neutral pose is reconstructed from only 11 images rather than requiring a long video. Eleven predefined target head poses are used, and for each pose the closest non-blurry frame is selected.

• Gaussian-to-triangle coupling works: Assigning exactly one Gaussian to each mesh triangle, with densification and pruning disabled, gives a one-to-one correspondence that makes geometry regularization and later surface reconstruction more straightforward. The decoupling of the mesh from Gaussian optimization lets the authors use mesh regularization and other mesh-based considerations without adversely affecting Gaussian Splatting.

• Soft constraints matter for geometry: Omitting soft constraints disconnects the Gaussians from their triangles and results in very spurious geometry, according to the ablation shown in the paper's Figure 2. Omitting segmentation supervision lets Gaussians incorrectly explain regions of the image their triangles should not be associated with, again producing spurious geometry, even though the reconstructed image still matches ground truth in both ablations.

• Eye regularization prevents a specific failure: Without the eye loss (L_eyes), the model matches the target image by letting eyeball Gaussians occlude eye socket Gaussians, which would yield a mesh with an inaccurately small eye socket. Adding the loss alleviates this.

• Geometry is accurate enough for a neural texture: Because the neural texture computes color by alpha-blending Gaussians along the normal direction instead of along the ray direction, and because pixels that overlap Gaussians but not the triangulated surface are not shaded at all, the approach places a premium on accurate geometry.

• Fine-tuning improves the neural texture: Since the Gaussians are used differently in texture space, they are retrained using their current parameters as a warm start. After fine-tuning, the model generates even cleaner details than standard Gaussian Splatting.

• A cheaper variant is possible: Replacing alpha-blending with a direct summation allows for purely 2D Gaussians in UV space, which is more computationally efficient.

• Mipmapping is easier in texture space: A mipmap can be created by reducing both the number of texels and the number of Gaussians, which is straightforward in texture space, in contrast to the difficulties of significantly reducing the number of Gaussians for level-of-detail representations in world space.

• Gaussian Splatting may be replaceable after asset creation: The authors observe that while Gaussian Splatting handles camera extrinsics well and enables accurate surface geometry, after asset creation it seems it can be replaced with a simpler neural or even spherical harmonic texture with no loss of efficacy.

• Quantitative metrics: The provided content does not report numerical error metrics, benchmark scores, or comparisons against named datasets; results are presented qualitatively through figures and ablations.

Methodology in Plain English

Capture. The subject is filmed with the rear camera of an iPhone 14 held in a fixed position, recording a 4K-resolution monocular video. The subject slowly rotates their head while holding a neutral expression, outdoors in shaded conditions to minimize harsh lighting and specular highlights. Each frame is center-cropped to 2160 × 2160 pixels.

Initialization. A pretrained landmark detection network finds facial and skull landmarks on each frame. Because the camera is fixed, the skull landmarks give head pose (rotation and translation) by comparison against a 3D canonical template — a rough estimate is sufficient here. Eleven predefined target head poses drive frame selection: for each pose the closest non-blurry frame is chosen, where closeness compares estimated rotation to target rotation and blurriness is measured by the variance of the Laplacian of grayscale pixel values. These two terms are combined with weights and minimized automatically. A front-facing image is passed through a depth estimator, and the image, depth, and landmarks go into the MetaHuman Animator to produce a triangulated surface. That mesh is topologically consistent with other MetaHumans but a poor fit to the subject, so it serves only as an initial guess. Triangles for the teeth, inner mouth, and eyelashes are removed, leaving a face mesh of 47,944 triangles.

Modified Gaussian Splatting. One Gaussian is attached to each mesh triangle, and densification and pruning are switched off during training. Several soft regularization terms encourage geometric features to vary smoothly across the mesh, using a Laplacian-like neighbor-averaging penalty. These are applied to the displacement between each Gaussian center and its triangle centroid, to the Gaussian's local normal, and to the distance from the Gaussian's outer boundary point to the triangle centroid. To make normals consistent across the mesh, rotation matrices are built from UV texture coordinates rather than from edge directions that vary significantly; U is retained as the consistent direction and V is orthogonalized to it. Eyeball Gaussians, which are disconnected from the face mesh, are regularized by linking bi-directional nearest-neighbor pairs between eyeball and face triangle centroids. Boundary points account for the combined contribution of a Gaussian and its k-nearest neighbors in UV space, rather than a single Gaussian.

Segmentation supervision. A segmentation model using the Mask2Former architecture is trained on synthetic data generated in texture space, since MetaHuman textures are already semantically labeled and labels are consistent across MetaHumans. Labels cover face, nose, nostril, top lip, bottom lip, eyes, ears, non-face, plus a background label that is expanded into the foreground to emphasize occlusion boundaries and give strong silhouette cues. Each Gaussian inherits a label from its triangle, and triangle labels come from a template texture that deliberately differs from the training labels — the face label is expanded into hair and neck regions because hairlines vary between people and forcing triangles to an incorrect hairline causes spurious forehead deformation, and the face/neck boundary is expanded because of non-linear deformation there. Non-face foreground pixels are ignored during training while background labels are retained.

Geometry refinement. Camera extrinsics and Gaussian parameters are optimized jointly, then extrinsics are frozen. Each iteration the Gaussian parameters are re-optimized to provide supervisory outer boundary points, and mesh vertices are perturbed to bring triangle centroids toward those boundary points, with regularization keeping each vertex's perturbation near the average of its edge-connected neighbors and penalizing changes in normal direction. Two geometry refinement iterations are performed: the first over a MetaHuman per-region PCA formulation, the second over individual vertex positions.

Texture de-lighting. A relightable Gaussian avatar model replaces the usual color with a diffuse term (albedo times an integral of environment lighting against a learned radiance transfer function, both represented in spherical harmonics) plus a view-dependent specular term with a learned surface normal and specular sharpness. Because de-lighting without a light stage is underconstrained, the problem is regularized by the PCA facial texture representation from the Metahuman dataset, optimizing PCA coefficients to explain albedo while minimizing reliance on the relightable Gaussians.

Texture space and asset conversion. World-space triangles are transformed to texture space using their existing UV coordinates, moving the Gaussians into a 3D UVW texture space the same way they move from canonical to world space, with W encoding the orthogonal distance to the parent triangle. A view-dependent neural texture is computed by splatting the Gaussians perpendicularly with an orthographic camera while still defining color as a function of the world space camera view. The final result is converted into a MetaHuman framework.

Why This Matters

Impact on research. The work shows a route to close the domain gap between synthetic and real images using neural rendering while still producing explicit, constrained geometry, rather than relying on advanced shading models to bridge that gap. It also demonstrates that Gaussian Splatting and NeRF-style representations can complement each other, and that an explicit surface can bridge neural rendering and classical graphics.

• Real-time face avatars: Virtual reality, video conferencing, gaming, and feature films all need face avatars that are identity-accurate, controllable, relightable, and efficient enough for real-time use.

• Democratized content creation: Because the method needs only commodity hardware and a limited number of views — not multiple calibrated cameras or a light stage — a user can capture a subject with a phone and obtain a usable asset.

• Existing game and film pipelines: The output triangulated surface with de-lit textures plugs into standard graphics pipelines without modifying geometry, lighting, renderer, or other assets.

• Level-of-detail and asset management: Because it is straightforward to build mipmaps by reducing texels and Gaussians together in texture space, neural textures may be more practical for scalable asset pipelines.

Industry relevance. The authors are affiliated with Epic Games as well as Stanford University, and the pipeline is explicitly designed around MetaHuman assets and the maturity of the standard graphics pipeline. The paper argues that less invasive approaches are likely to facilitate quicker industry adoption, noting that ray tracing was embraced in non-realtime content creation before reaching real-time applications, and that neural rendering methods will likely be treated similarly. The demonstration of use in a text-driven asset creation pipeline reinforces this framing.

Future Directions

• Confirming the replacement claim: Determining when a fine-tuned Gaussian neural texture can be swapped for a simpler neural or spherical harmonic texture without loss of efficacy, and what geometry accuracy that requires.

• Removing the need for the relightable model at render time: Since de-lighting is regularized by a PCA texture prior, an open question is how much that prior constrains results for subjects or lighting conditions outside the MetaHuman dataset distribution.

• Broadening capture flexibility: The system already supports joint training on disparate images with incompatible lighting, including flashlight capture; extending this to more varied capture setups and capture devices is a natural next step.

• Extending beyond neutral poses and the face region: The reconstruction targets a neutral pose and removes teeth, inner mouth, and eyelash triangles; extending the constrained-Gaussian formulation to expression-dependent deformation and to intra-oral and hair geometry remains open.

Target Audience

Researchers and practitioners in computer vision and computer graphics working on neural rendering, 3D face reconstruction, avatars, and relightable models; technical artists and engineers at game, film, and XR studios who need to integrate neural rendering output into a standard graphics pipeline; and readers already familiar with Gaussian Splatting, Gaussian Avatars, and spherical-harmonic lighting who want to see how explicit constraints can turn a neural representation into a production-usable mesh and texture.

Authors’ abstract

We leverage increasingly popular three-dimensional neural representations in order to construct a unified and consistent explanation of a collection of uncalibrated images of the human face. Our approach utilizes Gaussian Splatting, since it is more explicit and thus more amenable to constraints than NeRFs. We leverage segmentation annotations to align the semantic regions of the face, facilitating the reconstruction of a neutral pose from only 11 images (as opposed to requiring a long video). We soft constrain the Gaussians to an underlying triangulated surface in order to provide a more structured Gaussian Splat reconstruction, which in turn informs subsequent perturbations to increase the accuracy of the underlying triangulated surface. The resulting triangulated surface can then be used in a standard graphics pipeline. In addition, and perhaps most impactful, we show how accurate geometry enables the Gaussian Splats to be transformed into texture space where they can be treated as a view-dependent neural texture. This allows one to use high visual fidelity Gaussian Splatting on any asset in a scene without the need to modify any other asset or any other aspect (geometry, lighting, renderer, etc.) of the graphics pipeline. We utilize a relightable Gaussian model to disentangle texture from lighting in order to obtain a delit high-resolution albedo texture that is also readily usable in a standard graphics pipeline. The flexibility of our system allows for training with disparate images, even with incompatible lighting, facilitating robust regularization. Finally, we demonstrate the efficacy of our approach by illustrating its use in a text-driven asset creation pipeline.