RTGaze: Real-Time 3D-Aware Gaze Redirection from a Single Image

AI summary

Overview

Research area: Computer vision, specifically 3D-aware facial image generation and gaze redirection.

Technical level: Advanced. The paper builds on neural radiance fields (NeRFs), triplane 3D representations, cross-attention conditioning, and knowledge distillation from pretrained 3D generative models.

Scope: The paper presents RTGaze, a feedforward system that redirects the gaze of a single face image to a target pitch/yaw direction while preserving identity and 3D consistency, and does so in real time (61 ms per image on a single NVIDIA 3090 GPU).

What This Paper Is About

Gaze redirection takes a photo of a face and re-renders it as if the person were looking in a different direction, without changing who they are. Existing 2D methods warp or regenerate pixels and lose 3D consistency under large head poses, while existing 3D methods such as GazeNeRF produce better geometry but require a slow GAN inversion step (roughly one minute at inference). RTGaze's goal is to combine the 3D consistency of NeRF-based approaches with the speed of a single feedforward pass, using only one input image and a gaze direction.

Key Contributions

1. A real-time 3D-aware gaze redirection model that is reported to surpass state-of-the-art methods in inference speed, redirection accuracy, and image quality across multiple datasets, processing each image in 61 ms.
2. A gaze-controllable facial representation module that uses two distinct encoders (one for high-frequency, one for low-frequency features) plus a gaze injection module that merges the gaze prompt into the high-frequency features via cross-attention.
3. Distillation of 3D face priors from a pretrained 3D portrait generation network into the gaze redirection model, applied as an L1 distillation loss on geometric depth images computed from the teacher and student models.
4. A mask-guided 2D reconstruction objective that separates face-region and eye-region losses, with a larger coefficient on the eye region to improve eye quality.

Main Findings

• Inference speed: RTGaze completes encoding in 0.026 s and rendering in 0.035 s, for a total of 0.061 s (61 ms) per image, measured as an average over 100 samples on a single NVIDIA 3090 GPU. The abstract states this is 800× faster than the previous state-of-the-art 3D-aware methods.
• Comparison speed: HeadNeRF takes 60 s encoding, 0.058 s rendering, 60.058 s total; GazeNeRF takes 60 s encoding, 0.060 s rendering, 60.060 s total. Both require an inversion process to update parameters from the input image.
• Image quality on ETH-XGaze (Table 1): RTGaze achieves FID 38.346, PSNR 19.007, LPIPS 0.262, SSIM 0.715. GazeNeRF achieves FID 81.816, PSNR 15.453, LPIPS 0.291, SSIM 0.733; HeadNeRF achieves FID 69.487, PSNR 15.298, LPIPS 0.294, SSIM 0.720; ST-ED achieves FID 115.020, PSNR 17.530, LPIPS 0.300, SSIM 0.726. The paper notes RTGaze beats other methods on most metrics and is comparable on SSIM.
• Redirection accuracy and identity across three datasets (Table 2): On ColumbiaGaze, RTGaze reports LPIPS 0.249, ID 61.765, Gaze error 7.625, Head error 3.326, versus GazeNeRF's 0.352 / 23.157 / 9.464 / 3.811. On MPIIFaceGaze, RTGaze reports 0.251 / 46.098 / 9.409 / 6.444, versus GazeNeRF's 0.272 / 30.981 / 14.933 / 7.118. On ETH-XGaze, RTGaze reports 0.262 / 60.708 / 9.047 / 3.631, versus GazeNeRF's 0.291 / 45.207 / 6.944 / 3.470 — so GazeNeRF has a lower gaze error on ETH-XGaze while RTGaze leads on LPIPS and ID.
• Gaze injection ablation (Table 3): Injecting the gaze prompt into the low-frequency feature yields FID 67.298, ID 38.517, Gaze 18.973, Head 5.409, versus FID 38.346, ID 60.708, Gaze 9.047, Head 3.631 when injecting into the high-frequency feature. Injecting into both feature types is reported to fail to converge.
• Loss ablation (Table 4): Reconstruction loss alone gives FID 101.053, ID 47.251, Gaze 9.332, Head 4.208. Adding perceptual loss gives FID 54.682, ID 52.518, Gaze 10.911, Head 3.700. Adding 3D face prior distillation gives FID 38.346, ID 60.708, Gaze 9.047, Head 3.631.
• Qualitative observations: ST-ED is reported to struggle with identity preservation and to incorrectly retain the unmasked green background; HeadNeRF and GazeNeRF lose facial details such as texture and fine-grained hair; results without 3D prior distillation show shape distortions.

Methodology in Plain English

RTGaze takes three inputs during training: a source image, a gaze prompt, and a frontal image used only to obtain the 3D prior. At inference it needs only a single 2D portrait and a target gaze direction.

The pipeline works in three stages. First, a hybrid encoder splits the work: a CNN extracts high-frequency features for fine appearance detail, while a DeepLabV3 network pretrained on ImageNet followed by a vision transformer encoder extracts low-frequency global geometry. The gaze prompt is a two-dimensional vector of pitch and yaw angles; it is embedded with an MLP and injected into the high-frequency features through a cross-attention layer, where the high-frequency feature is the query and the gaze embedding is both key and value. The injected features are then fused with the low-frequency features to form the final gaze-controllable representation.

Second, this representation is passed to a decoder that produces a triplane 3D representation. Third, neural rendering uses that triplane and a target camera pose to synthesize the redirected image.

For the 3D prior, the authors feed a frontal image with the same identity and gaze as the target into a pretrained 3D portrait generation model, which produces its own triplane and depth map. They compute depth via volume-rendering-style accumulation along camera rays and apply an L1 loss between the teacher depth and the student depth. The authors note that only depth is distilled, not the synthesized image, because the pretrained model's images do not always maintain appearance consistency with the targets.

Training combines three losses: a mask-guided reconstruction loss split into face and eye regions, the depth distillation loss, and a VGG16-based perceptual loss. The model is trained end-to-end with AdamW, learning rates of 1e-5 for the encoding part and 1e-5 for the rendering part, batch size 4, for 50 epochs. Loss coefficients for the reconstruction, distillation, and perceptual terms are set to 1, 1, and 0.8; the face and eye reconstruction coefficients are 1 and 2. Images are normalized and resized to 512x512, camera poses are processed following EG3D, and face/eye masks come from face parsing models. Training takes around 18 hours on two NVIDIA A100 GPUs with 40GB memory.

Why This Matters

Impact on research: The main tension in 3D-aware gaze redirection has been that NeRF-based models give strong 3D consistency but need slow per-image optimization (here, roughly one minute for both HeadNeRF and GazeNeRF). RTGaze removes the inversion step entirely by learning a direct feedforward mapping from image plus gaze prompt to a triplane representation, and by distilling geometric priors from a larger model into a lightweight one instead of paying the cost at inference.

Real-world applications:

• Virtual reality and telepresence, where avatar eye contact must be updated in real time.
• Digital humans and conversational agents, where gaze must respond to a user interactively.
• CG film-making and content production, where changing an actor's gaze in post-production currently requires manual work.
• Video conferencing and data augmentation for gaze estimation training sets.

Industry relevance: A 61 ms per-image runtime on a consumer-grade NVIDIA 3090 GPU matters because it puts 3D-aware gaze correction within the frame budget of interactive applications, rather than requiring offline processing or multi-GPU server infrastructure. The authors note the method holds potential specifically for scenarios with high real-time requirements.

Future Directions

• Generalization to video and temporal consistency, since the paper evaluates only on still images with no temporal coherence evaluation.
• Handling the convergence failure observed when gaze is injected into both high- and low-frequency features — understanding and fixing this would allow appearance and geometry to be controlled simultaneously.
• Testing on uncontrolled, in-the-wild inputs, since evaluation uses ETH-XGaze, ColumbiaGaze, and MPIIFaceGaze, which are collected under controlled multi-camera or fixed-pose setups.
• Scaling to higher resolutions and more expressive gaze ranges, since the paper reports generation across a large range of head poses and gaze directions but trains at 512x512 and does not report an upper bound on gaze angle.

Target Audience

Researchers and engineers working on neural rendering, 3D-aware face generation, and gaze redirection, as well as practitioners building real-time avatar, VR/AR, or telepresence systems. Readers need background in NeRFs, triplane representations, and attention mechanisms to follow the method section in detail, though the results tables and ablations are readable by anyone familiar with standard image quality metrics (SSIM, PSNR, LPIPS, FID) and gaze/head angular error.

Authors’ abstract

Gaze redirection methods aim to generate realistic human face images with controllable eye movement. However, recent methods often struggle with 3D consistency, efficiency, or quality, limiting their practical applications. In this work, we propose RTGaze, a real-time and high-quality gaze redirection method. Our approach learns a gaze-controllable facial representation from face images and gaze prompts, then decodes this representation via neural rendering for gaze redirection. Additionally, we distill face geometric priors from a pretrained 3D portrait generator to enhance generation quality. We evaluate RTGaze both qualitatively and quantitatively, demonstrating state-of-the-art performance in efficiency, redirection accuracy, and image quality across multiple datasets. Our system achieves real-time, 3D-aware gaze redirection with a feedforward network (~0.06 sec/image), making it 800x faster than the previous state-of-the-art 3D-aware methods.