OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild

AI summary

Overview

• Research area: Computer vision — 3D appearance-based gaze estimation, semi-supervised learning, and cross-domain generalization.
• Technical level: Intermediate to Advanced (assumes familiarity with pseudo-labeling, vision transformers, and multimodal encoders such as CLIP).
• Scope: The paper introduces OmniGaze, a semi-supervised framework that uses roughly 1.4 million unlabeled face images plus a reward model to make 3D gaze estimation generalize to unseen, in-the-wild conditions.

What This Paper Is About

3D gaze estimation models are typically trained on a handful of well-annotated datasets, so they degrade badly when deployed on faces with different appearance, lighting, head pose, or camera setup. The authors address this by pairing limited labeled gaze data with large-scale unlabeled face images collected from six public sources, and by training a reward model that decides which automatically generated pseudo-labels are trustworthy enough to learn from. The goal is a gaze estimator that works reliably outside the lab without requiring new manual annotation.

Key Contributions

1. A semi-supervised framework for 3D gaze estimation (OmniGaze) built on a three-phase teacher–student pipeline: train a teacher on labeled data, generate pseudo-labels for unlabeled faces, then train a student on both sets with reliability weighting.
2. A reward model that scores pseudo-label reliability using multimodal cues, combining CLIP visual embeddings, scene-specific gaze descriptions generated by prompting a Multimodal Large Language Model, and the geometric 3D gaze direction vector — plus a secondary scorer that fuses the student's own prediction similarity.
3. A curated large-scale unlabeled corpus of roughly 1.4 million face images drawn from CelebA, VGGFace2, FaceSynthetics, SFHQ-T2I, VFHQ, and WebFace, deliberately spanning varied poses, lighting, occlusion, and demographics.
4. Demonstrated scalability as a data engine, achieving state-of-the-art results on five benchmarks under in-domain and cross-domain settings, and strong zero-shot generalization on four unseen datasets while adding no inference-time cost (the reward model is discarded after training).

Main Findings

• In-domain accuracy improves substantially over prior art: OmniGaze reaches 2.97° angular error on MPIIFaceGaze, 4.07° on EyeDiap, 5.40° on RT-Gene, 9.12° on Gaze360, and 6.72° on IVGaze — reductions of 0.64°, 0.71°, and 1.15° on the first three compared with the strongest baselines.
• Cross-domain generalization is consistent: When trained on ETH-XGaze or Gaze360 and tested on MPIIFaceGaze and EyeDiap, OmniGaze achieves 5.07°, 4.84°, 4.95°, and 5.75°, beating domain-generalization methods such as AGG, CLIP-Gaze, and LG-Gaze.
• Zero-shot transfer to unseen datasets is strong: Without fine-tuning, the ViT-L variant reaches 3.03° on MPIIFaceGaze, 4.15° on EyeDiap, 9.01° on RT-Gene, and 10.43° on IVGaze — notably outperforming the in-domain-trained FullFace (4.93°) on MPIIFaceGaze.
• Scale matters, and so does selection: In ablation, adding unlabeled data alone improves MPIIFaceGaze from 6.17° to 4.97°; adding reward-based pseudo-label selection pushes it to 3.44°, showing the two mechanisms are complementary.
• Both filtering and reweighting are needed: Filtering low-confidence pseudo-labels alone yields 4.18° on MPIIFaceGaze, reweighting alone yields 3.84°, and combining them yields 3.44°.
• Multimodal reward cues each contribute: Semantic gaze descriptions alone improve the reward baseline from 4.52° to 3.69°; 3D direction vectors alone give 4.03°; using both gives the best result of 3.44°.
• Gains are not a byproduct of backbone size: With a lightweight ResNet-18 backbone, OmniGaze attains 3.46°/4.37°/5.89° on MPIIFaceGaze/EyeDiap/RT-Gene, versus 4.48°/5.56°/7.45° for the same-backbone baseline and 4.00° for 3DGazeNet.

Methodology in Plain English

The authors start by assembling a large pool of face images that have no gaze labels, deliberately choosing sources that differ widely in who appears, where the photo was taken, and how the face is oriented. They then train an ordinary gaze estimator on the labeled datasets available; this "teacher" model is used to guess gaze directions for every unlabeled image, producing pseudo-labels.

The problem is that many of these guesses are wrong, and unlike classification, gaze labels are continuous angles, so the usual trick of dropping anything below a confidence threshold does not directly apply. Their solution is a separate reward model that looks at three things at once: the image itself (encoded with CLIP), a short natural-language description of where the person appears to be looking (produced by asking a multimodal language model a fixed question about the image), and the pseudo-label converted from yaw/pitch angles into a 3D unit vector. The reward model combines these through cross-attention and outputs a confidence score between 0 and 1, which is further refined using how closely the student model agrees with the pseudo-label.

Pseudo-labels scoring below 0.5 are dropped; the rest are kept and their loss contributions are scaled by their confidence. The reward model is trained alongside the student by treating real ground-truth labels as perfectly reliable examples and pseudo-labels as potentially unreliable ones, so it learns to separate the two. Every ten epochs, the teacher is refreshed with the student's weights to regenerate pseudo-labels, closing a feedback loop that gradually improves both the labels and the estimator. At deployment, the reward model is thrown away, so the final network is exactly the same size as a standard gaze estimator.

Why This Matters

This work shows that the chronic data bottleneck in gaze estimation can be partially sidestepped by mining the vast supply of unlabeled face images already available, provided there is a principled way to separate good pseudo-labels from bad ones. It also demonstrates a transferable idea: using a language model's scene understanding as a supervisory signal for a geometric regression task is a genuinely novel use of multimodal models.

Real-world applications:

• Virtual and augmented reality — reliable gaze tracking across diverse users and lighting conditions enables foveated rendering and natural avatar interaction.
• Driver monitoring systems — detecting distraction or drowsiness requires gaze estimation that holds up under changing illumination, head pose, and occlusion inside a vehicle cabin.
• Human-computer interaction and accessibility — eye-driven interfaces for users with motor impairments depend on gaze tracking that does not fail when the user turns their head or wears glasses.
• Medical and clinical screening — gaze patterns are diagnostic cues for conditions such as autism spectrum disorder, and robust estimation supports remote or at-home assessment.

Industry relevance: Companies building AR/VR headsets, automotive safety systems, and assistive technology all need gaze models that survive domain shift, and the paper's approach adds no inference-time overhead — the reward model is only used during training, making deployment straightforward. The framing of OmniGaze as a scalable "data engine" that can auto-annotate new face images is directly relevant to teams that cannot afford large-scale manual gaze labeling.

Future Directions

• Closing the remaining gap on complex benchmarks: Zero-shot errors on RT-Gene and IVGaze (roughly 9–10°) remain much higher than on MPIIFaceGaze and EyeDiap, suggesting that extreme poses and wide-field-of-view captures are still underserved.
• Extending the reward paradigm to other regression tasks: The pseudo-label reliability problem is general to continuous outputs — head pose, hand pose, or depth estimation could plausibly reuse the same multimodal reward design.
• Scaling and refining the unlabeled corpus further: The paper shows gains from 1.4M unlabeled images; whether performance continues to improve with 10M or 100M, and what diversity actually drives gains, remains open.
• Improving the reward model itself: Replacing CLIP and a single prompted MLLM with stronger or more specialized vision-language models, or learning the reward jointly with richer temporal cues, could sharpen pseudo-label selection.

Target Audience

Researchers and graduate students in computer vision working on gaze estimation, domain generalization, or semi-supervised learning will find the core contribution most valuable. Practitioners building gaze-dependent products (AR/VR, automotive, HCI) benefit from the practical recipe and the demonstration that no architectural changes or added inference cost are required. Those interested in multimodal learning will find the reward-model design — using MLLM-generated textual gaze descriptions to supervise a geometric regression task — the most transferable idea.

Authors’ abstract

Current 3D gaze estimation methods struggle to generalize across diverse data domains, primarily due to i) the scarcity of annotated datasets, and ii) the insufficient diversity of labeled data. In this work, we present OmniGaze, a semi-supervised framework for 3D gaze estimation, which utilizes large-scale unlabeled data collected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse collection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning a broader distribution, OmniGaze adopts a standard pseudo-labeling strategy and devises a reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by an off-the-shelf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized to select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OmniGaze achieves state-of-the-art performance on five datasets under both in-domain and cross-domain settings. Furthermore, we also evaluate the efficacy of OmniGaze as a scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.