G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding

AI summary

Overview

Research area: Egocentric (first-person) video understanding — specifically action recognition and action anticipation in wearable-camera footage, using graph-based scene representations and eye-gaze as a structural signal.

Technical level: Intermediate. The paper assumes familiarity with scene graphs, vision-language models, object grounding, and Transformer-based temporal aggregation, but the core idea (gaze decides which objects stay in the graph) is easy to grasp.

Scope in one sentence: The paper proposes G3Ego, a framework that builds action scene graphs from sparsely sampled first-person frames, prunes them with the camera wearer's gaze into compact representations, and temporally aggregates the resulting graph embeddings for action recognition and anticipation on MECCANO and EGTEA Gaze+.

What This Paper Is About

Most egocentric action models borrow large video models pretrained on third-person (exocentric) video datasets, then fine-tune them on first-person data. This is computationally and annotation-heavy, and it can be a poor fit because first-person actions are often determined by just a handful of hand–object interactions. G3Ego asks whether a compact, structured graph built from sparsely sampled images can do the job instead, and whether the wearer's gaze can serve not as an extra input feature but as a structural rule that decides which entities belong in the graph at all.

Key Contributions

1. A gaze-guided graph framework (G3Ego) that uses gaze as a structural pruning cue rather than an auxiliary modality or attention signal, producing compact action scene graphs centered on action-relevant entities.
2. An automatic, annotation-free pipeline combining vision-language captioning, dependency parsing, open-vocabulary object grounding, and single-forward-pass hand/object feature extraction to construct these graphs from video without dense scene-graph supervision.
3. A graph embedder plus temporal aggregation module that turns a sequence of gaze-pruned graphs into a single activity representation, evaluated on both action recognition and action anticipation.
4. Competitive results with substantially more compact representations than dedicated video-based models, including the highest reported Macro-F1 on MECCANO recognition under class-imbalanced evaluation.

Main Findings

• Gaze pruning helps over full graphs. On MECCANO with an LSTM and 10 frames, pruning improved Top-1 from 37.34 to 37.90, Top-5 from 70.39 to 70.88, and Macro-F1 from 8.61 to 10.63 compared with Full Graphs (FG). Gaze is used only for graph pruning, not as a recognition feature.
• Object features matter a lot. Adding object-region features to global frame features raised MECCANO Top-1 from 25.58 to 37.34 and Top-5 from 60.75 to 70.39 (Macro-F1 8.29 to 8.61).
• More sampled frames help. Within the same temporal window, going from 1 to 10 to 32 frames raised Top-1 from 31.85 to 39.50 and Macro-F1 from 4.72 to 12.34.
• The proposed temporal aggregation is the strongest aggregator. At 32 frames: MLP reached Top-1 35.28 / Macro-F1 7.71; GNN (two-layer, four-head GAT) reached 33.79 / 8.70; LSTM reached 39.50 / 12.34; the proposed temporal aggregation reached 41.91 / 15.87.
• Hand features give the best MECCANO recognition result. Adding the 20-dimensional hand/hand–object descriptor to G3Ego with temporal aggregation reached 46.48 Top-1, 82.04 Top-5, and 21.34 Macro-F1 (105M trainable parameters). Removing the global RGB features dropped performance sharply to 26.99 Top-1 and 3.83 Macro-F1.
• MECCANO recognition comparison. G3Ego's 21.34 Macro-F1 is the highest reported in Table 2. The best Top-1 in that table is UCF Swin3D-B at 52.82 (176M parameters, RGB+Depth); LUBECK UniFormer ensemble reached 51.82 Top-1 / 83.35 Top-5 (540M), UNIBZ 8-model ensemble 52.57 / 81.53 (194M), SlowFast with RGB 45.16 / 73.75 (68M), Swin-B + HOCL+OSL 44.81 / 77.01 / 16.70 (176M), and AAG 33.48 / 69.32 / 9.52 (24M). The majority-class baseline is 27.31 / 51.22 / 0.72.
• EGTEA Gaze+ recognition. G3Ego reached per-split mean accuracies of 61.68 (S1), 56.34 (S2), and 51.46 (S3), for the highest average mean accuracy at 56.49. Its average Top-1 of 64.25 is second only to GC-TSM (65.10) among methods in the table, which excludes approaches relying on exocentric pretraining.
• Action anticipation on MECCANO (δ = 1 s). G3Ego reached 25.20 Top-1, 61.67 Top-5, and 4.20 Macro-F1 with 15M trainable parameters. Its Macro-F1 is the highest in Table 4; several baselines have higher Top-1 (e.g. MMTF-RU 29.75, VLMAH 28.90, AVT 27.43), but the majority-class baseline alone achieves 27.21 Top-1, which the paper argues makes Top-1 uninformative on this long-tailed benchmark.
• Graphs get much smaller and better connected. On MECCANO, G3Ego reduced the average node count by 72.4% and edge count by 66.1% (13.57 nodes and 8.13 edges for FG versus 3.75 and 2.75 for G3Ego), while global efficiency rose from 0.257 to 0.771 and average maximum shortest-path distance from the camera wearer fell from 3.29 to 2.00. On EGTEA Gaze+, nodes fell from 4.51 to 3.77, edges from 3.41 to 2.77, average max distance from 2.03 to 2.00, and global efficiency rose from 0.703 to 0.769.
• MECCANO produces much larger graphs than EGTEA Gaze+, which the authors attribute to longer VLM captions mentioning many objects during toy-motorbike assembly, making pruning especially valuable there.
• Cached, lightweight downstream training. Frozen DINOv3 and GroundingDINO extraction plus graph construction are performed once and cached; the temporal model is then trained on the cached graphs. The temporal model consumes 0.435 GFLOPs (pruned) versus 0.47 GFLOPs (full) per graph sequence, while the frozen components are far heavier: Qwen3-VL-32B uses 27,500.8 GFLOPs and 63.91 GiB peak memory, DINOv3 ViT-L/16 uses 121.76 GFLOPs and 1,176.52 MiB, and GroundingDINO uses 1,312.93 GFLOPs and 2,160.50 MiB.

Methodology in Plain English

The pipeline runs in six stages, per frame, over a clip of sparsely sampled frames:

1. Caption each frame. A frozen vision-language model (Qwen3-VL-32B-Instruct) writes a single sentence describing the camera wearer's action, resized to 448 input and capped at 160 new tokens.
2. Parse the caption. spaCy dependency parsing pulls out the main verb, objects, attributes, and relations. Compound nouns are kept intact (e.g. "board game") while descriptive phrases (e.g. "white doors") are split into base object plus attribute.
3. Extract a global visual descriptor for the frame with a frozen DINOv3 ViT-L/16 encoder, producing a 1024-dimensional vector.
4. Ground objects and hands with an open-vocabulary detector (GroundingDINO with a Swin-T backbone) in a single forward pass, yielding 256-dimensional object features plus bounding boxes, and a 20-dimensional hand descriptor. The hand vector concatenates normalized hand bounding-box coordinates and a side indicator with the bounding box and class index of the interacted object; a hand is treated as interacting with an object when their boxes overlap, and missing detections are zeroed out. Where several instances of an object appear in a frame, features are mean-pooled. Attributes and relations are stored as multi-hot matrices.
5. Build and prune the graph. Each frame becomes a graph with nodes for the camera wearer, the main verb (initialized with the global frame descriptor), auxiliary verbs, and objects, with edges for verb–object interactions, auxiliary–main verb dependencies, and prepositional relations. Gaze then prunes it: the gazed object is the one whose bounding-box center is closest to the gaze coordinate, and the retained node set is the camera wearer, the main verb, the gazed object, and the action-critical objects directly connected to the main verb. Edges are restricted to the retained nodes.
6. Embed and aggregate. A trainable graph embedder maps each pruned graph to a fixed 64-dimensional vector using dictionary-based embeddings for verbs, objects, and relations, multi-query pooling for object features and auxiliary verbs, an MLP for triplets, and an optional projection of the global visual feature. A learnable positional embedding is added, a Transformer encoder with multi-head self-attention processes the sequence, temporal attention pooling produces a 352-dimensional activity representation, and a fully connected layer with softmax yields action probabilities.

Frozen components are never trained. Training uses Adam with a base learning rate of 3×10⁻⁴, weight decay 1×10⁻⁵, a linear scheduler with factor 0.95, inverse-frequency-weighted cross-entropy loss, batch size 16, 20 epochs, and best-checkpoint selection by Macro-F1, with 32 frames used for graph construction.

Why This Matters

Impact on research. The paper challenges the assumption that strong egocentric action models require large-scale exocentric video pretraining. By showing that frozen image-based encoders plus structured gaze-pruned graphs can be competitive — and can lead on Macro-F1 under class imbalance — it argues for evaluating egocentric models beyond Top-1, and for treating gaze as a mechanism that shapes representation rather than just another input channel.

Real-world applications (potential, based on the paper's framing):

• Wearable AR and smart glasses that interpret what the user is doing while keeping on-device computation small, which matters because the trainable temporal model costs only 0.435 GFLOPs per sequence on cached graphs.
• Industrial and assembly assistance — MECCANO is toy-motorbike assembly, so skill monitoring, step verification, and next-step prompting on a factory floor are natural fits.
• Kitchen and cooking assistants, the setting of EGTEA Gaze+, where step tracking and anticipation could support hands-free guidance.
• Assistive technologies and robotics learning from demonstration, where first-person recordings of manipulation could be turned into compact, interpretable interaction graphs rather than dense video features.

Industry relevance. The pipeline's cost profile is split between a one-time, cacheable feature-extraction stage using large frozen models and a lightweight trainable head, which is attractive for products that need fast inference after an offline indexing pass. The authors also credit support from the Dutch Research Council (NWO) VENI programme (project 244507) and University of Twente high-performance computing for the computational resources.

Future Directions

• Longer temporal dependencies. The conclusion lists extending G3Ego to model longer temporal context as future work, noting that anticipation currently uses a shorter effective temporal window (15M parameters versus 105M for recognition).
• Richer interaction modeling. The authors plan to incorporate richer hand–object and human–object interactions, which currently enter only as a 20-dimensional descriptor.
• More expressive graphs. Capturing higher-order semantic relationships in egocentric video is named as an open direction.
• Evaluation beyond accuracy. The paper argues for metrics that behave sensibly under long-tailed distributions, since the MECCANO majority-class baseline already reaches 27.21 Top-1 in anticipation.
• Questions left open by the paper. Several ablations are deferred to the supplementary material rather than resolved in the main text: the VLM semantic prior, random pruning as a control, and explicit gaze as an input. Table 6 also reports 105M parameters for both pruned and full TempAgg variants while the anticipation model is stated as 15M in the main text, leaving the exact parameter accounting across configurations not fully reconciled in the main paper.

Target Audience

Researchers and graduate students in computer vision and wearable computing working on egocentric video, action recognition, and anticipation. It is also relevant to scene-graph and graph-learning researchers interested in structured video representation, to practitioners building smart-glasses or assistive-robotics systems who care about parameter and FLOP budgets, and to anyone studying how gaze can be used as a structural signal for multimodal perception. Readers without background in vision-language models or grounding detectors will find the pipeline conceptually clear but will need to consult cited work for component details.

Authors’ abstract

Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person actions depend on a small number of hand-object interactions involving only a few relevant entities. We propose G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene. From sparsely sampled frames, G3Ego constructs action scene graphs from vision-language descriptions, grounded objects, and hand cues, and then prunes irrelevant entities using the camera wearer's gaze. The resulting graph embeddings are temporally aggregated for action recognition and anticipation. Unlike prior work that uses gaze primarily as an auxiliary modality or attention signal, G3Ego incorporates gaze directly into graph construction, producing efficient and interpretable representations focused on action-relevant interactions. Experiments on EGTEA Gaze+ and MECCANO show that G3Ego achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining. These results demonstrate the effectiveness of gaze-guided graph representations for egocentric action understanding.