EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision

AI summary

Overview

Research area: Egocentric computer vision — specifically visual query localization (VQL) over the Ego4D-VQ benchmark, spanning both 2D temporal localization (VQ2D) and 3D spatial localization (VQ3D).

Technical level: Advanced. The paper combines meta-learning with closed-form/iterative optimization of convolutional filters, discriminative correlation filters, segmentation foundation models, and a geometry-grounded Transformer for camera pose and depth.

Scope: The paper proposes a dual-branch, memory-driven framework that unifies 2D video query localization and 3D spatial query localization in egocentric video, and reports state-of-the-art results on the Ego4D-VQ benchmark.

What This Paper Is About

Visual query localization asks a system to find where and when a specific object appears in a first-person video, given only a single cropped image of that object as a query. Egocentric footage is hostile to this task: the camera moves violently, objects blur, and a target's appearance and scale shift constantly, so a static one-shot query often fails to match the target later in the video. EAGLE's goal is to replace the conventional "detect-then-track" pipeline with two mutually reinforcing branches — a segmentation branch that acts as a precise identifier and a tracking branch that acts as a robust navigator — each backed by an online memory bank that accumulates high-confidence observations of the target over time, and then to extend the 2D result into 3D using a feedforward geometry model instead of fragile structure-from-motion.

Key Contributions

1. A dual-branch framework for egocentric VQL that synergizes segmentation (pixel-level appearance cues) with tracking (geometric-temporal constraints), explicitly framed as overcoming the limitations of traditional "detect-then-track" paradigms where bounding-box detectors contaminate the target's appearance with background.

2. An online episodic memory consolidation mechanism for both branches, inspired by avian memory consolidation. High-confidence observations are selectively written into a structured appearance memory bank (AMM) and a geometry memory bank (GLM), supporting long-term adaptation to target variation while suppressing distractor interference.

3. An efficient unified 2D-to-3D localization pipeline that back-projects refined 2D VQL outputs into 3D space using VGGT for camera pose and depth, rather than the SfM/COLMAP approach used by prior VQL-3D methods.

4. State-of-the-art results on the Ego4D-VQ benchmark for both VQ2D and VQ3D, with reported gains in accuracy, robustness, and efficiency, plus ablation studies isolating the contribution of each memory component.

Main Findings

• VQ2D test-set results: EAGLE reaches 0.46 tAP₂₅, 0.40 stAP₂₅, 53.51% Recall, and 62.70% Success. The previous best listed, RELOCATE, scores 0.43 tAP₂₅, 0.35 stAP₂₅, 50.60%, and 60.10%. The paper reports this as surpassing RELOCATE by 6.9% in tAP₂₅, 14.3% in stAP₂₅, 5.8% in Recall, and 4.3% in Success.

• VQ2D validation-set results: EAGLE reaches 0.47 tAP₂₅, 0.42 stAP₂₅, 52.09% Recall, 61.29% Success, versus RELOCATE's 0.41, 0.33, 50.50%, and 58.03%. Reported gains are 14.6%, 27.3%, 3.1%, and 5.6% in the same four metrics, with comparable inference speed.

• VQ3D test-set results: EAGLE records 89.02% Success, 96.14% Success*, 1.84 L2 error, 1.21 Angle error, and 92.42% QwP. Against EgoLoc-v1 (88.64, 96.15, 1.86, 1.24, 92.05), the paper reports Succ and QwP improvements of 0.4% and L2 and Angle error reductions of 1.1% and 2.4%.

• VQ3D validation-set results: EAGLE records 84.77% Success, 98.54% Success*, 1.18 L2, 0.42 Angle, and 85.68% QwP, versus EgoLoc-v1's 81.13, 98.10, 1.45, 0.55, and 84.73. Reported gains are 4.5% in Success, 0.4% in Success*, and 1.12% in QwP, with L2 and Angle errors reduced by 18.6% and 23.6%.

• The appearance memory bank matters most: Removing O_AMM entirely causes the largest ablation drop — 11.9% in tAP₂₅, 40% in stAP₂₅, 23.4% in Recall, and 12.2% in Success. The second-largest degradation comes from replacing SAM with a pre-trained STA model during pseudo-mask generation, which the authors attribute to SAM producing higher-quality, more discriminative pseudo-labels.

• The geometry memory bank is crucial for long-term tracking: Removing O_GLM yields decreases of 10.6% in tAP₂₅, 29.8% in stAP₂₅, 18.7% in Recall, and 13.4% in Success.

• The two memories need opposite update policies: The optimal update strategy for GLM is the inverse of AMM's. Allowing GLM's initial query to be replaced (a FIFO update) degrades performance, because the segmentation branch operates at pixel level and must track the latest appearance, while the discriminative tracking branch operates on region-level geometric descriptors and benefits from keeping the initial query as a fixed identity anchor against occlusion and appearance drift.

• All AMM components contribute: Ablating the pseudo-label modulator network, the target re-weighting network, or allowing the initial query to be replaced within O_AMM each degraded performance relative to the full configuration (0.47 tAP₂₅, 0.42 stAP₂₅, 52.09% Recall, 61.29% Success).

• VGGT accelerates 3D geometry: The pipeline reduces processing time from the several minutes or even hours required by COLMAP to only a few seconds, while delivering geometry estimations of comparable accuracy.

Methodology in Plain English

EAGLE runs two parallel branches on the same video clips, built on a shared frozen backbone (a pre-trained ViT fine-tuned on EgoTracks).

The identifier (segmentation branch). Starting from an initial mask produced by SAM on the query crop, a small trainable convolutional network turns that binary mask into a richer multi-channel "pseudo-label" encoding boundaries and centers. A second, similarly structured network re-weights the loss so learning focuses on important target regions. The branch's convolutional parameters are then optimized by a meta-learner, which solves a weighted squared-error objective using steepest descent iterations with an adaptively computed step size. The branch keeps an appearance memory bank: whenever a retrieved frame's predicted mask has mean confidence of at least 0.6, that mask is cropped and added to the bank, and a few more optimization iterations are run from the current estimate. This is the "memory consolidation" idea — the model keeps re-shaping itself around the target's evolving appearance.

The navigator (tracking branch). A discriminative correlation filter learns a convolutional kernel over a geometry memory bank that holds both the original static query and dynamically updated high-confidence observations (scaled by a factor of 1.5 and evicted under a FIFO policy). The training loss blends least-squares and hinge behavior through a spatially weighted residual, so positions near the target center are fitted exactly while background positions only incur a penalty if they exceed zero. The filter update step size is derived by a quadratic approximation of the loss and, in practice, a Gauss-Newton approximation using the Jacobian. During online inference, if the score map fails to produce sufficiently high responses — measured as exceeding 60% of the historical frame length — the filter is updated from the original static snapshot; otherwise the most recent dynamic snapshot is used.

Fusing the two. The tracking branch's score map passes through a conv-BN-ReLU block to match the dimensionality of the mask encoding, the two are added element-wise, and a decoder produces the final segmentation score map. That mask yields both the segmentation output and the bounding-box trajectory used for evaluation.

Going to 3D. The 2D output, camera pose, and depth are fed into VGGT (the 1B version), which infers per-frame camera parameters, a dense depth map, and a per-pixel depth uncertainty map in one feedforward pass. Because VGGT's coordinate system is self-consistent but arbitrary, a 7-DoF similarity transformation is solved by least squares to align predicted point clouds with the ground-truth Matterport scan coordinate system. Each frame's mask center is then back-projected into 3D via the pinhole camera model. Because single-view estimates are fragile, results are aggregated across views with a fused weight that multiplies a semantic confidence (a 1/3-1/3-1/3 combination of average mask probability, average probability above a threshold, and maximum probability) by a geometric confidence derived as the exponential of the negative scaled VGGT depth uncertainty. The aggregated world coordinate is transformed back into each observing camera's frame to produce the final 3D displacement vector.

Training details. Frames are uniformly scaled to 448×448 by cropping along the longer side and zero-padding, then filtered with a Laplacian operator with window size 100. The segmentation branch is trained on Ego4D, EgoTracks, and VISOR; for the first two, SAM-generated masks serve as ground truth since only bounding boxes are annotated. The meta-learner uses 10 initial iterations and 3 update iterations; memory bank size is 50 for both banks. Optimization uses AdamW for 25,000 iterations with a peak learning rate of 0.0025, weight decay 0.05, and a linear schedule with 2,500 warm-up iterations, on GTX4090 GPUs. During inference, both memory banks are updated continuously for the first 100 historical frames and then every 25 frames; mask confidence serves as the temporal score, filtered with a 5-frame median filter, with the final interval thresholded at 0.8 × max(s_conf).

Why This Matters

Impact on research. The paper pushes against two entrenched assumptions in VQL: that detection with bounding boxes is the right front end, and that structure-from-motion is the necessary route to 3D. It shows that pixel-level masks plus a second, geometry-oriented memory can outperform a "detect-then-track" stack, and that a feedforward geometry foundation model can replace an SfM pipeline that the authors describe as fragile in texture-less or high-motion scenarios. The ablation finding that the two memories require opposite update policies — synchronized for pixel-level appearance, anchored for region-level identity — is a concrete design insight for anyone building memory-based trackers.

Real-world applications.

• Augmented and virtual reality: persistent spatial anchoring of a specific real object in a headset's field of view, which the paper names as a motivating application alongside embodied AI.
• Embodied AI and robotics: a robot retrieving a specific previously-seen object ("the last time I saw that cup") in a real environment, using the 3D displacement output to navigate to it.
• Episodic memory assistants: lifelogging and assistive systems for people with memory impairments, querying a personal video history with a single snapshot of an object.
• Video editing and search: locating the final occurrence of an object in long first-person footage for retrieval or post-production.

Industry relevance. The efficiency argument is direct: replacing COLMAP's minutes-to-hours of processing with VGGT's few seconds is the difference between an offline research pipeline and something deployable. Smart-glasses and AR platforms need temporal and spatial object localization to run near-interactively on wearable hardware, and the paper explicitly frames the unified 2D-3D pipeline as a step toward more realistic embodied episodic memory retrieval.

Future Directions

• Scaling and generalizing the memory banks. The paper fixes memory bank size at 50 with a simple confidence threshold of 0.6 and a FIFO policy for dynamic entries. Whether adaptive bank sizing, learned eviction, or confidence thresholds tuned per scene would help is not explored.

• Robustness beyond Ego4D. All quantitative results come from the Ego4D-VQ benchmark — which the paper calls the unique publicly available benchmark for VQL. Whether the memory consolidation mechanism transfers to other egocentric datasets, or to non-egocentric video, is an open question.

• Improving the 2D-to-3D coupling. The multi-view aggregation rests on the stated assumption that the target's 3D location remains relatively stable within a short observation window, which the authors call reasonable for most human-object interaction scenarios in Ego4D. Targets that move substantially within that window are not addressed. The paper also notes it does not predict the size or rotation of the 3D bounding box — ground-truth size and rotation are used only for visualization, with the predicted 3D location treated as the box center — so full 3D box estimation remains unsolved here.

• Understanding the AMM/GLM update asymmetry. The authors explain the opposite update policies through feature granularity, but this is an interpretation rather than a measured mechanism. A more principled account of when to anchor versus when to update could inform other memory-based architectures.

Target Audience

Researchers and graduate students in computer vision working on egocentric video understanding, visual object tracking, few-shot segmentation, or embodied AI, particularly those already familiar with the Ego4D benchmark or the detect-then-track VQL literature. It is also relevant to practitioners building AR/VR or robotics systems that need object-level spatial memory, and to engineers interested in substituting learned geometric foundation models for classical structure-from-motion in a production pipeline. Readers need comfort with meta-learning, correlation filters, and camera geometry to follow the method section in full.

Authors’ abstract

Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-ofthe-art performance on the Ego4D-VQ benchmark.