GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing

AI summary

Overview

• Research area: Computer vision — generic object tracking (GOT), with a cross-over into 3D geometric perception (VGGT-style geometry transformers) and knowledge/model editing (null-space constrained editing).
• Technical level: Advanced. The method builds on transformer-based track-by-detection trackers, singular value decomposition, null-space projection, and ridge-regularized correlation matrices.
• Scope (one sentence): The paper proposes GOT-Edit, an online model-editing scheme that injects 3D geometric cues inferred from ordinary 2D video frames into a 2D semantic tracker while preserving the tracker's semantic discrimination ability.
• Paper details: arXiv:2602.08550v3 [cs.CV], 24 Feb 2026; authors Shih-Fang Chen and Yen-Yu Lin (National Yang Ming Chiao Tung University), Jun-Cheng Chen (Academia Sinica), I-Hong Jhuo (Microsoft AI). License CC BY-NC-SA 4.0. Project page: https://chenshihfang.github.io/GOT-EDIT

What This Paper Is About

Generic object tracking asks a system to follow an arbitrary object specified by a bounding box in the first frame of a video. Almost all modern trackers learn from 2D data only, so they struggle when the target is partially occluded, surrounded by similar-looking distractors, or heavily deformed. GOT-Edit's goal is to give such a tracker a sense of 3D geometry — object boundaries and scene structure — using only the 2D frames it already receives, without any RGB-D sensor, depth maps, or point clouds.

Key Contributions

1. Geometry-aware tracking without extra 3D input. The authors integrate semantic and geometric knowledge into generic object tracking using only 2D video streams, deriving geometric cues from a pre-trained Visual Geometry Grounded Transformer (VGGT) rather than from depth cameras or point clouds.
2. Online model editing with a null-space constraint. Inspired by the offline null-space editing of AlphaEdit, they design an online variant that projects geometry-induced weight perturbations into the null space of the semantic features, so new geometric knowledge is added without degrading the dominant semantic features.
3. A dual-predictor architecture. Built on ToMP, the tracker runs two model predictors — a semantic one and a geometry one — that generate localization-head weights in parallel; the two predictors share architecture and weights but have separate lightweight convolutional heads for semantic-weight prediction and perturbation-weight prediction.
4. Extensive benchmark validation. Experiments across AVisT, NfS, OTB, GOT-10k, LaSOT, TrackingNet, VOT2020, and VOT2022 show improved robustness and accuracy, especially under occlusion and clutter.

Main Findings

• Improved results over the DINOv2-based baseline: GOT-Edit-378 (DINOv2-L semantic backbone, VGGT geometry backbone, Base+VastTrack training) reaches SUC 64.5 on AVisT, 71.1 on NfS, 75.0 on OTB, AO 80.2 / SR75 79.8 / NPr 84.8 / Pr 82.9 on GOT-10k, SUC 75.0 and NPr 91.0 on LaSOT, and SUC 86.7 on TrackingNet, compared with ToMP-378 (reproduced) at 62.0, 69.0, 71.5, 77.5, 75.8, 83.7, 80.8, 72.7, 89.0, and 84.2 respectively. The paper reports a gain of about 2–3% across datasets relative to ToMP-378.
• Strong VOT robustness: On the VOT challenge, GOT-Edit obtains Robustness 89.8 on VOT-STb2022 and 90.3 on VOT-ST2020, versus 87.3 (VOT-STb2022) for PiVOT, 85.9 / 85.5 for MixFormerL, 86.7 for OSTrackSTB, 84.9 for TransT_M, and 81.8 / 78.9 for ToMP.
• Best OP50 under a shared semantic backbone: With DINO features held constant, GOT-Edit-378_Vast scores OP50 74.4 on AVisT, 89.3 on NfS, and 85.9 on LaSOT, ahead of GOT-Edit-378 (73.7 / 88.7 / 86.1), ToMP-378 (72.6 / 85.7 / 84.8), and LoRAT-378 (72.4 / 85.6 / 85.1).
• Naive fusion is harmful; constraint-based fusion is not: In the attribute ablation (GOT-Edit-252), naively adding geometry improves 3D-related attributes such as occlusion, visibility, and background clutter, but degrades non-3D-related attributes such as distractor, fast motion, and illumination (for example, one column drops from 49.38 to 47.18, and two others from 60.39 to 58.73 and 72.02 to 71.13). Adding the null-space constraint restores and surpasses the baseline on every listed attribute (67.95, 62.67, 46.93, 50.27, 71.60, 66.33, 67.85, 62.90, 73.23 versus the baseline's 64.32, 57.14, 42.21, 49.38, 68.97, 62.93, 64.25, 60.39, 72.02).
• Component ablation (resolution 252, trained on the classical four datasets): SUC goes from 59.2 / 68.5 / 70.7 (AVisT / NfS / LaSOT) with semantic features only, to 55.8 / 66.3 / 67.6 with VGGT geometry features alone (a large drop), to 59.9 / 67.5 / 70.9 with VGGT's DINO-head semantics plus VGGT geometry, to 60.2 / 68.5 / 71.3 with independent DINO semantics plus VGGT geometry, to 61.5 / 69.3 / 72.7 after adding the null-space constraint, and finally to 62.0 / 70.2 / 73.8 after adding whitening and regularization.
• Sources of the gain: The online editing strategy improves the baseline by an average of 2.5%, and the null-space constraint together with regularization yields gains of 1.8% on AVisT, 1.7% on NfS, and 2.5% on LaSOT over the unconstrained fusion. VGGT's own semantic features underperform an independent DINO backbone, which the authors attribute to the DINO backbone inside VGGT having been fine-tuned on large-scale 3D data, distorting its original semantic representations.
• Attribute-level analysis (Figure 3): On OTB the tracker gains notably on background clutter, occlusion, and rotation versus baseline ToMP-L378; on AVisT it improves most attributes but trails PiVOT on Imaging Effects under low-light conditions; on LaSOT it beats most attributes but is similar to or slightly below some trackers on viewpoint change and fast motion, because visual geometry becomes less effective when the scene or object moves rapidly or the viewpoint shifts substantially.

Methodology in Plain English

1. Two kinds of features from the same frames. For a set of reference frames (two are used) and the current frame, the system extracts semantic features with a DINOv2 ViT-L backbone and geometric features from the DPT head of a frozen VGGT model. VGGT is the piece that infers camera pose, point maps, and depth-like structure from just one or a few 2D images. Frames fed to VGGT are concatenated spatially so multi-frame interaction improves the geometric features.
2. Alignment and gated fusion. The geometric features are passed through a convolutional alignment network so their dimensionality and resolution match the semantic features, then combined with a gating mask: fused feature = semantic feature + mask ⊙ aligned geometric feature, where the mask is predicted by a lightweight convolution plus sigmoid and lies in [0, 1].
3. Fused features feed the model predictor. The concatenated reference and current fused features (with positional encodings and a learned foreground embedding multiplied by the reference labels) go through a Transformer encoder–decoder. The decoder outputs the weights of

Authors’ abstract

Human perception for effective object tracking in 2D video streams arises from the implicit use of prior 3D knowledge and semantic reasoning. In contrast, most generic object tracking (GOT) methods primarily rely on 2D features of the target and its surroundings, while neglecting 3D geometric cues, making them susceptible to partial occlusion, distractors, and variations in geometry and appearance. To address this limitation, we introduce GOT-Edit, an online cross-modality model editing approach that integrates geometry-aware cues into a generic object tracker from a 2D video stream. Our approach leverages features from a pre-trained Visual Geometry Grounded Transformer to infer geometric cues from only a few 2D images. To address the challenge of seamlessly combining geometry and semantics, GOT-Edit performs online model editing. By leveraging null-space constraints during model updates, it incorporates geometric information while preserving semantic discrimination, yielding consistently better performance across diverse scenarios. Extensive experiments on multiple GOT benchmarks demonstrate that GOT-Edit achieves superior robustness and accuracy, particularly under occlusion and clutter, establishing a new paradigm for combining 2D semantics with 3D geometric reasoning for generic object tracking. The project page is available at https://chenshihfang.github.io/GOT-EDIT.