RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation

AI summary

Overview

Research area: Computer Vision — specifically video scene graph generation (VidSGG), the task of detecting objects in video and labeling the evolving relationships between them.

Technical level: Intermediate. The paper assumes familiarity with DETR-style query-based detection, temporal transformers, and standard scene-graph metrics (Recall@K, mean Recall@K under with-constraint / no-constraint settings).

Scope: The paper diagnoses a failure mode it names "relational semantic inertia" and proposes a one-stage transformer, RTAformer, that injects calibrated adjacent-frame relation-query differences into predicate decoding, evaluated on Action Genome.

What This Paper Is About

Video scene graph generation models must track objects that persist across frames while also catching the moment a relationship between those objects changes — for example, a person switching from "looking at" a cup to "not looking at" it. The authors observe that existing methods lean heavily on cross-frame consistency, so a model can keep predicting the old predicate even after the relation has flipped. The goal is a framework that separates the modeling of stable entities from the modeling of changing relations, so predicate predictions stay responsive to transitions.

Key Contributions

1. Characterizing relational semantic inertia. The authors name and document a failure mode in VidSGG where a model retains a previous predicate prediction despite a relation change between entities that remain visible. They quantify it: their reimplementation of the OED baseline scores R@10 of 34.3 on relation-transition frames versus 45.9 on non-transition frames under the with-constraint setting, an 11.6-point gap.

2. RTAformer, a one-stage framework with disentangled temporal streams. The model refines instance queries and relation queries in separate temporal streams — the instance stream aggregating persistent entity evidence, the relation stream handling relation change — while keeping single-stage inference without external trackers or temporal post-processing.

3. Transition-Aware Delta Injection (TADI). Adjacent-frame relation-query displacements are decomposed into an RMS magnitude and an RMS-normalized direction, then modulated by a learned scale gate and a foreground-confidence factor before being injected into the target relation query as an explicit cue for predicate updates.

4. Spatio-Semantic Instance Alignment (SSIA). A training-only contrastive regularizer (symmetric InfoNCE) on adjacent-frame instance-pair queries that encourages consistent cross-frame grounding, so the displacement signal used by TADI is less corrupted by representational drift. It adds no inference overhead.

Main Findings

• Gains over UNO with matched backbones. Using the same frozen DINOv2 backbone, RTAformer beats UNO at every backbone scale and under both constraint settings. With ViT-L/14 it reaches R@10/20/50 of 44.6/52.9/60.2 (with constraint) and 47.2/57.8/65.9 (no constraint), improving R@10 over UNO by 5.3 and 6.4 percentage points respectively. The R@10 margin grows with backbone scale, from 1.0/2.5 points at ViT-S/14 to 5.3/6.4 points at ViT-L/14.

• Class-balanced recall also improves. With ViT-L/14, RTAformer reports mR@10/20/50 of 30.5/38.3/44.2 (with constraint) and 37.2/53.5/64.4 (no constraint), the highest reported mR@K among methods that report it, ahead of ARN and DIFFVSGG.

• Larger gains precisely on relation-transition frames. In a backbone-controlled comparison against OED (both with frozen DINOv2 ViT-B/14), RTAformer improves R@10 by 4.8 percentage points on the transition subset versus 1.7 on the non-transition subset under with constraint; under no constraint the gains are 4.1 and 1.9 points. On the full Action Genome benchmark the same pair scores 42.2/50.1 versus 38.7/46.6 (with constraint, R@10/R@20).

• Efficiency is not sacrificed. RTAformer uses 123.8M total parameters and 9.9M temporal-module parameters versus OED's 126.8M and 12.9M, with comparable 1041.0 vs 1041.6 GFLOPs and 31.35 vs 31.43 FPS.

• Displacement magnitude tracks real transitions, imperfectly. Across 105,356 persistent pair adjacencies, transition prevalence rises monotonically from 20.9% to 55.4% across equal-count deciles of displacement magnitude r_q, against an overall rate of 33.7%. Transition cases have a larger mean r_q (0.396 vs. 0.289), and r_q alone gives an AUROC of 0.621 without fitting a classifier (0.633/0.636/0.632 within attention, spatial, and contacting relations).

• Foreground confidence is a weaker cue. The gating confidence c_q is lower for transition cases than non-transition cases (0.517 vs. 0.591), with an AUROC of 0.442 when larger values are read as indicating transitions — so TADI uses it as a modulator rather than a transition detector.

• Ablations isolate each part. Starting from a frame-wise spatial baseline (39.7 R@10 with constraint, ViT-B/14), adding SSIA gives 40.8, adding DTA gives 41.2, and combining both gives 42.2. Within the relation stream, TADI (42.2) outperforms raw delta injection (41.9), direction-only injection (41.7), temporal aggregation with no injection (41.3), and the spatial+SSIA baseline (40.8).

• Against released baseline configurations. On the transition subset, RTAformer's R@10 exceeds OED's by 8.0 and 8.6 percentage points under with-constraint and no-constraint settings respectively, and also exceeds TEMPURA at all evaluated cutoffs under both settings.

Methodology in Plain English

The model works in two stages conceptually, though it runs as a single stage at inference.

First, each frame is parsed on its own. A frozen DINOv2 vision transformer plus a transformer encoder produces visual features, and two cascaded decoders turn 100 learnable triplet queries into instance-pair queries (who is paired with whom) and then relation queries (what they are doing). The same parser is applied to the target frame and to a set of reference frames, including the immediately preceding frame.

Second, temporal refinement runs in two separate lanes. The instance lane keeps the strongest, most confident reference queries to accumulate evidence about objects that persist. The relation lane does something different: for each query index it computes the difference between the target frame's relation query and the same-index query from the previous frame. That difference is split into a magnitude and a direction; the direction passes through a small learned network, while the magnitude drives a bounded gate and is also multiplied by a confidence product from both frames. The result is added back to the target query before temporal aggregation, so the predicate decoder receives an explicit "something changed here" signal instead of only smoothed context.

Because this subtraction assumes the same query index refers to the same entity pair in both frames — an assumption DETR-style decoding cannot guarantee — the authors add a training-time regularizer. Using Hungarian matching and a ground-truth check that the object has the same category and overlaps sufficiently (IoU above 0.3) in the previous frame, they pull the adjacent-frame embeddings of matched instance-pair queries together with a contrastive loss. That keeps the indices semantically stable without costing anything at test time.

Training is decoupled into two phases in the style of OED: first the spatial parser with the backbone frozen plus the alignment loss, then the temporal module with the backbone and spatial parts frozen.

Why This Matters

Impact on research. The paper reframes a fuzzy complaint — "the model is stale" — as a measurable phenomenon with a named failure mode, a constructed evaluation subset, and a diagnostic metric (displacement AUROC). That gives the VidSGG community a concrete slice of the benchmark to optimize against, rather than relying only on aggregate recall that can hide transition errors. The ablation against raw-delta and direction-only injection also shows that careful calibration, not just adding a difference term, is what produces the gain.

Real-world applications:

• Embodied navigation and manipulation, where an agent's action selection depends on whether a person is currently holding or merely near an object.
• Video question answering and video-language reasoning, where "is the person still holding the cup?" requires the model to notice the moment the relation ends.
• Surveillance and safety monitoring, where the interesting event is precisely a change in interaction (someone approaching versus touching versus leaving) between people who are continuously present.
• Video indexing and retrieval, where searching for "person holding a tool" should not return clips where the tool was only nearby earlier in the shot.

Industry relevance. The efficiency numbers matter here: the method improves accuracy while slightly reducing both total and temporal-module parameter counts relative to the OED baseline, with essentially unchanged GFLOPs and FPS (31.35 vs 31.43). That makes it a drop-in upgrade for pipelines already built on one-stage, tracker-free VidSGG, rather than a redesign requiring external tracking infrastructure.

Future Directions

• Guaranteeing cross-frame query correspondence. The authors acknowledge in their limitations that DETR-style decoding does not guarantee exact correspondence between same-index queries across frames, and that SSIA only encourages it. A decoding scheme with explicit index persistence would remove the assumption TADI depends on.

• Improving the transition cue itself. Displacement magnitude alone reaches an AUROC of 0.621 — informative but far from decisive, and the confidence gate is anti-correlated with transitions (AUROC 0.442). Better cues, or a learned combination of displacement with appearance and pose signals, are an obvious next step.

• Generalizing beyond Action Genome and SGDET. The paper states explicitly that evaluation is limited to Action Genome under the SGDET setting, and that broader evaluation across VidSGG datasets and protocols would strengthen the case for generalization.

• Extending the transition-subset methodology. The construction of relation-transition and non-transition frame subsets (25,883 and 21,474 frames from 35,523 transition adjacencies out of 105,356 persistent pair adjacencies) is a reusable evaluation idea that could be applied to other models and benchmarks as a diagnostic standard.

Target Audience

Researchers and graduate students working on video scene graph generation, temporal video understanding, or DETR-style set-prediction architectures will get the most from this paper, since the method builds directly on the OED/ARN/UNO one-stage lineage and assumes fluency with that vocabulary. Practitioners building video analytics pipelines — surveillance, robotics perception, video search — will find the efficiency comparison and the tracker-free inference design practically relevant. Readers new to the area should first read the OED baseline the paper repeatedly compares against, because much of the contribution is framed as a correction to it.

Authors’ abstract

Video scene graph generation (VidSGG) aims to capture objects and their evolving relationships in videos. This requires accounting for both the persistence of object identities and abrupt changes in relationships. Existing methods aggregate information across frames to exploit temporal context, but an emphasis on consistency can obscure these changes. As a result, a model may retain its previous prediction even after the underlying relation has changed. We refer to this failure mode as relational semantic inertia. To address this issue, we propose Relation-Transition-Aware Transformer (RTAformer), a one-stage framework that separately models entity persistence and relation changes. It injects calibrated differences between adjacent-frame relation queries to guide predicate updates, while cross-frame instance alignment during training helps reduce cross-frame representational drift. Experiments on Action Genome show improvements in recall and mean recall over existing methods, with ablation studies supporting the contributions of the proposed components and comparisons using matched backbones showing larger gains on relation-transition frames than on non-transition frames.