RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

AI summary

Overview

Research area: Computer Vision — visual relationship detection, scene-graph generation, and open-vocabulary recognition.

Technical level: Advanced. The paper assumes familiarity with scene-graph benchmarks (VG150, PSG, HICO-DET), contrastive text encoders, and standard recall metrics, though its central arguments about evaluation shortcuts are readable without deep modelling background.

Scope: The paper introduces a relation prediction model whose inputs (image, regions, predicate strings) are all supplied at inference, a machine-generated verified relation corpus, and a six-axis evaluation protocol, and then uses them to show that standard scene-graph recall largely measures corpus agreement rather than the relation in the image.

What This Paper Is About

Open-vocabulary object detection and promptable segmentation both let the model's taxonomy become an input — you hand them a class list or click a region, and the model adapts without retraining. Relation prediction has not made that transition: scene-graph models are still trained and evaluated on the 50 predicates of VG150 or the 56 of PSG, and their relation heads are conditioned on object class labels, which ties them to one detector and one label space.

The paper identifies three non-modelling obstacles to fixing this: no relation corpus is both free-text and verified, an architecture conditioned on labels cannot accept a vocabulary it was never trained on, and the standard metric rewards agreement with the training corpus, so enlarging the vocabulary is scored as a regression. The goal is to remove all three and to measure what each contributes.

Key Contributions

1. RelateAnything, a 53M-parameter relation model that takes an image plus regions from any source and returns scored relations over a predicate vocabulary supplied at inference as strings. Object class labels are never an input at any stage, so the region source can be swapped without retraining, and the vocabulary is a bank of text embeddings rather than a learned classifier, so changing it is a matrix substitution. It runs at 20 ms per frame on a single GPU.

2. RA-4M, described as the first relation corpus the author is aware of that is both free-text and verified against the geometry of the boxes it names: 474,413 images, 4,282,531 relations, 10,102 distinct free-text predicates, 9.03 relations per image, generated at a cost of 104 GPU-hours.

3. OV-SGG-Bench, a six-axis evaluation protocol (A1 transfer, A2 precision, A3 open vocabulary, A4 deployment, A5 graph quality, A6 spatial) with a composite score, motivated by measurements of three priors that standard recall rewards and two scoring conventions whose effect exceeds the differences they are used to measure.

4. An empirical study organised around what scene-graph recall actually measures, whether relation prediction benefits from being coupled to object prediction, and whether in-domain measurement predicts cross-dataset transfer.

Main Findings

• Standard recall is largely a prior-matching score. A frequency table over ground-truth object categories, given no pixels at all, exceeds a trained model on the metric by which leaderboards are ordered, while falling well below it per predicate. The vocabulary a training corpus shares with a benchmark also predicts recall on that benchmark.

• Cross-dataset gains are overstated fivefold in-domain. In-domain measurement overstates cross-dataset gains by approximately 5 times, and a change selected in-domain can lose out of it. This is why every axis in OV-SGG-Bench is scored across datasets and no benchmark contributes a training image.

• Large margins over the comparable-scale baseline. On three benchmarks evaluated cross-dataset and a fourth evaluated zero-shot, RelateAnything beats OvSGTR, the strongest open-vocabulary method of comparable scale, on every recall and precision metric, with mean recall higher by a factor of 2.3–3.5 and rare-predicate recall higher by a factor of 5–21. The margins persist with the same structure when a real detector supplies the regions.

• Very large parameter advantage over a VLM-based scene-graph model. Against a scene-graph model built on a 3B vision-language model, RelateAnything leads on both metrics on all three benchmarks at under 2% of the parameters, and trails on micro recall only when both are scored through a free-text matcher. (The figure caption notes ROBIN-3B is the stronger baseline on axis A1.)

• Composite protocol score. On the six-axis composite over A1, A2, A4, A5 and A6, RelateAnything scores 40.1 against 11.8 for OvSGTR; OvSGTR is described as the one system runnable on every axis and so the only one that can carry a composite.

• Text embedding spaces cannot distinguish antonyms. The teacher encoder (dino.txt, 2048-d) places above and below at a cosine similarity of 0.95, in front of and behind at 0.94, and to the left of and to the right of at 0.99, indistinguishable from its synonym pairs at 0.96 on average. Distilling a 512-dimensional student with an antonym-repulsion term raises synonym-versus-inverse AUC from 0.85 to 0.99, lowers the mean inverse cosine from 0.92 to 0.09 while synonyms stay at 0.71, and reduces hubness from 8.1 to 1.3.

• Coupling objects and relations has no shared representation to exploit. Relation supervision sharpens object identity in the dense features without creating relational ones, and the relation score is 87–93% pair context against 0.1% object identity, so there is no shared representation to justify a shared backbone.

• Wide vocabularies make supervision positive-unlabeled. At a vocabulary of ten thousand predicates, a pair annotated riding is also unstatedly sitting on; penalising the unstated predicate teaches the model to suppress correct answers. The fix is to discount negatives by an estimated probability, fitted on the 706k box pairs of the training mixture carrying more than one annotation.

• The corpus is denser and far wider than its source annotations. On the same images and boxes, RA-4M is 1.7 times denser (9.03 versus 5.29 relations per image), has 107 times the vocabulary, 1.4 nats higher predicate entropy, and reproduces 73 of 94 source predicate classes as exact strings. Mean object degree rises from 1.88 to 3.20 and the share of images whose graph is a single connected component rises from 70.9% to 82.6%.

• Geometric verification is deliberately conservative. The gate rejects 11.3% of raw candidates and fires only where box geometry logically constrains the predicate; unconstrained predicates pass unchecked and are counted as residual risk. Gaze is the largest residual class: roughly 22% of looking at / watching annotations have disjoint boxes and cannot be verified. After gating, 0% of containment and 0.1% of contact relations have disjoint boxes.

• Pair sampling is cheap. The sampler retains 99.79% of annotated positives; exhaustive scoring costs 1.02 times and yields no measurable gain.

• Exact-string metrics are conservative for this model. Because it aligns to a synonym group, it often answers with a correct group member other than the annotated one (for instance wearing where VG150 reads wears), which exact-string metrics count as a miss.

Methodology in Plain English

Getting supervision without a label budget. An image with N objects contains N(N−1) candidate pairs, and for most of them whether a relation holds is a judgement call — which is why the field kept the 50 most frequent of the 36,549 predicate strings Visual Genome's annotators wrote. The authors instead run an open-weight vision-language model (Gemma 4, 26B) over images annotated with numbered markers placed on the boxes, so that grounding is an input to the annotator rather than something inferred from its output. Each proposed relation then passes a deterministic geometric gate that rejects what box geometry contradicts and leaves what geometry cannot constrain unchecked and counted. Verification never invents a predicate string; it either rejects the relation or swaps the roles. As a separate step, spatial relations are restated from the other endpoint with probability one half so that the corpus is direction-balanced.

Making the vocabulary itself an input. The architecture splits into a visual path and a text path. A fine-tuned DINOv3 backbone encodes the image at 448×448, reading dense patches at three depths; per-object features come from coordinate-aware soft pooling where a query built from the box position attends over all patches, so an object's representation can absorb its context. Each ordered pair is represented by its two object features, the union box, a "contact" region between the boxes, and 19 geometric features, then refined by a transformer where pairs attend to each other and to the scene. A predicate-conditioned gate mixes two scoring branches — one geometry-led, one appearance-led — with the mixing weight read from the predicate's text embedding alone, so it is defined for strings never seen. The predicate vocabulary is embedded offline by a frozen text encoder into a bank of unit vectors that no learned layer transforms, so swapping vocabularies is a matrix substitution.

Handling ten thousand predicates in training. Training uses a batch-local InfoNCE loss: for each annotated pair, the positive is the annotated predicate's synonym group aggregated as a weighted mean rather than a single string, and negatives are discounted by an estimated probability that they are also true. Annotated predicates are never down-weighted and directional inverses always receive full weight, since separating above from below is the one signal that must not be softened. A separate distillation step fixes the text encoder, using an antonym-repulsion hinge on known inverse pairs plus a neighbourhood-preservation term, without which a student reaches the same separation at an effective dimension of 24 rather than 44.

Two towers. The "zero-shot tower" trains on RA-4M plus leakage-filtered raw Visual Genome annotations using all their free-text strings rather than the 50 of VG150 (40,615 images, 758,547 relations, 17,742 predicate strings), with a mixture of 91.9%/8.1% by image and 84.6%/15.4% by relation. The "released tower" additionally includes the HICO-DET training split at 21.0% of images, which is a 5.0% relation share because HICO-DET annotates 1.94 relations per image against 9.02 for the leakage-filtered RA-4M; its mixture is 72.7/6.3/21.0 by image and 80.6/14.4/5.0 by relation. Both are scored against the same 19,103-string predicate bank (RA-4M's 10,102 plus 9,001 strings from three further corpora whose relations are not trained on), so the zero-shot tower carries 108 of HICO-DET's 116 predicate strings as columns without training on its relations.

Measuring the right thing. The authors first analyse what scene-graph recall rewards, then build the protocol around those findings: every axis is scored across datasets and no benchmark contributes a training image; two axes are scored against negatives a human adjudicated, where a confident false positive costs as much as a miss; and the six axes are reported together and aggregated in a way that penalises imbalance, since each can be satisfied individually by a different shortcut.

Why This Matters

Impact on research. The paper's sharpest claim is not about a model but about a metric: it argues the field's leaderboard-ordering number is substantially a prior-matching score, and that in-domain evaluation overstates cross-dataset gains by roughly 5 times. If that holds, a large body of reported scene-graph progress is measuring corpus agreement. The paper also names a defect that bounds every method regressing onto off-the-shelf text embeddings — antonyms embedded at 0.95 to 0.99 cosine similarity — and shows it is fixable in the objective rather than in the visual model. Finally, it supplies a release (model, corpus, benchmark) that others can build on or attack.

Real-world applications:

• Robotics and manipulation, where a robot needs relations like parked by, hanging from or on top of between objects it has already segmented, and where the region source changes per deployment (the paper notes IndoorVG targets indoor robotics).
• Assistive and accessibility systems that describe a scene's relationships rather than just listing objects, with regions coming from whatever detector or segmenter is available.
• Autonomous driving and traffic scene understanding, where an object detector supplies regions and the relation vocabulary must be chosen per scenario rather than fixed at training time.
• Image and video search or content moderation, where a user supplies arbitrary relation strings at query time and expects scored relations without retraining.

Industry relevance. The interface properties matter operationally: no object labels as input means a deployed system can swap detectors or segmenters without retraining, and a text-embedding vocabulary means changing the answer space costs a matrix substitution rather than a training run. The 20 ms per frame figure on a single GPU, alongside a real-time object detector, puts it in the latency budget of live systems. The under-2% parameter comparison against a 3B vision-language model matters where deployment cost, not peak accuracy, is the constraint.

Future Directions

• Verification for geometrically unconstrained predicates. The gate only fires where box geometry logically constrains the predicate; gaze is the largest residual class, with roughly 22% of looking at / watching annotations having disjoint boxes that cannot be verified. A verification mechanism for those classes would close a known coverage gap.
• Whether the text-encoder correction generalises. Antonyn-repulsion is applied to known inverse pairs in the distilled student. How the approach extends to other relation types, and to vocabularies whose inverse pairs are not enumerated, is left open.
• Whether the evaluation findings transfer to other tasks. The paper argues shared triplet mass between training corpus and benchmark predicts recall. Whether the same prior and scoring-convention effects distort other open-vocabulary tasks is not addressed.
• Scaling the vocabulary bank. The bank already mixes 10,102 RA-4M predicates with 9,001 strings from three other corpora; whether widening it further, or including corpora in other languages, changes behaviour is an open question.
• Closing the free-text-matcher gap. The model trails on micro recall only when both systems are scored through a free-text matcher, which the paper attributes partly to group-based alignment. Reducing that gap without losing the rare-predicate advantage is a natural next step.

Target Audience

Researchers and engineers working on scene-graph generation, visual relationship detection, and open-vocabulary perception — particularly those who report or consume mean recall numbers on VG150, PSG or HICO-DET and need to know what those numbers do and do not measure. It is also relevant to practitioners building deployed perception systems that must accept regions from arbitrary detectors or segmenters and predicate vocabularies chosen at runtime, and to anyone using off-the-shelf contrastive text embeddings for relation or spatial reasoning, for whom the antonym-similarity finding is directly actionable. Beginners will find the modelling sections dense, but the evaluation critique and the corpus pipeline are readable on their own.

Authors’ abstract

Open-vocabulary detection accepts any class list at inference, and promptable segmentation returns regions without class names: the taxonomy has left the model and become an input. Relation prediction has not. Scene-graph models are still trained and evaluated on the 50 or 56 predicates of one annotation style, their relation head conditioned on object labels and so tied to one detector. Three obstacles explain this, none primarily modelling: no relation corpus is both free-text and verified, a label-conditioned architecture cannot accept a vocabulary it was not trained on, and the standard metric rewards agreement with the training corpus, so a larger vocabulary scores as a regression. We present RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings. Object labels are never an input, so the region source can change without retraining, and the vocabulary is a bank of text embeddings, not a learned classifier. It runs at 20 ms/frame. Training over 19,103 predicates requires positive-unlabeled supervision and a text encoder that separates antonyms, which contrastive encoders embed at cosine 0.95. To supply the supervision we build RA-4M, 474k images and 4.3M relations over 10,102 free-text predicates, generated against numbered box markers and geometrically verified. To measure it we build OV-SGG-Bench, six axes scored across datasets that the priors standard recall rewards cannot satisfy. On three cross-dataset benchmarks and a fourth zero-shot, RelateAnything has 2.3-3.5x the mean recall of the strongest open-vocabulary method of comparable scale, margins that survive a real detector, and leads a 3B-VLM scene-graph model on both metrics at under 2% of its parameters. In-domain measurement overstates transfer gains ~5x. Model, corpus and benchmark are public.