ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

AI summary

Overview

Research area: Computer vision, specifically text-promptable segmentation — open-vocabulary instance tracking and semantic discovery, with a stated design target of 3D reconstruction and novel view synthesis pipelines.

Technical level: Intermediate. The paper describes a modular cascade (region proposal, embedding matching, VLM verification, mask generation) in concrete architectural terms, but relies on concepts from existing literature (contrastive vision-language embeddings, memory-based video object segmentation, VLM judges) that a reader should be familiar with.

Scope in one sentence: ENEAS is a single text-promptable method that either tracks one specific object through a frame sequence or discovers every instance of a named category, using a fast embedding filter plus a conditionally invoked vision-language model judge to reject look-alikes that visually match the prompt.

What This Paper Is About

Current text-promptable segmentation foundation models (the paper names SAM, SAM 2 and SAM 3) fail in three ways on uncurated capture data: they hallucinate the target when it has left the field of view, they fragment a single object into its parts during extreme close-ups, and they segment things that merely look like the target — statues, paintings, reflections — as if they were real instances. The authors' goal is to add a verification layer on top of strong perception so that segmentation is not just geometrically correct but semantically correct, without paying the latency cost of running an expensive reasoning model on every candidate.

Key Contributions

1. A unified text-guided segmentation method handling both instance tracking and semantic discovery in one framework, applicable to ordered video and to unordered image collections.
2. Text-driven initialization for memory-based tracking: the SeC tracker, previously limited to point interaction, is extended with an open-vocabulary grounding stage, giving a text-prompted tracker that reports absence instead of drifting and keeps the object whole under extreme scale change.
3. Semantic verification only where needed: sigmoid-based embedding matching with prompt ensembles, plus a conditional VLM judge invoked only inside an uncertainty interval, with contextual neighbour masking to isolate each candidate. This keeps the VLM activation rate — and therefore latency — low.
4. An empirical study of ontological robustness: on a real capture of hyper-realistic statues, the method removes the false positives SAM 3 produces while preserving most of its recall. The paper states that a set of ablations quantifies each component's contribution and the accuracy–latency trade-off of the uncertainty interval, but the ablation results are not present in the available content (the text truncates at the start of Section 5).

Main Findings

• Church Statues quantitative comparison: Under the prompt "person", SAM 3 reaches 11.1% precision, 83.7% recall and 19.5% F1. ENEAS-2B (the default configuration) reaches 94.7% precision, 73.5% recall and 82.8% F1. ENEAS-4B reaches 97.5% precision, 79.6% recall and 87.6% F1. The paper describes the 2B result as a four-fold F1 improvement over the baseline.
• Precision-first trade-off: The authors explicitly accept lower recall in exchange for high precision, motivated by asymmetric error costs in 3D reconstruction — a false positive masks part of the asset being reconstructed, while a false negative leaves a visible artifact.
• Scale with judge capacity: Moving from the 2B to the 4B semantic judge recovers instances that were ambiguous to the smaller model, improving recall while keeping near-perfect precision. The paper presents this as a configurable trade-off between an efficient baseline (2B) and maximum rigor for offline processing (4B).
• SA-Co/VEval — Instance Tracking (SA-V Test subsample): ENEAS HOTA 26.70 versus SAM 3 26.51; DetA 7.92 versus 7.84; AssA 90.77 versus 90.18; LocA 87.86 versus 89.22; TETA 17.86 versus 16.65.
• SA-Co/VEval — Semantic Discovery: ENEAS HOTA 9.23 versus SAM 3 9.19; DetA 6.16 versus 6.21; AssA 14.28 versus 13.85; LocA 74.25 versus 75.38; TETA 9.53 versus 9.10.
• Where the SA-Co gains come from: ENEAS improves end-to-end tracking metrics (HOTA and TETA) mainly through stronger association quality (AssA) with comparable detection quality (DetA), while SAM 3 remains slightly better at localization (LocA).
• Robustness to target absence: On the Blue Painting sequence, Grounded SAM (Grounding DINO with SAM 2.1) hallucinates detections on curtains and hair when the target is absent or occluded; ENEAS reports a true negative (zero mask) using the propagated temporal memory of SeC.
• Spatial integrity under close-ups: On deep close-ups, Grounded SAM and SAM 3 segment high-frequency internal details (brushstrokes, figures within the painting) rather than the whole object, while ENEAS returns a complete mask.
• Continuous discovery: On Moving Boxes with the prompt "chair", Grounded SAM over-segments tables, cardboard boxes and walls as chairs; ENEAS rejects geometrically similar distractors. ENEAS also re-evaluates every frame, so new instances entering the scene are found without re-prompting.
• Failure mode that remains: In extreme close-ups where surrounding context is entirely absent, ENEAS can exhibit leakage to the baseline's behaviour because the VLM has no visual cues, though the paper states the frequency and severity of such false positives are reduced across the sequence relative to SAM 3.
• Latency of the rejected design: The first design phase, using exhaustive dense captioning with composite visual prompting, started at a minimum of 15 seconds per frame and escalated in dense scenes, which the authors judged inviable for video. The final conditional verification reportedly runs at approximately one second on resource-constrained hardware.
• Hardware setup: Efficiency and latency experiments used a single NVIDIA L4 GPU (Ada Lovelace, 24 GB VRAM) with an Intel Xeon CPU running 12 virtual cores at 2.20 GHz; reported latency covers the full pipeline including data loading and CPU-bound preprocessing.

Methodology in Plain English

ENEAS takes a natural-language prompt and a set of frames — ordered video or an unordered image collection — and returns binary masks. It branches on what the prompt means.

If the prompt names one specific instance ("the blue painting"), the grounding operator Florence-2-Large finds that region once on a reference frame, and the SeC tracker propagates it through the rest of the sequence using its memory of frames already processed. Because the tracker carries a concept-level representation of the object rather than matching appearance alone, it can output an empty mask when the object is gone instead of snapping to whatever looks similar.

If the prompt names a category ("person", "chair"), every frame is re-evaluated through a cascade of increasingly expensive stages, each seeing fewer candidates than the last. First, Florence-2 proposes candidate regions permissively, merging duplicates. Second, each candidate is scored against the category by SigLIP 2, using a pairwise sigmoid formulation so each image-text pair is judged independently — meaning a microphone or accessory in the crop cannot suppress the score of the target the way a softmax normalization would. Multiple phrasings of the category are averaged into one score. Two thresholds split candidates: those above the acceptance threshold are kept, those below the rejection threshold are dropped, and the middle band — the uncertainty interval — goes forward. Third, only those ambiguous candidates are shown to a Qwen3-VL judge, with neighbouring candidates' pixels masked out so the verdict concerns the candidate alone, and with a fixed answer format instead of free-form reasoning to keep cost low. The accepted set is finally turned into one mask per instance by the SAM 2 segmentation head.

The design was reached through four documented development phases: exhaustive scene captioning with composite visual prompting (abandoned for hallucinated labels, merged bounding boxes and prohibitive latency); text-conditioned proposals filtered by softmax contrastive matching (abandoned because softmax forces classes to compete and suppresses the target score); sigmoid-based embedding matching with native-aspect-ratio encoding and prompt ensembling (adopted); and optimization of the VLM judge (masking context, disabling deep reasoning).

Why This Matters

The paper argues that the failures it targets share a root cause — strong perception coupled with weak verification. Its contribution is structural: decoupling geometric proposal from semantic verification, so that a cheap embedding filter handles the easy cases and an expensive reasoning model is reserved for genuine ambiguity. This reframes the problem from "detect better" to "verify what you detected", which matters for any downstream task where a single misclassified distractor is destructive.

Real-world applications the paper points to or that follow directly:

• Preprocessing captures for 3D reconstruction and novel view synthesis, where transient visitors must be removed without touching the objects being scanned — the motivating scenario for the Church Statues capture.
• Removing distractors before NeRF or 3D Gaussian Splatting reconstruction, so that floaters and ghosts never enter the representation, as opposed to the alternative of modelling transients inside the reconstruction from photometric residuals.
• Video analysis and automated inventory or scene monitoring, where the paper notes that over-segmented output is unusable for automated analysis.
• Archival and heritage digitization, where the distinction between an artifact and a living subject is exactly the kind of judgement current models fail.

Industry relevance: the design explicitly targets offline, cost-effective cloud inference on a single NVIDIA L4 rather than high-end HPC hardware, and the conditional VLM invocation means cost scales with scene ambiguity rather than object count. Code and models are released at github.com/speridlabs/eneas, and the 2B/4B judge variants give deployers a choice between throughput and semantic rigor.

Future Directions

• Complete the ablation study. The paper promises ablations quantifying each component's contribution and the accuracy–latency trade-off of the uncertainty interval, but these results are not present in the available content. Section 5 is truncated before any numbers appear.
• Close the context-free close-up gap. The authors acknowledge that in extreme close-ups with no surrounding context, their method leaks the same way the baseline does, because the VLM has nothing to reason about. Recovering context — from neighbouring frames, other views of the same scene, or the 3D reconstruction itself — is the obvious next step.
• Choose and tune the uncertainty interval. The two thresholds τ_rej and τ_acc are described as the knobs trading latency against semantic rigor, and the paper defers their study to a section whose results are not available here. How to set them automatically, and how they behave across categories and scenes, remains open.
• Broaden the ontological evaluation. The quantitative semantic claim rests on a single capture, Church Statues, with manually produced ground truth and a single prompt ("person"). Whether the precision advantage generalizes to other categories, other captures, and other kinds of look-alikes (reflections, mannequins, screens) is untested here.
• Reconcile the SA-Co trade-off. ENEAS wins on AssA and TETA but loses LocA to SAM 3 on both splits. Whether the localization gap can be closed without giving up association quality is an open engineering question.

Target Audience

Researchers and practitioners in computer vision working on open-vocabulary segmentation, referring and reasoning segmentation, video object tracking, and vision-language model cascades. It is particularly relevant to engineers building 3D reconstruction or novel view synthesis pipelines who need to remove dynamic distractors from a capture before reconstruction, and to anyone benchmarking text-promptable foundation models on data where the hard part is semantic rather than geometric. Readers new to the area will need background in promptable segmentation models, memory-based video object segmentation, and contrastive vision-language embeddings; readers familiar with those will find the design-evolution narrative — four phases and why each was abandoned — the most directly useful part.

Authors’ abstract

We present ENEAS, a unified, text-promptable method for instance tracking and semantic discovery. Text-promptable segmentation models, including the latest foundation models such as SAM 3, still suffer from temporal hallucinations, spatial fragmentation, and semantic misclassification: they fail to report target absence when an object leaves the field of view, segment local textures instead of the complete object during extreme close-ups, and prioritize visual features over ontological reality, so that visually similar artifacts such as statues, paintings, or reflections are segmented as target entities. ENEAS works two ways from a single method: precise tracking and high-quality segmentation of a unique instance, and open-concept discovery of every instance a text query names, resolved by a semantic verification layer. For tracking, we extend the geometrically robust SeC architecture, previously limited to point interactions, with a text-prompting adapter and leverage its temporal memory, so that the target is held through disappearance without drifting to distractors and kept whole even when it fills the entire view. For discovery, the verification layer combines high-speed visual embedding matching with conditional VLM refinement, invoking semantic reasoning only for ambiguous candidates, which filters out the ontological errors that visual-only models cannot distinguish while keeping latency low. Designed with 3D reconstruction in mind, where a single misclassified distractor corrupts the asset, ENEAS unlocks high-quality semantic tracking and segmentation of video, of broad libraries, and of collections of temporally or spatially unordered data, together with the discrimination to tell true instances from their doppelgangers: things that look alike but are not the same. The code and models are available at https://github.com/speridlabs/eneas