ESCA: Contextualizing Embodied Agents via Scene-Graph Generation

AI summary

Overview

Research area: Computer Vision / Embodied AI — multi-modal large language models (MLLMs) and structured visual grounding.

Technical level: Advanced. The paper assumes familiarity with scene graphs, CLIP-style vision-language models, Partially Observable Markov Decision Processes (POMDPs), and neurosymbolic learning.

Scope: The paper presents ESCA, a framework that improves MLLM-based embodied agents by feeding them structured spatial-temporal scene graphs generated by a new CLIP-based model called SGClip, along with the dataset and training pipeline used to build it.

What This Paper Is About

Embodied agents built on MLLMs (models like GPT-4o that see images and read text) frequently fail in physical or simulated environments because they cannot reliably connect low-level visual details — object positions, attributes, spatial relationships — to high-level language goals. The authors show that up to 69% of agent failures in their experiments stem from such perception errors. ESCA addresses this by inserting a scene-graph generation step into the agent's perception loop, giving the reasoner and planner a structured, symbolic picture of what is actually in the scene rather than forcing the MLLM to infer it implicitly.

Key Contributions

1. ESCA framework — a general method for contextualizing MLLM-based embodied agents via selective scene graph generation, where the MLLM first decides which objects, attributes, and relations matter for the current instruction, then receives only that subset as grounded context.
2. A transfer protocol consisting of two customizable prompts (concept extraction and visual summarization) that adapt ESCA to new tasks, action spaces, and reasoning styles without retraining the core system.
3. SGClip, a promptable, open-domain scene graph generation model built on CLIP, trained via a neurosymbolic pipeline that aligns automatically generated captions with the model's own scene graph predictions — no human-labeled scene graph annotations required.
4. ESCA-Video-87K, a dataset of 87K video-caption pairs (derived from LLaVA-Video-178K) enriched with object traces, open-domain concepts, and linear temporal logic specifications, plus extensive evaluations across four embodied environments and four MLLMs.

Main Findings

• Consistent gains across all models and environments: ESCA-augmented MLLMs outperformed their non-contextualized baselines on EB-Navigation, EB-Manipulation, EB-Habitat, and EB-Alfred. On EB-Navigation, the open-source InternVL-2.5 with ESCA surpassed base GPT-4o.
• Beats existing grounding modules: Compared to adding Grounding DINO or YOLO alone, ESCA provided additional gains — Gemini-2.0 improved by over 10% on EB-Navigation, and GPT-4o already boosted by YOLO gained another ~6% on EB-Manipulation.
• Large perception error reduction: Manual inspection of 60 EB-Navigation tasks showed ESCA cut the proportion of failures attributable to perception errors from 69% to 30%, shifting the error mix toward reasoning and planning.
• Strong zero-shot generalization for SGClip: Trained on ESCA-Video-87K, SGClip outperformed CLIP on the out-of-domain OpenPVSG, Action Genome, and VidVRD benchmarks for both entity class and binary relation prediction, and performance scaled with dataset size (1K → 10K → 87K).
• Data-efficient downstream transfer: After fine-tuning on just 1% of ActivityNet training data, SGClip beat state-of-the-art zero-shot video recognition baselines; with 5% (~800 videos) it reached 92.10% accuracy, approaching a fully supervised InternVideo2-6B.
• Qualitative behavioral improvement: Case studies showed agents with ESCA (e.g., InternVL navigating to a kettle) identifying targets early and moving decisively, while baseline agents failed to recognize targets and drifted or collided.

Methodology in Plain English

An MLLM agent normally looks at an image and produces a description, plan, and action in one loosely structured pass. ESCA replaces the free-form "visual description" step with four modular stages:

1. Concept extraction — the MLLM reads the instruction and current image and outputs a structured JSON list of relevant entities, attributes, and relations (e.g., "kettle," "on," "stove"), rather than a prose paragraph.
2. Object identification — Grounding DINO turns those concepts into bounding boxes, and SAM2 refines them into pixel-accurate segmentation masks.
3. Scene graph prediction — SGClip scores each concept against each segment, producing probabilistic facts: unary facts like 0.9::red(σ₁) and relational facts like 0.7::on(σ₁, σ₂). SGClip uses three inference modes — softmax over entity classes, a contrast between an attribute and its negation, and a color-tinted region comparing a relation phrase against a <norel> token.
4. Visual summarization and validation — the MLLM converts the graph back into natural language, combining it with marked-up images, key segments, and an analysis of past actions.

SGClip itself is trained without human scene graph labels. Captions from GPT-4 are parsed into temporal logic specifications (using operators like "until," "finally," "always"), object traces come from GD + SAM2, and the model is fine-tuned by checking whether its predicted scene graphs satisfy those specifications — a differentiable alignment computed through the Scallop neurosymbolic programming language, combined with semantic and contrastive losses.

To adapt ESCA to a new environment, researchers only edit two prompts: one telling the model what concepts to extract, and one telling it how to summarize the grounded scene for that task's planner.

Why This Matters

Impact on research: The paper argues that the dominant bottleneck in embodied agents is perception grounding, not reasoning or planning capability. Its demonstration that a lightweight, open-domain scene graph model like SGClip can be trained without human annotation, and that it improves both open-source and proprietary MLLMs alike, suggests a modular path forward — improving agents by improving structured perception rather than scaling end-to-end models. The neurosymbolic training approach (aligning predictions against automatically derived temporal logic specifications) is also a generalizable recipe for learning structured visual representations from weak supervision.

Real-world applications:

• Home robotics: agents that reliably navigate to and manipulate specific household objects based on natural-language instructions.
• Assistive caregiving: agents supporting aging or disabled users, where misidentified objects or positions carry real safety costs.
• Warehouse and logistics automation: robotic arms needing precise attribute and spatial grounding for pick-and-place under clutter.
• Human-robot collaboration: training or physical-activity scenarios requiring agents to track and respond to dynamic interactions.

Industry relevance: The framework is MLLM-agnostic and uses off-the-shelf components (Grounding DINO, SAM2, CLIP), making it a practical drop-in enhancement for companies building on top of commercial or open-source vision-language models without retraining them. The reported data efficiency of SGClip fine-tuning (1–5% of ActivityNet) is directly relevant to teams with limited labeled data.

Future Directions

• Latency reduction: The authors note LLM-based high-level planning introduces latency, making the current system unsuitable for real-time low-level control. A faster or more tightly integrated planner is an open engineering problem.
• Extending to 3D: ESCA currently operates on 2D image inputs and lacks point cloud or depth support, limiting depth-aware spatial precision. Adding 3D representations is a natural extension for manipulation tasks.
• Formal verification during execution: The framework lacks mechanisms for formally verifying intermediate and final states. Combining scene graph structure with runtime verification could catch failures before they compound.
• Broader transfer testing: The transfer protocol was validated on four embodied benchmarks; whether it generalizes to highly dissimilar domains (e.g., multi-agent settings, outdoor navigation) remains open.

Target Audience

This paper is most valuable to researchers and engineers working on embodied AI, vision-language models, and robotic perception — particularly those interested in structured visual representations, neurosymbolic learning, or methods for grounding MLLMs without large-scale human annotation. Readers should have background knowledge of CLIP-style architectures, scene graphs, and standard embodied agent benchmarking to fully follow the technical sections; the high-level framing and results, however, are accessible to anyone familiar with MLLM-based agents.

Authors’ abstract

Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, existing MLLMs do not reliably capture fine-grained links between low-level visual features and high-level textual semantics, leading to weak grounding and inaccurate perception. To overcome this challenge, we propose ESCA, a framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs. At its core is SGCLIP, a novel, open-domain, promptable foundation model for generating scene graphs that is based on CLIP. SGCLIP is trained on 87K+ open-domain videos using a neurosymbolic pipeline that aligns automatically generated captions with scene graphs produced by the model itself, eliminating the need for human-labeled annotations. We demonstrate that SGCLIP excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks. ESCA with SGCLIP improves perception for embodied agents based on both open-source and commercial MLLMs, achieving state of-the-art performance across two embodied environments. Notably, ESCA significantly reduces agent perception errors and enables open-source models to surpass proprietary baselines. We release the source code for SGCLIP model training at https://github.com/video-fm/LASER and for the embodied agent at https://github.com/video-fm/ESCA.