BridgeEQA: Virtual Embodied Agents for Real Bridge Inspections

AI summary

Overview

Research area: Computer Vision / Embodied Question Answering (EQA) and vision-language models, applied to infrastructure (bridge) inspection.

Technical level: Advanced.

Scope: This paper introduces BridgeEQA, a 2,200-question open-vocabulary EQA benchmark built from real bridge inspection reports, a new image-citation evaluation metric, and EMVR, a scene-graph-based navigation method for answering inspection questions with vision-language models.

What This Paper Is About

Existing Embodied Question Answering benchmarks mostly test vision-language models on short, simple queries in small household-style scenes, which does not reflect the kinds of reasoning required in real deployments. The authors argue that bridge inspection is a demanding alternative testbed: it requires multi-scale reasoning, long-range spatial understanding, and answers that must be justified with specific photographic evidence and aligned to a standardized rating rubric. Their goal is to build such a benchmark from real inspection data and to propose a method that outperforms the standard long-context baseline on it.

Key Contributions

1. BridgeEQA, a real-world EQA benchmark for infrastructure inspection with expert-grounded supervision: 2,200 question-answer pairs over 9,586 images from 200 bridges across 73 towns, presented as an example of a new EQA problem class called Inspection EQA.
2. Image Citation Relevance, a new metric that measures semantic similarity between the images an agent cites and a reference set of inspector-validated images (validated at a Spearman correlation of 0.817 against three human annotators).
3. Embodied Memory Visual Reasoning (EMVR), a method that formulates question answering as traversal over an image-based scene graph constructed without GPS or spatial sensors.
4. Comprehensive baselines benchmarking contemporary VLMs (Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, Grok 4 Fast) and EQA methods on BridgeEQA.

Main Findings

• Baselines leave a substantial gap: The strongest OpenEQA-style baseline, Multi-Frame VLM, scores 0.576 Answer Correctness and 0.687 Image Citation Relevance with Grok 4 Fast, and only 0.484 and 0.694 respectively with Gemini 2.5 Flash.
• EMVR improves over Multi-Frame VLM: Using Grok 4 Fast, EMVR improves condition rating accuracy within ±1 by 9.34 percentage points, Image Citation Relevance by 20.2 percentage points, and Answer Correctness by 7.2 percentage points.
• Alternative framing of the same gains: The contributions section reports improvements of 13.6% in condition rating accuracy, 29% in visual evidence grounding, and 12.5% in answer quality over non-navigational baselines. The paper reports these percentages differently in these two places.
• Best Answer Correctness: Grok 4 Fast with EMVR VLM w/ Images + SG reaches 0.648; Gemini 2.5 Flash with EMVR VLM w/ SG Only reaches 0.609, both above their respective Multi-Frame VLM scores.
• Best visual grounding: Grok 4 Fast with EMVR VLM w/ Images + SG achieves 0.889 Image Citation Relevance, the highest in Table 2.
• Scene graph context alone helps Multi-Frame VLM: Adding scene graph context raises Multi-Frame VLM's scores (e.g., 0.581 vs. 0.484 on Gemini 2.5 Flash and 0.802 vs. 0.694 Image Citation Relevance on the same model).
• Two dominant failure modes: Poor image citations (citing irrelevant images leads to wrong ratings) and hallucinated image citations (the model invents filenames that do not exist). The authors suggest low-quality citations can act as a proxy for detecting hallucinations or poor answer generation.
• Dataset quality checks passed: Faithfulness 0.997, Answer Relevancy 0.997, Answerability 0.996, and Inspector Relevancy 0.980 using RAGAs Faithfulness & Answer Relevancy, RAGalyst Answerability, and an LLM-as-a-Judge Inspector Relevancy score.
• Rating distribution is realistic: NBI ratings center on 5-7 (Fair to Good), with rating 6 (Satisfactory) the most common at 397 questions.
• Question mix (random sample of 300 QA pairs): Aggregative 38.5%, Comparative 27.2%, Relational 21.3%, Spatial 17.5%, with questions frequently combining multiple types.
• Human agreement context: The paper cites that expert inspectors typically agree within ±1 condition rating, and that expert inspectors show 98% consistency between assigned ratings when ratings fall within ±1 of a median rating.

Methodology in Plain English

The authors started from unstructured PDF bridge inspection reports in the Vermont Agency of Transportation public database. The first step was the data itself: they filtered reports, pages, and images for quality (minimum 20 images per report, removal of administrative pages and images smaller than 200x200 pixels that were mostly logos or thumbnails), then randomly sampled 200 reports, producing 9,586 images at an average of 47.93 per report.

They used Gemini 2.5 Flash as a zero-shot parser to structure the report text, map photographs to inspector notes, and pull out component-level NBI condition ratings on the 0-9 scale, with Gemini 2.5 Pro as a fallback when parsing errors or hallucinations appeared. Human review and automatic checks validated the output before question-answer pairs were generated. The dataset was split into 1,100 training and 1,100 test QA pairs, and evaluation used the 1,100 test pairs.

To make an agent able to navigate the image collection, they built scene graphs where each node is an image rather than a detected object, since bridge inspection lacks foundation models that can densely detect components like bearings and expansion joints. Each node carries an image name, a "central focus" label in inspector terminology, a description, and edges to related images with natural-language relationship descriptors (hierarchical, structural, spatial adjacency, condition similarity, component membership). This construction uses only images, with no GPS or external spatial sensors.

Their method, EMVR, reformulates the task as a Markov decision process. The agent starts with only the scene graph structure and takes function-call actions: Move to a neighboring node, Compare two or more images, Reason about a single image, or Respond with an answer, cited images, and a condition rating. The motivation is that feeding all images into a long context at once suffers from positional bias ("lost in the middle"), so letting the agent retrieve and promote relevant images to the front of the context window should help.

Evaluation uses three signals: open-vocabulary text answers scored by LLM-as-a-judge (Answer Correctness), condition rating accuracy (exact match and within ±1), and the new Image Citation Relevance metric, where Gemini 2.5 Flash acts as a judge comparing agent-cited images to the reference set on a 0.0-1.0 scale, penalizing over-selection only if an agent cites more than 5 times the reference set (in practice all methods chose fewer than 6 images and were never penalized sharply).

Why This Matters

Impact on research: The paper defines an "Inspection EQA" problem class with a quantitative checklist (questions depending on multiple views, answers tied to a rating scale, reference image sets, and citation relevance correlating with human agreement) so that bridges, dams, tunnels, and pipelines can be compared directly. It also contributes a reusable evaluation idea: judging whether an agent cites the right evidence, not just whether its text answer is correct.

Real-world applications:

• Automated assistance for routine bridge inspections, which the National Bridge Inspection Standards require at least every two years for public highway bridges longer than 20 ft.
• Prioritizing maintenance and rehabilitation by predicting component-level NBI condition ratings for decks, superstructures, substructures, abutments, and wingwalls.
• Remote and autonomous inspection using drones or robotic platforms, given the paper's framing of aging infrastructure: nearly 40% of U.S. bridges have reached or exceeded their 50-year design life, and over 10% are subject to load restrictions.
• Generating grounded inspection documentation that links condition claims to specific photographs, mirroring how inspectors justify ratings.

Industry relevance: This targets a labor-intensive, costly, and safety-critical workflow with standardized professional rubrics, and the authors acknowledge partial financial support from the Texas Department of Transportation (grant 0-7181), indicating direct practitioner interest.

Future Directions

• Extending Inspection EQA to other assets: The paper explicitly frames the problem class as applicable to dams, tunnels, and pipelines, which would test whether the checklist and benchmark transfer across domains.
• Closing the remaining performance gap: Even the best configuration (Grok 4 Fast, EMVR VLM w/ Images + SG) reaches 0.648 Answer Correctness, leaving substantial room before model answers match expert judgment.
• Using citation quality as a hallucination signal: The authors' observation that poor or hallucinated citations accompany wrong answers suggests detecting citation failures could be developed into a routing or rejection mechanism.
• Broader model coverage: Sub-30B-parameter open-source VLMs were excluded from the main comparison because they could not reliably follow the required structured-output and function-calling formats and have substantially lower context windows; the paper notes results where applicable appear in the Supplementary Material, leaving open how smaller models could be adapted.
• Graph structure sensitivity: The paper states that the effect of node and edge count is analyzed in the Supplementary Materials, so the design of the scene graph itself remains a tunable factor.

Target Audience

Researchers working on embodied question answering, open-vocabulary VQA benchmarks, and vision-language model evaluation; civil and structural engineering researchers interested in AI-assisted inspection; and practitioners in transportation agencies or inspection technology companies who want a measurable target for automated condition assessment. Readers should be comfortable with vision-language models, agentic tool-use policies, and the NBI condition rating scale, though the paper explains the inspection context in accessible terms.

Authors’ abstract

Deploying embodied agents that can answer questions about their surroundings in realistic real-world settings remains difficult, partly due to the scarcity of benchmarks for episodic memory Embodied Question Answering (EQA). Inspired by the challenges of infrastructure inspections, we propose Inspection EQA as a compelling problem class for advancing episodic memory EQA. It demands multi-scale reasoning and long-range spatial understanding, while offering standardized evaluation, professional inspection reports as grounding, and egocentric imagery. We introduce BridgeEQA, a benchmark of 2,200 open-vocabulary question-answer pairs (in the style of OpenEQA) grounded in professional inspection reports across 200 real-world bridge scenes with 47.93 images on average per scene. We further propose a new EQA metric Image Citation Relevance to evaluate the ability of a model to cite relevant images. Evaluations of state-of-the-art vision-language models reveal substantial performance gaps. To address this, we propose Embodied Memory Visual Reasoning (EMVR), which formulates the inspection EQA task as a Markov decision process. EMVR shows strong performance over the baselines. Code and dataset are available at https://drags99.github.io/bridge-eqa/