A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans

AI summary

Overview

• Research area: Computer Vision / medical vision-language models (VLMs), agentic AI tool use, and radiological spatial reasoning on CT imaging.
• Technical level: Intermediate. The modular agent concept and the evaluation logic are accessible, but the paper assumes familiarity with VLM basics, object detection, and common accuracy/F1 metrics.
• Scope (1 sentence): The paper builds and evaluates a modular medical imaging agent that answers binary spatial relation questions about axial CT slices by combining language parsing, YOLO-based anatomical detection, and deterministic geometric comparison, rather than letting a VLM predict the answer end-to-end.

What This Paper Is About

Modern vision-language models perform well on many medical imaging tasks but are known to be weak at controlled spatial reasoning, often landing near chance level on questions about whether one anatomical structure is left of, right of, above, or below another. That weakness matters because radiology reports depend on correctly describing the relative positions of anatomy and findings, and because end-to-end VLM answers cannot be traced back to image evidence. The authors' goal is to show that a modular agent, which decomposes the spatial question into explicit stages and delegates the final decision to deterministic geometry, can answer these questions far more accurately and with auditable intermediate steps.

Key Contributions

1. Modular agent architecture: A task-oriented medical imaging agent that keeps a standard conversational VLM interface while internally delegating visual reasoning to specialized, auditable tools orchestrated with LangChain.
2. Explicit spatial verification pathway: A four-stage pipeline (tuple extraction, ontology matching, YOLO-based localization, deterministic geometric verification) that replaces hallucination-prone VLM prediction with detection plus coordinate comparison.
3. Stage-wise evaluation and failure attribution: An evaluation that goes beyond black-box accuracy by assigning every incorrect or invalid case to the earliest failing pipeline stage, plus an open-source release of code to extend the agent with new medical reasoning tools.

Main Findings

• Hybrid agent substantially outperforms direct prompting: Qwen2-VL + hybrid agent reached 94.1% accuracy and 94.2% F1 on the held-out MIRP RQ1 test set, versus 51.6% accuracy and 56.8% F1 for direct Qwen2-VL prompting, an absolute accuracy gain of 42.5 percentage points.
• Second hybrid configuration also performs strongly: MedGemma + hybrid agent reached 91.6% accuracy and 91.3% F1, compared with 51.8% accuracy and 67.2% F1 for direct MedGemma on the same held-out test set.
• Direct VLM baselines sit near chance: Direct MedGemma (51.8% accuracy) and direct Qwen2-VL (51.6% accuracy) stayed close to chance level, consistent with previously reported MIRP RQ1 leaderboard results (GPT-4o 50.5% / 35.2 F1, Gemma 3 50.9% / 57.3 F1, Pixtral 50.7% / 43.2 F1, MedGemma 50.3% / 60.7 F1).
• No invalid binary outputs: The paper reports that no invalid binary outputs were observed in the evaluated configurations.
• Remaining errors are mostly perceptual: Of the 55 incorrect or invalid cases for the best hybrid configuration, imprecise localization was dominant at 28 cases (50.9%), followed by parsing/query-extraction errors at 15 cases (27.3%) and missing detections at 12 cases (21.8%). No failures were attributed to question extraction, routing, ontology matching, geometry ambiguity, or formatting/runtime issues.
• Performance is bounded by tool quality: Because the geometric verifier is deterministic, remaining errors are traceable to the upstream tools, especially the detector, rather than to an opaque multimodal prediction.
• The comparison is not strictly paired: The paper states that its evaluations were performed on a held-out test set that differs from the leaderboard data, so the leaderboard rows are included for context and are not strictly paired comparisons, which also explains the difference between previously reported and newly evaluated MedGemma results.

Methodology in Plain English

The authors treat "Is the liver left of the spleen?" style questions as a controlled subproblem rather than asking a VLM to answer directly. In the hybrid setup, the same prompt is given to an agent as to the baseline VLMs. A VLM-based controller extracts the core question and routes it to a dedicated spatial verification pathway, with LangChain handling prompt handling, tool invocation, routing, and intermediate state management (not additional reasoning). The pathway then runs four stages: convert the question into a structured tuple such as (liver, left of, spleen) using a lightweight parser with a VLM fallback; map the extracted entities to canonical detector classes, returning an invalid result with an audit entry if that fails; localize the structures with a YOLO-based detector, passing only the highest-confidence bounding box per class forward; and compute the truth value from the horizontal or vertical ordering of the detected object centroids. No neural model directly predicts the final truth value.

The detector was trained on the MIRP RQ1 training cohort using segmentation-derived bounding boxes, supplemented with images from the AMOS and BTCV datasets, with strict patient-level isolation so no test-cohort patients appear in detector training. The final detector covers 61 anatomical classes and was trained on 421,023 instances. Evaluation used a held-out test set of 938 image-question pairs. Direct VLM decoding was deterministic with temperature set to 0, and all systems received identical prompts asking for a single character answer (1 or 0). The paper reports accuracy, F1, and invalid-output rate, with strict evaluation counting invalid, failed, or non-binary outputs as incorrect.

Why This Matters

The work suggests that the spatial grounding gap in medical VLMs is not necessarily a fundamental limit of the underlying models, but something that can be addressed architecturally by delegating the spatial decision to explicit, verifiable components. For research, it reframes evaluation: instead of reporting only a black-box accuracy number, the agent exposes intermediate representations so errors can be assigned to specific modules. That shift matters for clinical trust, where a plausible answer is insufficient if it cannot be traced to image evidence.

Real-world applications:

• Radiology report support: grounding laterality and positional statements about anatomy and findings in verifiable image evidence, reducing the risk of statements that cannot be traced to the image.
• Structured image understanding: building block for converting free-text clinical queries into structured, machine-checkable relations over anatomy.
• Quality assurance and auditing: module-level failure attribution can flag whether an error came from language parsing or from detection, supporting targeted review in clinical pipelines.
• Foundation for extensible clinical agents: the same auditable architecture is designed to accept additional tools such as presence verification (already implemented) and measurement-oriented reasoning (described as a future extension).

Industry relevance: the design separates a familiar conversational interface from auditable internal tools, which is a practical template for medical AI products that must justify their outputs to regulators, clinicians, or hospital IT. The paper's code release explicitly supports extending the agentic framework with new medical reasoning tools.

Future Directions

• Improve the perception tools: since imprecise localization (28 of 55 errors, 50.9%) and missing detections (12 of 55, 21.8%) dominate remaining failures, better or more robust detectors are the most direct lever.
• Reduce parsing errors: parsing/query extraction accounted for 15 of 55 errors (27.3%), pointing to better language-to-tuple conversion.
• Extend beyond pairwise 2D relations: the current implementation is limited to pairwise 2D spatial relations in axial CT slices and does not address full volumetric reasoning, distance-sensitive relations, containment, overlap, or multi-structure consistency; the authors state that tools for 3D routing or volumetric measurement can be integrated into the same auditable architecture.
• Add additional clinical pathways: measurement-oriented reasoning, presence verification, and structured report generation are named as extensions the modular framework is designed to accommodate.

Target Audience

Researchers and practitioners working on medical vision-language models, agentic AI for healthcare, and radiological image understanding, as well as clinical AI engineers interested in auditable, tool-based system designs. It is also relevant to readers studying spatial reasoning benchmarks and evaluation methodology in multimodal models. Readers without a background in VLM evaluation or object detection will find the high-level argument accessible, but the stage-wise failure analysis and benchmarking details assume some technical familiarity.

Authors’ abstract

Reliable spatial understanding is an important prerequisite for future medical vision-language systems that aim to support radiological report generation and structured image understanding. While modern vision-language models (VLMs) show promising performance on many medical imaging tasks, recent evidence suggests they remain weak in controlled spatial reasoning and often fail to reliably ground spatial relations in image evidence. Given that radiological reasoning hinges on understanding the relative positions of anatomical structures and findings, this spatial weakness poses risks to diagnostic accuracy. We present a modular medical imaging agent for binary spatial relation verification in axial CT slices. Instead of directly predicting spatial answers end-to-end, the system decomposes the task into explicit stages: language parsing, anatomical localization, and deterministic geometric verification. Natural-language queries are converted into structured relation tuples, queried organs are localized with a YOLO-based detector, and the final spatial decision is computed from object centers using deterministic geometric rules. We evaluate the approach on the held-out MIRP spatial QA benchmark and compare it against representative end-to-end VLM baselines. The best-performing hybrid configuration reaches 94.1% accuracy and 94.2% F1, outperforming direct Qwen2-VL prompting by 42.5 percentage points in accuracy, while preserving interpretable intermediate representations and auditable reasoning stages. The results suggest that explicit modular spatial verification can serve as a promising building block for future report-oriented medical imaging agents.