X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

AI summary

Overview

Research area: Computer vision and multimodal medical AI — specifically tool-augmented vision-language agents for chest X-ray interpretation, trained with reinforcement learning.

Technical level: Intermediate. The paper assumes familiarity with multimodal large language models, supervised fine-tuning, and policy-gradient reinforcement learning (GRPO), but its core ideas — when to call a tool, and whether a measurement is valid — are described in accessible terms.

Scope: The paper introduces X-Ragent, a two-stage SFT + GRPO post-training framework that teaches an 8B vision-language model to selectively acquire auxiliary evidence and to accept quantitative radiographic measurements only when a validity gate permits, evaluated on three chest X-ray benchmarks.

What This Paper Is About

Chest X-ray models can describe images but struggle to connect what they see to the localized anatomical evidence and numeric measurements a diagnosis actually requires. Tool-augmented agents can supply that evidence, but existing systems lack a mechanism for deciding when extra evidence is needed, and they sometimes override tool outputs with textual priors or apply measurements under acquisition conditions where those measurements are meaningless. X-Ragent addresses this by separating two things: which optional evidence the model learns to seek, and whether deterministic quantitative measurements are allowed into the reasoning at all.

Key Contributions

1. Evidence-grounded CXR reasoning framework. X-Ragent decouples learned auxiliary-evidence seeking (visual refinement and knowledge lookup, chosen by the policy) from validity-gated quantitative measurement acquisition (cardiothoracic ratio, lung-field asymmetry, thoracic shift, exposed only when a view_check applicability test passes).

2. Structured agentic post-training. The authors build Rad-CoT, a 7,715-sample corpus of causally ordered tool-augmented trajectories plus report-style diagnostic reasoning, and use it for agentic supervised fine-tuning of Qwen3-VL-8B-Instruct, followed by GRPO with an accuracy-gated reward: process terms for measurement consistency (R_CXR) and tool-use cost (R_tool) apply only to trajectories whose answer is already correct.

3. Empirical effectiveness across three CXR benchmarks. X-Ragent reports the strongest results on ChestAgentBench (fully held out), CheXbench, and SLAKE, including raising ChestAgentBench accuracy from 63.1% to 71.7% over the strongest baseline, with gains on all seven CXR interpretation capabilities.

4. Diagnostic analysis of selective routing and validity gating. The paper quantifies that the learned policy invokes tools more often on questions the base model gets wrong, and that withholding inapplicable measurements improves accuracy specifically on non-frontal or atypical radiographs.

Main Findings

• State-of-the-art on all three benchmarks. X-Ragent reaches 71.7% on the fully held-out ChestAgentBench, an average of 77.3% across the three CheXbench subsets, and 93.0% on standalone SLAKE. It improves over the best reported results by 8.6, 9.2, and 2.6 points on ChestAgentBench, CheXbench, and SLAKE respectively.

• Large gains over the base model let an 8B model beat bigger systems. Relative to its Qwen3-VL-8B base model, post-training adds 16.6, 11.8, and 20.2 points on the three benchmarks, surpassing GPT-4o-based MedRAX and Llama-3.2-90B-Vision. On CheXbench, gains come from the SLAKE (+19.5) and OpenI reasoning (+15.8) subsets; on the 115-question Rad-Restruct subset X-Ragent matches its base model at 70.4%.

• Out-of-domain transfer. ChestAgentBench is entirely excluded from both SFT and RL training, yet X-Ragent outperforms the strongest baseline on all seven capabilities (detection 73.8, classification 70.7, localization 73.3, comparison 70.5, relationship 72.9, diagnosis 69.3, characterization 72.1) and improves over its base model by 14.6–18.3 points on every capability.

• SFT is the bigger single gain; RL mainly improves transfer. In the pipeline ablation, SFT alone yields the largest single jump (+10.8 average), while adding RL contributes more on held-out ChestAgentBench (+3.0) than on in-domain CheXbench (+1.2). RL-only training falls 6.3 points below SFT-only, which the authors attribute to the difficulty of eliciting structured reasoning from reward alone.

• Tool access helps most where reasoning is multi-step. Enabling tools adds 4.3 points on ChestAgentBench and 4.1 on CheXbench but only 0.9 on SLAKE, which is already near saturation at 92.1% (and where one item on the 114-question test set is worth 0.88%).

• Gating process rewards by correctness matters. Adding R_tool and R_CXR without the accuracy gate (69.8 / 75.6 / 91.2) underperforms using either term alone (70.1 / 75.9 / 92.1 and 70.6 / 76.4 / 92.1). The gated multiplicative form (Eq. 7) gives the best results on all three benchmarks: 71.7 / 77.3 / 93.0.

• Routing is selective, not uniform. On ChestAgentBench, X-Ragent invokes auxiliary tools on 64.9% of questions the base model missed versus 43.6% of questions it answers correctly (1.49×). On CheXbench the rates are 61.5% versus 45.9% (1.34×). Corrections outnumber regressions, with rescue-to-damage ratios of 3.15× and 2.58×.

• Validity gating helps exactly where it should. view_check prevents measurement execution for 24.5% of inputs, with 31% ultimately withheld after a mask-quality check. In a controlled comparison using the same measurement backend, gating raises accuracy on 613 non-frontal or atypical CXRs from 69.17% to 70.15% (+0.98 points), leaves 1,887 frontal CXRs unchanged at 72.18%, and lifts overall accuracy from 71.44% to 71.68% (+0.24 points). Paired changes show 8 corrections and 8 regressions on frontal inputs versus 9 corrections and 3 regressions on non-frontal/atypical inputs — all six net corrections come from the validity-sensitive subset.

• Inapplicable measurements can flip a correct answer. A case study shows that exposing a lung-field asymmetry measurement of ln(A_R/A_L) = −1.01 shifts the prediction from the correct option C to the incorrect option A, whereas withholding the invalid bundle preserves the correct answer.

• Data composition matters differently per subset. Removing tool-trajectory supervision lowers accuracy by 4.5, 6.1, and 7.0 points on ChestAgentBench, CheXbench, and SLAKE. Removing report-style CoT causes the largest drop on held-out ChestAgentBench (5.5 points), while removing VQA supervision mainly hurts SLAKE (−10.5 points), consistent with its closed-ended VQA format.

Methodology in Plain English

The researchers treat chest X-ray interpretation as a two-turn conversation with an agent rather than a single image-to-answer prediction.

Turn one — triage. Given the radiograph and a clinical question, the model gives a global assessment and then decides whether to call any of three optional tools: region_parse (localized anatomical inspection), roi_crop (magnifying a region), and evidence_lookup (retrieving condition-specific radiographic knowledge from a locally served Qwen3.5-27B). It may call none of them, and at most five.

A separate, non-learned path for numbers. In parallel, a deterministic backend computes the cardiothoracic ratio, lung-field asymmetry, and thoracic shift — but a view_check gate decides whether those measurements are exposed at all, based on acquisition suitability rather than model preference.

Turn two — integration. The model receives whatever observations the tools returned plus, only if the gate passed, the measurement bundle, and then produces the reasoning and final answer.

Training happens in two stages. First, agentic supervised fine-tuning on Rad-CoT: 7,715 examples built from MIMIC-CXR-JPG PA radiographs with LLaVA-Rad-derived findings and public medical VQA data, containing 1,995 multiple-choice, 720 closed-ended VQA, 1,000 binary-choice VQA, and 4,000 report-style diagnostic CoT examples. The report-style subset was synthesized with a locally deployed Qwen3.5-122B-A10B-FP8 teacher, then filtered down from 6,000 candidates to 4,000 by rejecting leakage, malformed reasoning, out-of-range lengths (rationales must be 40–900 words), and rationales too similar to the reference findings (similarity above 0.82 rejected). Crucially, tool observations are inserted only after the call is generated, preserving inference-time causal ordering, and the SFT loss masks those injected observations so the model is not trained to predict them.

Second, GRPO on 4,715 prompts for 300 steps with 4 rollouts per prompt. The reward is R(τ) = R_format(τ) + R_acc(τ)[1 + λ_m R_CXR(τ) − λ_t R_tool(τ)], where λ_m = 0.5 and λ_t = 0.05. The multiplicative accuracy gate means process-quality rewards for measurement consistency and tool economy can only re-rank trajectories that already produced the right answer — so a plausible-looking process cannot compensate for a wrong diagnosis. Because λ_t·K_max < 1, a well-formed correct trajectory always outranks an incorrect one.

Why This Matters

The paper targets a failure mode that is specific and clinically consequential: a model that cites a measurement it should never have applied, or that ignores a tool it just called in favor of its own textual prior. By making evidence acquisition a learned decision and measurement admission a deterministic check, the framework separates what the model is good at learning from what should not be left to a learned policy. The result is a small (8B) model that outperforms substantially larger general-purpose VLMs and a GPT-4o-based medical agent on several chest X-ray tasks, which matters for cost and deployability.

Real-world applications:

• Radiology triage support, where a model flags findings on chest radiographs and must not report measurements taken under incompatible acquisition conditions.
• Report drafting assistance, since X-Ragent is trained on report-style diagnostic reasoning and produces structured reasoning alongside its answer.
• Quality control for existing AI readers, by supplying a validity-gate pattern that can be attached to measurement outputs from other systems.
• Resource-constrained deployment, where an 8B model with local tool servers (a local Qwen3.5-27B for knowledge lookup) avoids sending data to third-party services — the ethics statement notes all LLMs were deployed locally so no MIMIC data left the environment.

Industry relevance: The gated-reward design is a general recipe for domains where process quality is easy to reward but easy to game — it shows that gating process terms behind outcome correctness prevents reward hacking. The separation of learned tool routing from a deterministic validity backend is also a practical architecture pattern for regulated or safety-critical AI, where some decisions should remain rule-based rather than learned.

Future Directions

• Generalizing the validity gate beyond the current checks. view_check and the mask-quality check govern three measurements (cardiothoracic ratio, lung-field asymmetry, thoracic shift). Whether the same gating pattern extends to other measurements and other imaging modalities is not reported.

• Closing the Rad-Restruct gap. X-Ragent matches rather than improves on its base model at 70.4% on the 115-question Rad-Restruct subset, so some visual VQA behavior appears untouched by post-training. The paper does not explain why.

• Scaling the tool set and interaction depth. The action set is three tools with K_max = 5 and two turns. Whether longer horizons or a broader tool library yield further gains is an open question, especially because visual-refinement and knowledge-lookup tools receive no explicit process supervision and are optimized only indirectly through task success.

• Extending out-of-domain evaluation. ChestAgentBench provides one fully held-out benchmark built from Eurorad cases. Whether the reported transfer holds across other institutions, scanners, and patient populations is not reported and would require additional held-out sets.

• Understanding the SFT/RL division of labor. The paper's finding that SFT supplies the core reasoning behavior while RL mainly improves transfer (a 6.3-point shortfall for RL-only) invites further work on cold-start design and on why reward alone fails to elicit structured tool use.

Target Audience

This paper is most useful to researchers and engineers working on multimodal medical AI, tool-augmented LLM agents, and reinforcement-learning post-training for vision-language models. It is also relevant to clinical AI product teams interested in validity checks for quantitative outputs, and to RL practitioners looking for concrete evidence about how to structure reward terms so process supervision cannot override correctness. Readers without a background in vision-language models or policy optimization will find the problem framing and ablation results accessible, but will need familiarity with GRPO and multimodal architecture to follow the methodology in detail. The paper states X-Ragent is a research prototype, not a medical device, and should not be used for clinical decision-making.

Authors’ abstract

Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR reasoning.