InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

AI summary

Overview

Research area: Multimodal agent evaluation — specifically vision-language models (VLMs) performing claim verification inside interactive data visualizations.

Technical level: Intermediate. The benchmark design, metric, and experimental setup are explained accessibly, but familiarity with VLM agents, chart-understanding benchmarks, and evaluation metrics helps.

One-sentence scope: The paper introduces InSight, a 21,349-claim benchmark in which agents must actively click, hover, and scroll through real interactive Vega-Lite visualization notebooks to label natural-language claims as True, False, or Not Enough Information, and evaluates state-of-the-art models on both accuracy and the quality of their interaction traces.

What This Paper Is About

Existing chart-understanding and fact-checking benchmarks show a model a static image and ask a single question, which ignores how real data analysis works: evidence is hidden behind tooltips, spread across linked views, or revealed only after filtering and zooming. The authors build a benchmark where a model must navigate a live web-based visualization environment, decide whether a claim is supported, refuted, or unsupported by available evidence, and have its sequence of mouse and keyboard actions treated as an auditable record of its reasoning.

Key Contributions

1. A new task formulation — Interactive Visual Claim Verification. The paper extends text-based fact-checking (with its three-way True / False / Not Enough Information framing) into partially observable, multimodal, agentic environments where evidence must be actively acquired through a bounded action budget.

2. The InSight benchmark. 21,349 claims grounded in 297 human-authored analytical notebooks containing custom Vega-Lite visualizations. Unlike most prior datasets, claims come from real analyst narratives rather than synthetic templates, and the environments are fully interactive (average 4.5 views and 3.7 visualization specifications per notebook; 67% exhibit multi-view composition).

3. The Interaction Efficiency Score (IES). A metric that multiplies answer accuracy by the ratio of state-changing actions to total actions, so that correct answers produced without any interaction score zero. This is designed to expose correct-but-ungrounded predictions that accuracy alone would hide.

4. A granular evaluation of 14 model configurations across Gemini, GPT, Gemma, and Qwen families, including action-budget ablations (1, 10, 25 turns), label-disaggregated results, action-type profiles, and a taxonomy of failure modes.

Main Findings

• Interactive verification is far from solved. On a stratified 500-claim subset, the best model (GPT-5.5, 10-action budget) reaches only 57.2% accuracy against a 33.3% random baseline. Gemma 4 31B reaches 47.2% and Qwen 3.5 27B reaches 45.0%.

• Capability scales with model size, with a floor. Qwen accuracy falls monotonically from 44.8% (35B) to 34.0% (0.8B); Gemma drops from 47.2% (31B) to 31.2% (E2B). Sub-billion-parameter models barely exceed chance and appear to lack the minimum capacity for the task.

• Interaction can actively hurt. At a one-turn "no-interaction" baseline, Gemini 3.5 Flash (44.2%) and Qwen 3.5 27B (46.2%) score slightly higher than with a 10-action budget. Gemini's budget ablation is non-monotonic: 44.2% at 1 action, 41.6% at 10, and 50.0% at 25. A partially completed survey of a multi-view environment seems worse than none — the model has enough evidence to abandon its prior but not enough to replace it.

• IES separates accuracy from groundedness. Small models achieve near-zero IES (Gemma 4 E2B: 0.90%; Qwen 3.5 0.8B: 0.00%) even when they occasionally answer correctly, confirming they are not engaging with the environment. Gemma 4 31B matches Gemini 3.5 Flash on accuracy but scores lower on IES because a smaller fraction of its actions change environmental state.

• Falsification is the hardest operation. For most models, NEI claims yield the highest IES and False claims the lowest (e.g., Gemini 3.5 Flash: IES-NEI 32.63% vs. IES-True 18.63%). Recognizing the absence of evidence is easier than finding specific evidence that contradicts a linguistically plausible statement. This extends a known pattern from textual fact-checking into the agentic multimodal setting.

• Distinct interaction signatures emerge. Gemini 3.5 Flash is the most active agent (6.27 actions/episode, 61.3% effective), using scroll (2.51/sample), hover (1.42), click (1.15), and page-down (0.73). GPT-5.5 is economical (3.13 actions) yet most accurate, favoring hover (1.07) and scroll (1.02) — suggesting targeted rather than broad exploration.

• Three recurring failure modes. Premature commitment (small models answer after one or two actions), unfocused exploration (many actions with low effective ratios), and action-type rigidity (mid-range models default to scroll and click, skipping hover, which is the primary mechanism for revealing tooltips and hidden values).

• Human validation supports the labels. Thirteen expert annotators produced 475 annotations over 294 sampled claims with 81.3% raw agreement with dataset labels. A Bayesian Dawid–Skene model recovered class prevalences within 3 percentage points of ground truth, with posterior mean accuracy 66.2% (95% CI [59.5%, 71.8%]).

Methodology in Plain English

The authors took existing graduate-level data visualization projects — notebooks where trained analysts paired custom Vega-Lite charts with written narratives about what they found — and turned those narratives into verifiable claims.

They processed the corpus in four stages. First, they extracted candidate spans from each narrative using a four-level semantic model, keeping only statistical and perceptual insights and requiring agreement across three independent LLM runs to filter hallucinations. Second, they decomposed compound spans into atomic, self-contained propositions, verified each is entailed by the source text, and assigned labels by majority vote. Third, they generated balanced False and NEI examples by controlled mutation: swapping antonyms (e.g., "increased" to "decreased"), swapping in other values from the same dataset (producing False), and swapping in values absent from the dataset (producing NEI). Each mutated claim was checked with a natural language inference model — contradictions needed scores above 0.9, NEI claims needed both low entailment and low contradiction (below 0.2). Fourth, human annotators performed the same task blind to labels to validate the pipeline.

For evaluation, each notebook renders as a real HTML page in a headless Chromium browser at a fixed viewport deliberately smaller than the page, so nothing relevant is ever visible in the initial screenshot. Models see only the RGB screenshot — no DOM, HTML, or accessibility tree. The action space covers scroll, page up/down, arrow keys, drag, click, shift-click, hover, and an answer action. Episodes run up to a maximum turn budget. IES is then computed per episode as accuracy multiplied by the fraction of actions that produced an observable state change, which is zero for any agent that answers immediately.

Why This Matters

Impact on research. The paper reframes visual reasoning as sequential decision-making and offers a concrete mechanism — interaction traces — for auditing whether a model's answer is grounded in what it actually saw. It also gives a metric that distinguishes "right for the wrong reasons" from genuine evidence-based inference, a gap that static benchmarks and accuracy scores cannot fill.

Real-world applications:

• Business intelligence and dashboard assistants. Copilots that answer questions about live Tableau or Power BI dashboards must find the right view, trigger tooltips, and filter before answering — exactly the InSight task.
• Automated fact-checking of data journalism. Verifying whether a chart in a news article supports its headline requires reading interactive graphics, not flat PNGs.
• Scientific and public-health communication. Claims drawn from dashboards (case trends, regional comparisons) need verification against the underlying interactivity, especially where absence of evidence matters.
• Accessibility and QA tooling. Interaction traces can reveal whether an automated agent genuinely explored a visualization or merely guessed, informing reliability assessments for deployed assistants.

Industry relevance. Organizations deploying GUI and browser agents (customer support, form automation, analytics copilots) all depend on the same capabilities InSight stress-tests: grounding in pixels rather than source code, long-horizon exploration under a budget, and knowing when to stop. The finding that more interaction can reduce accuracy is a direct warning against naive "give the agent more turns" scaling.

Future Directions

• Training for evidence-seeking behavior. The gap between static and interactive performance suggests models need explicit training on hypothesis formation and evidence acquisition, not just one-shot visual question answering.

• Understanding falsification. Why models struggle to find disconfirming evidence — and how they form and revise hypotheses across multi-turn exploration — remains an open question with direct implications for misinformation detection.

• Finer-grained grounding metrics. The authors acknowledge that IES credits any state-changing action regardless of whether the revealed evidence informed the final answer. Claim-specific evidence annotations would allow a stricter test of whether a correct answer depended on what was actually observed.

• Broader action spaces and interface generality. The current high-level action space abstracts mouse and navigation primitives. Richer modalities (complex gestures, semantic interface shortcuts) and different action-space designs could yield different interaction strategies and performance profiles, which the authors flag as unexplored.

Target Audience

Researchers and engineers working on multimodal agents, chart and visualization understanding, and automated fact-checking will find the benchmark and metric directly useful. Practitioners building analytics copilots or GUI automation will benefit from the empirical findings on interaction budgets and action-type usage. Benchmark designers and evaluation researchers will value the IES formulation and the human-validation protocol. The paper is accessible to graduate students with some background in vision-language models, though the task framing rewards readers already familiar with agentic environments such as OSWorld or VisualWebArena.

Authors’ abstract

Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.