E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

AI summary

Overview

Research area: Evaluation of vision–language models (VLMs) for financial chart reasoning, at the intersection of multimodal NLP, hallucination benchmarking, and financial technical analysis.

Technical level: Intermediate. The framing and motivation are accessible, but the scoring protocol (four chained metrics with a multiplicative reliability term) assumes some familiarity with hallucination metrics and classification-style diagnostic scores.

Scope: The paper introduces E2A-Bench, a 969-query benchmark that scores whether financial VLMs carry chart evidence reliably through claims, rationale, confidence, and a final BUY/SELL/HOLD/UNCERTAIN action, rather than scoring isolated claims or final-answer accuracy.

What This Paper Is About

Existing chart hallucination benchmarks are claim-centric (is a generated statement supported?) and financial multimodal benchmarks are task-centric (is the final answer correct?), so neither checks whether evidence stays traceable from chart facts through reasoning to a decision. That gap creates two blind spots: models can look reliable by refusing to make directional calls (coverage blindness), and evaluations never verify that the action and confidence actually follow from the stated rationale (chain blindness). E2A-Bench addresses both by evaluating the full evidence-to-action chain while explicitly measuring whether a model still commits to actionable BUY or SELL recommendations.

Key Contributions

1. E2A-Bench, a 969-query benchmark for coverage-aware evidence-to-action reliability in financial chart reasoning, built from 323 HS300 constituents across three input modalities (text-only, chart-only, multimodal), with a 30-trading-day OHLCV window per query and ten deterministic evidence anchors yielding 3,230 ground-truth facts.

2. A chain-level protocol covering four links: evidence grounding (UCR, Unsupported Claim Rate), reasoning–action consistency (RCI, Reasoning–Action Consistency Inconsistency), evidence–confidence calibration (ECI, Evidence–Confidence Inconsistency), and directional coverage, combined into NDR (Net Decision Reliability) as coverage multiplied by chain reliability — Cov_dir,θ · (1 − UCR_dir)(1 − RCI)(1 − ECI), with θ = 0 in the main paper.

3. An evaluation of 20 VLMs spanning proprietary, open-weight, and financially fine-tuned systems, uncovering three failures hidden by scalar hallucination scores: coverage-driven rank reversal, verification-induced coverage collapse, and BUY-side amplification after financial fine-tuning.

4. A framing of the "coverage–grounding frontier," arguing that reliability should be judged by whether a model grounds its recommendations while preserving actionable coverage, not by unsupported-claim reduction alone.

Main Findings

• Low hallucination does not imply reliable action. Gemma-4-E2B-It has the lowest overall unsupported-claim rate (UCR_all = 39.9%) but only 6.4% directional coverage and NDR = 5.1%, whereas Qwen2.5-VL-3B-Instruct reaches the highest NDR (32.1%) with 82.5% coverage.

• Model families show distinct reliability profiles. Open-weight models rank highest by NDR (Qwen2.5-VL-3B-Instruct at 32.1%, Qwen2.5-VL-7B-Instruct at 25.0%). The best proprietary model, Gemini-3.1-Pro, reaches 16.8% NDR. Financially adapted models are uneven: PyFi-QwenVL-3B-COT-47K reaches 25.3% NDR but has a high reasoning–action inconsistency rate (26.8), while Amsi-fin-o1 has high coverage (81.3%) but weaker grounding and calibration.

• Modality effects are model-dependent. Chart-only input gives the strongest average baseline NDR across all 20 models (19.2%), ahead of text-only (14.5%) and multimodal (15.6%), but the best modality differs per model: Qwen3.5-4B peaks with text-only input (37.9%), while Gemma-4-31B and Qwen3.5-2B peak with chart-only input (29.4% and 24.1%).

• Oracle-aided verification can collapse coverage. In chart-only intervention analysis, oracle-aided verification sharply reduces unsupported claims but reduces actionable coverage enough to yield a negative mean change in NDR.

• Financial fine-tuning redistributes risk rather than uniformly reducing it. Across strict base–fine-tuned pairs, NDR changes are inconsistent: Amsi-fin-o1 improves NDR by +1.5 percentage points, while PyFi-QwenVL-3B and PyFi-QwenVL-7B decrease it by 6.8 and 7.2 percentage points respectively.

• BUY-side amplification is consistent across fine-tunes. All three pairs shift strongly toward BUY, with BUY:SELL ratios rising from 1.41 to 6.10, from 0.85 to 4.00, and from 1.84 to 7.73 — a 4.21–4.68× amplification. Across the pairs, 455 of 2,907 paired responses shift from HOLD/UNCERTAIN/unparseable outputs to BUY, while matched evidence claims decrease by roughly 34–40%.

• Scaling is not monotonic. Holding the Qwen3.5 family fixed on chart-only inputs, UCR rises from 57.0 to 71.0 from 2B to 27B, and NDR peaks at 2B (24.1), drops at 4B (12.6), partially recovers at 9B (18.1), and declines again at 27B (13.1).

• Non-action dominates parsed outputs. Among 55,209 successfully parsed actions, HOLD and UNCERTAIN account for 63.6%.

• Human validation supports NDR. In a blinded pilot, NDR shows the strongest correlation with holistic human reliability judgments among the compared metrics (ρ = 0.466, p = 0.009), and principal model-level findings remain stable across 1,000 Monte Carlo perturbations of scorer outputs. The authors also audited 200 randomly sampled chart–label pairs and observed no inconsistency between rendered charts and their OHLCV-derived labels.

Methodology in Plain English

The authors start from the observation that a financial chart never implies one correct trade, so they avoid labeling ground-truth buy/sell actions at all. Instead, for each of 323 HS300 stocks they take a 30-trading-day window of open, high, low, close, and volume (OHLCV) data and compute ten deterministic facts using fixed rules — things like moving-average alignment, golden/death crosses, volume relative to its 20-day mean, breakouts outside the prior 20-day high–low band, regression slope of closing prices, annualized volatility, 5-day price change, volume–price divergence, price position within the 20-day range, and 3-day momentum acceleration. These facts are the "evidence anchors" (3,230 in total).

Each query is formatted three ways: text-only (serialized OHLCV records), chart-only (a candlestick chart with moving averages and volume bars), and multimodal (both). A model must return a technical analysis, a rationale, an action from {BUY, SELL, HOLD, UNCERTAIN}, and a confidence score from 1 to 5.

The response is then parsed into verifiable claims, a coarse rationale polarity (bullish, bearish, neutral, unknown), the action, and the confidence. Four links are scored. UCR checks what fraction of extracted claims are unsupported by their matched anchors. RCI flags a contradiction between rationale polarity and action (bullish rationale with SELL, or bearish with BUY). ECI flags a high-confidence (≥4) directional action when the anchors contain zero strong directional or extreme signals. Directional coverage measures how often the model commits to BUY or SELL at all. NDR multiplies coverage by the three chain-quality terms, treating a decision as unreliable if it is absent, unsupported, inconsistent, or overconfident under weak evidence. The authors note this multiplicative form is a necessary-condition view and does not assume statistical independence among factors.

Two comparison sets are evaluated: 20 VLMs across proprietary, open-weight, and financial fine-tune families, plus a stricter threshold analysis (θ > 0) that restricts coverage to directional actions backed by stronger chart evidence, reported in appendix section C.2. The authors also state that the protocol is modular — evidence anchors, claim patterns, and action schemas can be swapped while preserving the audit structure.

Why This Matters

Impact on research: The paper argues for shifting the evaluation unit from claims or final answers to evidence-backed action. It provides a template for diagnosing a specific failure mode — a model that looks safe because it abstains — that conventional hallucination rates cannot detect, and it shows that the widely reported practice of reporting one scalar hallucination number can misrank models by decision reliability.

Real-world applications:

• Selecting or comparing VLMs for chart-analysis and technical-signal tools, using coverage-aware reliability rather than a single hallucination score.
• Auditing financially fine-tuned models before deployment, specifically checking for systematic BUY-side skew introduced by adaptation.
• Reviewing or compliance-screening AI-generated trading commentary where an unsupported claim like a fabricated golden cross becomes the stated reason for a recommendation.
• Designing evidence-chain evaluations in other domains — the authors note that multi-level hallucination diagnostics and evidence-chain evaluation have been explored in tool use and scientific-document reasoning, and the protocol is explicitly modular.

Industry relevance: The paper is careful to state that NDR does not measure market forecasting or realized trading performance, and that a high score does not imply profitability, investment quality, or readiness for deployment. Rankings should therefore not be the sole basis for deploying systems in financial applications, and downstream use must follow applicable financial regulations and the licensing and redistribution requirements of the underlying market-data providers.

Future Directions

• Extend beyond a single market snapshot. E2A-Bench uses HS300 constituents at a fixed evaluation date (the paper does not report the specific date), so it does not yet cover bull, bear, and sideways regimes, other equity markets, other asset classes, or cross-market trading conventions.

• Improve claim detection coverage. The deterministic anchors and pattern-based claim detectors may miss rare paraphrases, nuanced technical claims, or domain-specific reasoning patterns; the authors suggest multilingual claim patterns, more robust semantic parsers, and professional adjudication for ambiguous cases while preserving evidence-grounded evaluation.

• Explain financial fine-tuning shifts. Since the fine-tuning corpora and objectives are not fully public, the BUY-side amplification and the 34–40% drop in matched evidence claims are treated as descriptive output shifts rather than causal explanations — an open question for follow-up work.

• Test whether the coverage–grounding frontier generalizes. The paper's central framing implies that the same coverage-versus-grounding trade-off should be measured in other decision-oriented multimodal settings, but this is not yet established.

Target Audience

Researchers working on multimodal model evaluation, hallucination measurement, and financial NLP; benchmark designers who need a worked example of chain-level, coverage-aware scoring; engineers and risk reviewers evaluating or fine-tuning VLMs for chart and market-commentary tasks; and practitioners who need to understand why a low hallucination rate alone does not certify that a model's recommendations are grounded, coherent, or calibrated.

Authors’ abstract

Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly claim-centric; they assess whether generated statements are supported, but not whether evidence remains traceable through rationale, confidence, and final action. We introduce E2A-Bench, a 969-query benchmark for financial chart reasoning, constructed from 323 HS300 constituents under three input modalities with deterministic OHLCV-derived evidence anchors. E2A-Bench evaluates grounding, reasoning-action consistency, evidence-confidence calibration, and directional coverage through UCR, RCI, ECI, and NDR, where NDR measures coverage-aware evidence-to-action reliability rather than realized trading performance. Evaluating 20 VLMs reveals three failures hidden by scalar hallucination scores: the lowest-UCR model ranks near the bottom by NDR due to only 6.4% directional coverage; oracle-aided verification reduces unsupported claims but can collapse coverage; and financial fine-tuning amplifies the BUY:SELL ratio by factors of 4.21 to 4.68 across strict base-fine-tuned pairs. These results show that financial VLM evaluation should trace the full evidence-to-action chain rather than rely on a single hallucination score. Code and data: https://github.com/wanng-ide/E2A-Bench