Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction

AI summary

Overview

Research area: Multilingual financial NLP — information extraction from financial disclosures, specifically linking numerical facts to IFRS accounting concepts across English, French and Greek reports.

Technical level: Intermediate. The tasks are described in plain terms, but the paper assumes familiarity with benchmark evaluation, precision/recall/F1, and accounting taxonomy concepts.

Scope (1 sentence): The paper introduces MultiFinGround, a benchmark built from FY2025 regulatory filings of 90 issuers in three languages, and shows that strong cross-language concept recognition does not imply complete recovery of the corresponding numerical evidence.

What This Paper Is About

Analysts comparing companies across countries need the same accounting facts from reports written in different languages. A model may correctly identify that a number represents "revenue" while still failing to retrieve every revenue figure in the passage — for example, the prior-year comparative needed to compute growth. Existing benchmarks test concept linking, extraction or reasoning, but not whether both succeed on the same multilingual evidence. This paper builds a benchmark that measures concept recognition and exhaustive fact recovery together on matched contexts, exposing omissions that recognition scores alone hide.

Key Contributions

1. A taxonomy-linked multilingual benchmark. MultiFinGround is constructed from FY2025 European Single Electronic Format (ESEF) reports from 90 issuers (30 each in English, French and Greek, each issuer appearing in only one language), aligned through shared IFRS Accounting Taxonomy concepts and annotated at the fact level using issuer-provided Inline XBRL tags, with traceable annotations covering 187 concepts.

2. Paired tasks that separate recognition from evidence recovery. Concept Recognition (T2a) and Exhaustive Fact Selection (T2b) run on the same 521 context groups as independent, stateless calls, so the paper can measure how often a model recognizes a concept correctly yet omits target facts in the same contexts. Two complementary tasks — Fact Identification (T1) with accounting attributes and Concept Discrimination (T2c) without candidate labels — probe adjacent difficulties.

3. A 15-model evaluation exposing the recognition–grounding gap. Eight API and open-weight reference models reach 97.3–99.4% recognition accuracy but miss at least one target fact in 26.6–41.9% of correctly recognized groups.

4. Diagnostic analyses of language, label presentation and specialization. Controlled variants vary which languages appear in the evidence and how taxonomy labels are shown (concept identifier plus English label; French/Greek ESMA labels only; all three languages), and compare finance-specialized models against their base models.

Main Findings

• Recognition is high, coverage is not. Across 521 groups, the eight reference models score 97.3–99.4% on T2a accuracy, yet the best T2b results are 90.4 concept-macro F1 and 71.8% exact-set accuracy (Qwen3.5-122B-A10B). Among correctly recognized groups, the share missing at least one target in some language ranges from 26.6% (Qwen3.5-122B-A10B) to 41.9% (Qwen3.8-27B).

• Omissions are not caused by over-selection. Per-language precision stays at 88.5–100% for the reference models while recall falls to 51.6–83.5%, indicating that failures are missed facts rather than indiscriminate selection.

• Greek is the weakest evidence language. Greek has the lowest T2b recall for seven of the eight reference models.

• More targets, more omissions. For Gemini 3.8 Flash the omission rate rises from 27.7% for recognized groups with 3–6 targets to 46.3% for groups with 7 or more; the paper notes this comparison does not isolate the effect of target count.

• Removing English barely hurts strong models. For reference models, T2a accuracy changes by −0.8 to +0.4 points when the English excerpt is removed. Finance models and bases are more sensitive: Fin-R1 loses 9.2 points, Qwen2.5-7B-Instruct 5.6, Llama-Krikri-8B-Instruct 3.8 and Plutus-8B-Instruct 3.3.

• English-only is harder than no-English. Seven of the eight reference models score 1.9–3.1 points below the joint condition when only English evidence is kept (McNemar p < 0.05); GPT-5.6 Luna drops 0.8 points (not significant). French-only scores stay within 1.2 points of joint.

• Recognition does not transfer to discrimination. T2c accuracy spans 35.6–95.1%. Gemini 3.8 Flash reaches 95.1%, Qwen3.5-122B-A10B 91.6% and Gemma 4 31B IT 89.0%, while DeepSeek-V4.1-Flash scores 98.3% on T2a but 35.6% on T2c. Every model clearly above chance is less accurate when another excerpt shares the odd excerpt's language.

• Localized taxonomy labels do not reliably help. On Greek-only items, showing the official Greek ESMA label without the identifier lowers accuracy for eight models by 1.7–5.0 points; only the Greek-adapted Plutus-8B-Instruct improves significantly (+5.8, p = 0.007). French-label effects are smaller and vary by model, and the three-language label condition changes reference-model scores by at most 1.0 point (none significant).

• T1 errors concentrate in accounting roles, not number spotting. Conditional on correct fact identification, reference models reach at least 99.9% type accuracy and 98.4% period accuracy, but balance accuracy is only 80.6–95.7%, with debit-balance flows the weakest class.

• Finance specialization gives little benefit here. All four finance-specialized models perform worse than their base models on T1, T2a and T2c. Fin-R1 improves T2b by 9.9 macro-F1 points, with gains of 12.5 F1 in French and 14.8 in Greek; Plutus-8B-Instruct scores 25.4 points below its base on Greek T1, with 80% of its T1 outputs invalid, most of them empty.

• Gains are not explained by simple word matching. A lexical-overlap baseline scores 70.2% on T2a; harder distractors cut it to 49.1% with four options and 36.3% with twenty, while reference models retain 94.4–98.7% and 89.4–97.5% respectively.

• Generation failures affect rankings. Qwen3.8 models exhaust their token budget on 23–34% of T1/T2b/T2c items, ODA-Fin-RL-8B on 43% of T2c, and Plutus frequently returns empty answers; strict scoring counts these as wrong.

• Context for scale. For reference, random baselines score 8.8 on T1 macro-F1 and 23.8 on T2a, while the majority/select-all baseline scores 7.4 on T1.

Methodology in Plain English

The researchers started from official European regulatory filings (ESEF) for fiscal year 2025. Because companies must tag their disclosed numbers with IFRS Accounting Taxonomy concepts using Inline XBRL, each number already carries an expert-assigned accounting meaning. The team rendered the reports as text and Markdown tables, kept each fact's position in the document, and used the shared IFRS concepts to link excerpts from different issuers writing in English, French and Greek — so the excerpts are genuine independent disclosures, not translations of one another. Models never see the original XBRL tags.

Four separate tasks were built on this data. In Fact Identification (T1), every number in a passage or table is labelled with an identifier, and the model must say which are financial facts and give each one's type (monetary, per share, shares), period type (instant or duration) and balance (debit, credit, or none). In Concept Recognition (T2a), the target facts are marked across three-language contexts and the model picks the shared concept from four candidate labels. In Exhaustive Fact Selection (T2b), the model is given the concept and must return all corresponding numbered facts, including prior-year comparatives. In Concept Discrimination (T2c), four excerpts are shown, three marking one concept and one marking a confusable concept, and the model must find the odd one out with no candidate labels offered.

Quality control used two financial experts who independently reviewed a stratified 10% sample per task (131 T1 items, 52 shared T2a/T2b groups, 53 T2c items), with agreement of κ = 0.87, 0.74, 0.84 and 0.88 respectively before discussion. The audit corrected 1.3% of T1 numbers and 2.1% of T2b facts, flagged 5.1% of T2a distractors as ambiguous, and confirmed the odd excerpt for 96.2% of audited T2c items; swapping in corrected annotations changed reference-model scores by at most 1.4 percentage points on audited subsets.

Fifteen models were queried with identical English instructions plus original-language evidence, no system prompt, no tools and no retrieval, with a 16,384-token output budget. Prompts averaged 1.2k input tokens (maximum 3.7k). Outputs that failed, ran out of budget, refused or broke the format were scored as wrong, and invalid rates are reported next to every score.

Why This Matters

Impact on research. The paper argues that concept-recognition accuracy, the usual headline metric for financial fact linking, is a poor proxy for whether a model actually retrieves the evidence. Reporting recognition and coverage jointly on matched instances gives a more diagnostic evaluation protocol, and the finding that finance-tuned models do not reliably beat their bases under this protocol is a caution for the specialization literature.

Real-world applications:

• Cross-border investment analysis. Comparing revenue, assets or per-share figures across companies filing in different languages, where missing the prior-year comparative breaks growth calculations.
• Regulatory and disclosure monitoring. Checking whether tagged disclosures are consistently recoverable, given that filings already carry machine-readable iXBRL annotations.
• Financial data pipeline QA. Using exhaustive-selection recall as an audit signal for automated extraction systems rather than spot-checking single values.
• Multilingual document assistants. Setting expectations for tools that answer questions over non-English reports, where Greek evidence in particular shows the lowest recall.

Industry relevance. The benchmark is built directly on the ESEF/iXBRL infrastructure that issuers already produce, and the paper explicitly notes that benchmark scores do not establish suitability for financial reporting compliance and do not replace professional review. The practical message for vendors is that a model claiming 98% concept accuracy may still drop a quarter or more of the needed numbers when they appear in multiple languages.

Future Directions

• Diagnosing why coverage lags recognition. The paper shows omissions rise with the number of targets (27.7% to 46.3% for Gemini on 3–6 versus 7+ targets) but states this does not isolate the cause; controlled experiments on target count and context length are the natural next step.
• Broadening languages and regimes. The limitations note that the benchmark covers three languages under a single reporting regime, and that language co-varies with issuer, sector and document style, so differences between single-language conditions cannot be attributed to language alone. Extending to more languages and reporting standards would test generality.
• Better treatment of untagged and ambiguous numbers. T1 leaves untagged amount-like numbers unscored, and most T2 contexts are table windows dominated by common primary-statement concepts; richer annotation could capture harder cases.
• Rethinking finance-specialized training. Since all four finance models underperform their bases on T1, T2a and T2c while Fin-R1 gains 9.9 macro-F1 on T2b, it is an open question whether specialization should target exhaustive selection specifically rather than concept knowledge.

Target Audience

Researchers and practitioners in financial NLP and information extraction who build or evaluate multilingual document systems; benchmark designers interested in paired, evidence-completeness evaluation; and data or applied-AI teams working with regulatory filings (iXBRL/ESEF), XBRL tagging, or cross-lingual retrieval over corporate reports. Readers focused on accounting-domain model fine-tuning will find the specialization results directly relevant.

Authors’ abstract

Comparing financial disclosures across languages requires identifying shared accounting concepts and recovering all numerical facts that express them. Existing benchmarks evaluate aspects of these capabilities, but provide limited insight into whether both succeed on the same multilingual evidence. In particular, it remains unclear whether language models can recognize these concepts across languages and exhaustively recover their corresponding numerical evidence from the same contexts. We introduce MULTIFINGROUND, a benchmark designed to investigate this recognition-grounding gap in multilingual financial reports. Constructed from FY2025 disclosures of 90 issuers in English, French, and Greek, MULTIFINGROUND aligns independently authored reports through a shared accounting taxonomy and provides traceable fact-level annotations. The benchmark evaluates financial fact identification, concept recognition, exhaustive fact selection, and concept discrimination without candidate labels. Across 15 evaluated models, eight API and open-weight reference models achieve 97.3-99.4\% concept recognition accuracy, yet miss at least one target fact in 26.6-41.9\% of correctly recognized groups. Further analyses reveal substantial variation in concept discrimination and limited gains from the evaluated finance-specialized models. Our findings demonstrate that strong concept recognition can coexist with incomplete evidence recovery, highlighting the importance of evaluating semantic recognition and evidence completeness together.