Fin_Agent at FinEval 2026: Task-Aware Methods for Multilingual Financial NLP

AI summary

Overview

Research area: Natural Language Processing — multilingual and low-resource financial NLP (named-entity recognition, pragmatic/implicit communication analysis, and retrieval-augmented question answering).

Technical level: Intermediate. The paper combines hand-written pattern rules with off-the-shelf large language models (Qwen2.5-32B, Qwen3-32B, BGE-M3); no model training is performed, but readers should be comfortable with NER, ROUGE scoring, retrieval, and quantization.

Scope: This is a system-description paper reporting one team's (Fin_Agent, University of North Texas) submissions to the three subtasks of the FinEval 2026 multilingual financial language understanding shared task, with leaderboard scores, ablations, and error analysis.

What This Paper Is About

The paper describes three separate systems built for a shared task on understanding financial language across languages: pulling structured entities out of Greek financial text, judging how strongly a Japanese company commits to a statement in investor Q&A, and answering analyst-style questions that require reading SEC filings plus news in five languages. The core problem is that financial NLP resources and benchmarks are concentrated in high-resource languages, and general-purpose multilingual LLMs perform unevenly on specialized financial terminology. The goal is to show that task-aware design, ranging from annotation-matched regex rules to handcrafted linguistic cues to retrieval-guided generation, can be competitive under limited supervision.

Key Contributions

1. Annotation-aware rules plus zero-shot LLM classification for Greek NER. The authors pair deterministic numeric extraction rules (matched to the released annotation boundaries) with Qwen2.5-32B-Instruct-AWQ classification of pre-generated textual candidate spans under asymmetric supervision, where GRFinNUM has labels and GRFinNER has none.
2. An interpretable, cue-based ordinal scorer for Japanese implicit communication. A rule-based scoring function over Japanese lexical cues (decisions, volition, conditionality, deferral, refusal, reopening, reaffirmation) produces five pseudo-logit scores for labels +2, +1, 0, −1, −2 without LLM inference, reaching 98.0% combined accuracy.
3. A retrieval-guided long-context QA pipeline with a controlled component analysis. BGE-M3 retrieval plus per-question Qwen3-32B generation, followed by rewriting and guards, is evaluated against a five-seed grid that separates the effects of retrieval and generation granularity.
4. A candid limitations and error audit, including stage-by-stage tracing of which post-processing step introduced which validator flags and a documented parameter count (33,329,878,016 total) for eligibility under the 70B limit.

Main Findings

• Greek NER leaderboard results: The hybrid system scored 0.9521 public F1 (rank 4) and 0.8884 private F1 (rank 6).
• Rules beat a neural baseline on numeric NER: On the 80-example validation split, the deterministic rules reached 0.9862 F1 versus 0.9287 for XLM-R-LARGE, a gap of 5.75 F1 points. On the 320-example training split the rules scored 0.9877 precision, 0.9683 recall, and 0.9779 micro-F1.
• Strong Japanese implicit communication accuracy: 15/15 correct on the public split (100.0%) and 34/35 on the private split (97.14%), for a combined 49/50 (98.0%) across 50 hidden test instances.
• Sinkhorn calibration had no effect: Sinkhorn normalization changed none of the 50 evaluation predictions, indicating the rule-based scorer alone determined the labels. On the fixed 26-example development split, raw argmax reached 0.3846 accuracy and 0.4000 macro-recall, while Sinkhorn reached 0.2692 accuracy and 0.3000 macro-recall.
• Modest multilingual QA scores: 0.3023 F1 (0.3040 precision, 0.3528 recall) on 24 public instances and 0.3419 F1 (0.3227 precision, 0.3899 recall) on 52 private instances.
• Public submission variants differ little: Replacing grouped generation without retrieval (F1 .2618, 37.6 mean words) by per-question generation with retrieval and rewriting (F1 .3023, 69.4 mean words) was the largest change in a table of five variants; the three separate rewrite variants differed by less than 0.008 F1.
• No stable advantage from retrieval or generation granularity in the controlled analysis: Across four configurations (grouped or per-question, with or without retrieval) run on 76 development instances with five seeds, mean F1 differed by less than 0.004, and every estimated paired effect was smaller than the variation across seeds. Example: Q reached 0.4564 ± 0.0054 and G reached 0.4547 ± 0.0151 under the default scorer.
• Answer length did not explain performance: Grouped answers averaged 33.3 words and per-question answers 51.6–55.8 words, against a 47.9-word reference average, yet F1 scores were similar.
• Raw generation overran the word limit: 23 of 76 answers exceeded 100 words before rewriting, with the longest at 153 words; after rewrite 7 still exceeded it, and the length guard brought the maximum to 99.
• Post-processing introduced some errors: All six missing evidence labels were introduced by the length guard, which touched 7 of 76 answers, and it added one of five non-verbatim quotation flags. Manual inspection also found one altered monetary scale the validator does not test.

Methodology in Plain English

Greek NER (Subtask 1). Because the two Greek datasets come with very different amounts of supervision, the team used two different mechanisms. Numeric entities (money, percentages, dates, quantities) are pulled out by ordered regular expressions applied in a fixed priority order, with a character-level claim array so a lower-priority rule cannot overlap a span already taken. Leftover bare numerals are typed by looking at nearby characters: 45 before and 40 after by default, or narrower 14/22-character windows for currency and unit tests. For textual entities (persons, organizations, locations), no training labels exist, so the team first generates a high-recall pool of candidate spans, using capitalization runs, legal forms, a gazetteer built from eight existing Greek NER datasets, person-shaped bigrams, and acronyms. Each candidate is then shown to Qwen2.5-32B together with the whole passage, and the model picks person, organization, location, or none. Only candidates with confidence of at least 0.90 are kept, and overlaps are resolved by preferring longer spans and then higher confidence. Because the model only labels spans that already exist in the text, every prediction stays anchored to the source.

Japanese implicit communication (Subtask 2). The team built a scoring function rather than training a model. A curated inventory of Japanese regular expressions counts evidence in the response for families such as DECIDED, DONE, NEG_HARD, NEG_SOFT, REOPEN, VOLIT, COND, NEUTRAL, NOPROMISE, REAFFIRM, and numbers/dates. Weighted counts accumulate into five scores, one per ordinal label. Careful linguistic handling matters: reaffirmation phrases like "there is no change" are removed before hard-negation counting so they are not mistaken for refusals, and bare ありません is deliberately excluded from the hard-negation list because it usually negates a fact rather than an action. Sinkhorn normalization under a uniform prior was applied on top but changed no predictions.

Multilingual QA (Subtask 3). Contexts are long and multi-source, so rather than trimming them the team inserts retrieved evidence into the prompt. BGE-M3 pulls the five best-matching financial-statement chunks and five best-matching news chunks per question, and these are placed before the final question while the original context stays intact. Qwen3-32B generates the answer with reasoning disabled. A rewrite pass targets 55–65 words, a length guard forces 45 words or fewer for answers still over 100 words, and a language guard re-renders answers in English when non-Latin characters are detected. Final checks flag, but do not fix, unsupported numbers and non-verbatim quotations. To separate the effects of retrieval from those of per-question generation, the team ran a four-cell grid on the development set across five seeds, holding model, decoding, context, and scorer constant.

Why This Matters

The paper is a useful counterweight to the assumption that bigger general-purpose models always win. In two of three subtasks, deliberate, interpretable, task-specific engineering matched or beat neural and LLM-centric alternatives, and the authors report that honestly rather than overclaiming.

Real-world applications:

• Financial document processing: Regex-plus-LLM extraction of monetary amounts, dates, and organization names from filings in low-resource languages, where labeled training data is scarce.
• Investor-relations analytics: Detecting whether management is committing, deferring, or refusing in earnings-call and Q&A transcripts, useful for analysts tracking changes in tone and resolve.
• Multilingual analyst assistance: Question answering over SEC statements combined with contemporaneous news in English, Chinese, Japanese, Spanish, and Greek.
• Compliance and disclosure review: Flagging answers whose numbers cannot be traced to source text or whose quotations are not verbatim, a first step toward auditable generated summaries.

Industry relevance: The work speaks directly to teams deploying LLMs under constraints, showing practical accounting against a 70B parameter cap (33.3B total used here), local 4-bit inference without API calls, and the observation that a required output element (the evidence label) needs a deterministic post-generation check, not just a validator that flags problems.

Future Directions

• Replace dataset-specific rules with transferable methods. The authors name dataset-specific rules and handcrafted cues as key limitations; a natural next step is learning or inducing these patterns rather than writing them per benchmark.
• Resolve the QA component question with more evidence. The controlled analysis used only 76 development instances and five seeds, and the paper notes the public split has just 24 instances; larger evaluations could determine whether retrieval or generation granularity helps at all, and whether the patterns for R&D and cash-flow questions (19 instances each) hold.
• Move from flagging to repairing. Validators currently detect untraceable numbers, non-verbatim quotes, and missing evidence labels without fixing them, and the length guard dropped the evidence label in six of seven answers it touched; deterministic structural enforcement is an obvious follow-up.
• Address the development-to-evaluation gap in Subtask 2. Development accuracy (0.3846 for the raw scorer) was far below test accuracy (98.0%), and the authors state they cannot conclusively determine which dataset characteristics caused the difference because evaluation labels and their distribution are unavailable.

Target Audience

Graduate students and applied researchers working on financial NLP, low-resource or multilingual information extraction, and shared-task systems will get the most from this paper. It is also valuable for practitioners building constrained LLM pipelines who want concrete evidence on where rule-based components still pay off, and for benchmark designers interested in how evaluation splits and annotation conventions shape reported results. Readers seeking novel model architectures or training methods will not find them here; the contribution is engineering insight and honest negative results.

Authors’ abstract

We present Fin_Agent, our submission to the FinEval 2026 shared task on multilingual fi nancial language understanding. Our sys tems address Greek financial named-entity recognition, Japanese implicit communication recognition, and multilingual financial ques tion answering. For Greek NER, we com bine annotation-aware numeric rules with zero shot Qwen2.5-32B classification, achieving 0.9521 public F1 and 0.8884 private F1. For Japanese implicit communication, an inter pretable cue-based ordinal scorer achieves 98.0% test accuracy. For multilingual financial QA, BGE-M3retrieval with Qwen3-32B gen eration achieves 0.3023 and 0.3419 ROUGE-1 F1 on the public and private splits. Key limita tions include dataset-specific rules, handcrafted cues, and limited supervision. Implementa tion code is available https://github.com/ Rashidnira/finagent-fineval-2026.