Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial Documents

AI summary

Overview

Research area: Multimodal foundation models, continual/lifelong adaptation, and domain-specialised language models for financial document understanding.

Technical level: Advanced. The paper assumes familiarity with vision–language architectures (SigLIP, LLaVA-style projectors), parameter-efficient fine-tuning (LoRA, QLoRA), knowledge distillation, and catastrophic-forgetting literature.

Scope (one sentence): The paper tracks a single 8.4B-parameter Japanese financial vision–language model through a three-phase staged training pipeline and measures, on five text-input benchmarks, how each phase helps, harms, or enables the others.

What This Paper Is About

Practitioners rarely adapt a multimodal foundation model to a specialised domain in one shot; they chain stages such as vision–language alignment, reasoning distillation, and domain tuning. The paper asks whether those stages simply add up, or whether later stages overwrite or depend on what earlier stages installed. Using Japanese financial disclosures — a domain that demands regulatory knowledge, multi-step numerical reasoning, and Japanese-language competence, and whose layout-heavy tables must be read as images — the authors run an ablation-style study across phases and a skip-phase variant, and find that the best checkpoint differs by task rather than always being the last one.

Key Contributions

1. Evidence of non-additive phase interaction on EDINET and JP Fin Harness, anchored to an un-fine-tuned baseline: asymmetric forgetting (procedural math skill survives later tuning, declarative financial knowledge does not), a domain phase that repairs rather than installs capability, and a suggestive cross-phase enabling effect.
2. A three-phase staged continual adaptation pipeline (vision–language alignment, mathematical reasoning distillation, financial-domain adaptation) that trains end-to-end on page images, with the visual stages framed as an adaptation vehicle whose in-modality skill the text-input benchmarks do not yet probe.
3. A reproducible synthetic-data pipeline generating OCR, extractive/reasoning visual QA, and finance-specific QA from government PDFs (Cabinet Office, FSA, MOF), with code released at github.com/AtsushiYanaigsawa768/Compass and the synthesised datasets released on the Hugging Face Hub.
4. An XML-structured chain-of-thought distillation recipe from Qwen3-30B-A3B-Thinking-2507, tagging responses as <Problem>, <Thinking>, and <Answer> to localise intermediate-reasoning errors.

Main Findings

• Baseline to Phase 1 improves every headline metric. Moving from the un-fine-tuned baseline (P0) to the alignment phase (P1) raises JP Fin Harness from 45.0 to 53.5, GSM8K from 58.5 to 64.3, EDINET fraud ROC-AUC from 0.516 to 0.554, and EDINET industry from 4.7 to 8.7.
• The math phase (P2) installs reasoning but destroys domain knowledge. GSM8K rises to 73.8 at P2, while JP Fin Harness collapses from 53.5 to 31.6 and EDINET industry slips from 8.7 to 7.9.
• The domain phase (P3) repairs more than it installs. GSM8K settles at 71.6 and EDINET fraud peaks at 0.614 in the single run, but JP Fin Harness only recovers to 49.6 — below its P1 value of 53.5 — and EDINET industry reaches 8.2, still below 8.7.
• Forgetting is asymmetric. Procedural math skill largely survives later financial tuning (73.8 to 71.6 from P2 to P3), whereas declarative financial knowledge is overwritten by the math phase. The authors flag this as entangled with phase ordering and corpus size (the math corpus is far larger), and describe it as a hypothesis about kinds of capability rather than an established law.
• Skipping Phase 2 gives the best domain results but forfeits reasoning. The P1→P3 ablation reaches the overall peaks on JP Fin Harness (56.2) and EDINET industry (10.4), yet GSM8K drops to 59.6 versus 71.6 and EDINET fraud falls to 0.499.
• Best checkpoint is task-dependent, not the most recent. Only GSM8K (P2) and single-run EDINET fraud (P3) peak right after the phase that supervises them; JP Fin Harness and EDINET industry peak in the skip-phase ablation instead.
• Subtask behaviour is heterogeneous. The P1 gain concentrates on chabsa (0.859 to 0.943), cma_basics, and security_sales_1; every subtask collapses at P2 and only partly recovers at P3; the skip-P2 variant is best on four of five subtasks, while cpa_audit alone peaks at P0 (0.276). The chabsa trajectory (0.943 to 0.581 to 0.935) mirrors the aggregate declarative-knowledge pattern.
• EDINET earnings is not interpreted. It stays near chance for all checkpoints (ROC-AUC approximately 0.46–0.52), so the authors explicitly decline to read its maximum as a capability difference.
• Single greedy runs are optimistic; multi-seed means are lower and re-order rows. Averaged over 3 seeds (temperature 0.7, top_p 0.95), GSM8K is 62.6 ± 0.5 at P1, 69.4 ± 1.4 at P2, and 68.8 ± 1.3 at P3, and the EDINET fraud peak moves from P3 to P1 at 0.562 ± 0.008 (P3 mean is 0.547 ± 0.005).
• A small model can rival larger ones on one narrow task. In its best single run, P3 reaches 0.614 ROC-AUC on EDINET fraud, above Llama-3.3-70B (0.59) and DeepSeek-R1 (0.56) as reported for the same text-input configuration in EDINET-Bench, and approaching GPT-5 (0.62). The paper stresses this parity rests on one run.
• EDINET industry remains a clear weakness. The best checkpoint (10.4) trails every external baseline (14.0–31.0). On JP Fin Harness, 56.2 exceeds Qwen-14B-Chat (49.1) and Meta-Llama-3-8B-Instruct (44.7) but trails Qwen2-72B-Instruct (67.7) and Claude 3.5 Sonnet (77.0). On GSM8K, P2's 73.8 exceeds Qwen-14B Chat (59.3) but stays below Llama 3 8B (84.5) and Qwen3 8B-base (89.84).
• A suggestive cross-phase enabling effect, reported as a hypothesis only. Removing the math phase drops EDINET fraud below the un-fine-tuned baseline (skip-P2 0.499 vs. P0 0.516), as if reasoning distillation were a prerequisite for the later phase to express fraud detection — but the gap is 0.017 ROC-AUC astride 0.5 on a roughly 200-example test set, and the ordering does not hold under multiple seeds.

Methodology in Plain English

The team built one multimodal model rather than a family of models. A vision encoder (SigLIP-v2 SO400M-patch14-384, roughly 0.4B parameters) reads document page images; a two-layer MLP projector (Linear → GELU → Linear) translates those visual features into the language model's embedding space; and the language model backbone is llm-jp-4-8B-Instruct (8B parameters), bringing the total to approximately 8.4B.

Training is staged, freezing different components at each step. Phase 1 aligns vision and language: first only the projector is trained on STAIR Captions and OCR data derived from government PDFs, then the projector plus the vision encoder are updated and the LLM is adapted with LoRA on extractive and reasoning QA from government PDFs, JA-VG-VQA conversational data, and JA-VG-VQA-500. Phase 2 distils chain-of-thought math reasoning via QLoRA on approximately 150K problems sampled across six public math corpora, using Qwen3-30B-A3B-Thinking-2507 as the teacher and re-annotating each problem into the XML tag structure. Phase 3 returns to finance: text-only adaptation on FinQA, ConvFinQA, and TAT-QA, then projector plus LoRA adapters on visual financial QA from government PDFs, with the vision encoder frozen throughout.

The synthetic supervision itself comes from Qwen3-VL-32B-Instruct as a teacher over single-page government PDFs: an OCR dataset, an extractive/reasoning visual-QA dataset where a second turn asks for evidence grounding, and a finance-specific QA dataset restricted to finance-relevant sub-sites. Evaluation uses five text-input benchmarks — GSM8K, two EDINET classification tasks scored by ROC-AUC, EDINET industry, and JP Fin Harness — plus a skip-Phase-2 ablation to isolate what the reasoning stage contributes. The authors state plainly that these are all text-input evaluations and do not probe the page-reading skill the visual stages target.

Why This Matters

Impact on research. The paper argues against treating multi-stage adaptation as a monotone improvement stack. Its practical conclusion is that the leverage lies in scheduling and parameter isolation — composed per-phase adapters, replay or interleaving, phase reordering — rather than in adding more stages or more data. It also supplies an openly released synthetic-data recipe for a low-resource, non-English, layout-heavy financial domain.

Real-world applications:

• Screening and analysis of Japanese regulatory filings (EDINET disclosures, FSA and MOF documents) where dense tables and multi-column statements must be read as images.
• Fraud or anomaly flagging in statutory financial reporting, the task on which the model's best single run matched or beat much larger systems.
• Financial-domain question answering over government economic reports, budgets, and statistical releases.
• Automated extraction and grounding of figures from scanned or PDF-native disclosures for audit and compliance workflows.

Industry relevance. Financial institutions and regulators dealing with non-English disclosure regimes face scarce domain data and expensive large-model deployment. The result that an 8.4B model can approach GPT-5 (0.62) on EDINET fraud (0.614, best single run) while trailing badly on EDINET industry (8.2 vs. 31.0) is a concrete argument for task-specific checkpoint selection and retention policies rather than "keep the last checkpoint."

Future Directions

• Multi-seed skip-P2 ablation. The authors name this the priority follow-up, since the skip-phase variant could not be included in the multi-seed evaluation for resource reasons, and the fraud single-run variance undercuts the enabling-effect hypothesis.
• Separating the causes of P3's shortfall. Whether P3 fails to regain P1's level because overwritten knowledge is intrinsically hard to recover, or simply because the P3 corpus is far smaller, would be separated by a volume-matched or interleaved P3.
• Benchmarks that actually measure in-modality page reading. The current text-input evaluations leave the visual capability the pipeline targets unmeasured.
• Multi-page training. All training is restricted to single-page inputs; multi-page documents are a natural extension, alongside testing whether the methodology transfers to other disclosure regimes (e.g. SEC 10-K filings).

Target Audience

Researchers and engineers working on multimodal or domain-specialised foundation models, particularly those designing multi-stage post-training pipelines and deciding which checkpoints to keep. It is also relevant to practitioners building financial-document systems for non-English or low-resource regulatory domains, and to anyone interested in continual-learning trade-offs, synthetic data generation from public documents, or distillation recipes for numerical reasoning.

Authors’ abstract

Staged post-training of multimodal foundation models chains vision-language alignment, reasoning distillation, and domain tuning, yet how each phase trades capabilities against the others remains under-characterised. Tracking an 8.4B-parameter model from an un-fine-tuned baseline through three training phases on Japanese financial documents (evaluated on five text-input benchmarks), we find the phases interact non-additively: math-only distillation overwrites the financial knowledge acquired earlier even as procedural math skill survives, and the domain phase repairs more than it installs. A skip-phase ablation is best for domain knowledge but forfeits numerical-reasoning gains, so per-phase contributions are not decomposable and the best checkpoint is task-dependent, with implications for checkpoint retention and scheduling rather than added stages.