Are Foundation Models Useful for Bankruptcy Prediction?

AI summary

Overview

Research area: Machine learning for financial risk prediction — specifically corporate bankruptcy forecasting with foundation models versus classical tabular ML.

Technical level: Intermediate. The core argument is accessible, but the evaluation leans on metrics (ROC-AUC, F₁-score, threshold calibration) and model families that readers should already recognize.

Scope: The paper delivers the first systematic comparison of two foundation models (Llama-3.3-70B-Instruct for text, TabPFN for tabular data) against five classical baselines for corporate bankruptcy prediction on large, highly imbalanced Central European financial datasets across five prediction horizons.

What This Paper Is About

Foundation models are increasingly applied to financial problems, but nobody had systematically checked whether they actually beat well-established methods on structured financial prediction. The authors take corporate bankruptcy forecasting — a task where reliable probability estimates matter — and run foundation models head-to-head against gradient boosting, neural networks, and logistic regression. The goal is to determine whether general-purpose models are a viable replacement for specialized, domain-tuned approaches.

Key Contributions

1. The first systematic comparison of foundation models (an LLM and a tabular foundation model) against classical ML baselines for bankruptcy prediction on large-scale structured financial data.
2. Evidence that LLM self-reported probability estimates are poorly calibrated and discretized, clustering around a few fixed values rather than producing smooth distributions — making them unsuitable for risk assessment.
3. Demonstration that TabPFN incurs substantial computational overhead (specialized GPU hardware, far longer inference) without commensurate performance gains.
4. Release of an evaluation across five prediction horizons (0 to 4 years ahead) on over one million financial statement records from the Visegrád Group, with detailed feature and target construction documented.

Main Findings

• Classical methods win across every horizon. XGBoost, CatBoost, and LightGBM consistently outperform both foundation models. Traditional ML maintains ROC-AUC above 0.99 at h=0, declining gradually to 0.85–0.89 at h=4.
• TabPFN underperforms on ROC-AUC but is competitive on F₁ at some horizons. TabPFN reaches 0.987 ROC-AUC at h=0 (dropping to 0.771 at h=4), behind all standard ML methods across most horizons, yet it beats MLP and logistic regression on F₁ at most horizons while still trailing gradient boosting.
• LLM approaches are the weakest on ROC-AUC and F₁. At h=4, strong classical methods achieve F₁ of 0.024–0.069, while foundation models reach only 0.012–0.024.
• LLM probability estimates are degenerate. Self-reported probabilities cluster around fixed values (0.1, 0.2, 0.7, 0.9) in both zero-shot and in-context learning variants, producing discrete rather than continuous risk scores.
• In-context learning helps ROC-AUC but not F₁. Llama-3.3 (ICL) improves ROC-AUC over zero-shot at several horizons (e.g., 0.966 vs 0.945 at h=0) yet its F₁ score at h=0 (0.114) is lower than zero-shot (0.141).
• Extreme computational overhead. On the 20,000-sample test subset: XGBoost (CPU, M4 Pro 4 threads) takes 0.007 ± 0.001 s at 2,833,570.17 samples/s; TabPFN (NVIDIA A100-40GB) takes 23.782 ± 1.668 s at 844.77 samples/s; Llama via API with 8 concurrent requests takes 5357.400 ± 205.563 s at 3.739 samples/s.
• Partition-then-predict scaling for TabPFN trades off metrics. The decision-tree partitioning variant (TabPFN-DT, used in the main analysis) yields 0.771 ROC-AUC / 0.024 F₁ at h=4, while bootstrap ensembles at 8, 16, and 24 subsets reach 0.797, 0.794, and 0.814 ROC-AUC respectively, all with F₁ of 0.018, 0.013, and 0.013.
• Best classical results per horizon (ROC-AUC / F₁): h=0 — XGBoost, LightGBM, CatBoost all 0.996 ROC-AUC, with F₁ of 0.465 (XGBoost); h=1 — XGBoost 0.968/0.200; h=2 — XGBoost 0.894/0.198; h=3 — CatBoost 0.901 ROC-AUC, MLP F₁ 0.071; h=4 — XGBoost 0.891 ROC-AUC, LightGBM F₁ 0.069.

Methodology in Plain English

The authors assembled five datasets from over one million financial statement records of companies in the Czech Republic, Hungary, Poland, and Slovakia (the V4 group), spanning 2006 to 2021, sourced from the Emerging Markets Information Service (EMIS) database. This covers 203,900 unique companies and 1,106,879 company-year observations, distributed across Poland (628,499), Hungary (358,486), Slovakia (62,141), and the Czech Republic (57,753).

A company is labeled as distressed if, in its final reporting year, it simultaneously had negative equity to total assets, negative EBITDA relative to total assets, and a current ratio below 0.6. The authors then shift the data back to create five prediction tasks — predicting bankruptcy 0, 1, 2, 3, and 4 years ahead — with bankruptcy rates below 1% in every dataset (e.g., 3,587 of 1,000,087 records at h=0, falling to 1,485 of 700,041 at h=4).

All models are tested on the same stratified 20,000-sample subset per horizon. The feature set contains 131 financial indicators, covering liquidity, profitability, solvency, turnover, growth, sector-relative measures, and metadata. Missing values are imputed with training medians; numeric features are standardized; categorical columns are label-encoded. Decision thresholds are tuned on validation data to maximize F₁, and models are evaluated with ROC-AUC and F₁.

Two foundation models are tested. Llama-3.3-70B-Instruct receives company data serialized into natural-language prompts with detailed ratio explanations, a financial analyst persona, and strict output formatting; it is tested zero-shot and with in-context learning using 20 examples (10 bankrupt, 10 non-bankrupt) chosen as "hard" cases via a proxy XGBoost model. Because TabPFN is designed for datasets under 10,000 samples, the authors use a partition-then-predict scheme in which a shallow decision tree splits the feature space (minimum 10,000 samples to split an internal node) and TabPFN predicts within each leaf. Five classical baselines — logistic regression, MLP, XGBoost, LightGBM, and CatBoost — are trained with grid-search hyperparameter optimization selected by F₁ on validation data (random_state=42).

Why This Matters

The result challenges the assumption that bigger, more general models automatically translate into better financial predictions. For a risk-sensitive domain where calibrated probabilities drive credit decisions and capital allocation, the finding that LLMs produce lumpy, poorly calibrated confidence scores is a direct caution against naive adoption.

Research impact: The paper establishes a baseline that future foundation-model-for-finance work must beat, and it frames probability calibration — not just ranking metrics — as a first-class evaluation criterion for structured financial prediction.

Real-world applications:

• Credit and lending decisions: Banks and lenders evaluating whether to swap gradient boosting scorecards for LLM-based scoring can see the accuracy, calibration, and cost trade-offs laid out directly.
• Regulatory and supervisory stress testing: Supervisors needing transparent, reproducible, hardware-light models for portfolio-level default risk have evidence that classical methods remain the practical choice.
• Investor and counterparty risk screening: Portfolio managers screening large universes of small and mid-sized European firms can see how far ahead (up to 4 years) prediction quality holds, and where it degrades.
• Cost-conscious model deployment: Teams budgeting for inference infrastructure get concrete timing data showing roughly 3.7 samples per second for the LLM versus millions per second for XGBoost on CPU.

Industry relevance: The computational economics are stark. A model requiring an NVIDIA A100-40GB GPU and roughly 24 seconds per 20,000 predictions (TabPFN) or over 5,300 seconds via API (Llama) — while scoring worse than a CPU model finishing in 0.007 seconds — makes the business case for foundation models in this task difficult to argue.

Future Directions

• Stronger reasoning LLMs. The authors explicitly note that Llama-3.3-70B-Instruct is not among current top models and suggest GPT-5-thinking, Claude Opus 4.1, DeepSeek-V3.1, and Qwen3 may perform better.
• Access to weights and logits. API-level access limited the study to self-reported probabilities; the authors argue that direct access to internal representations could yield more reliable confidence estimates.
• Hybrid multimodal approaches. Combining textual disclosures with numerical financial data is suggested as a route to better performance than either modality alone.
• Better scaling for tabular foundation models. The gap between TabPFN's design target (under 10,000 samples) and the hundreds of thousands of records here remains unresolved; the partitioning and bootstrap-ensemble workarounds tested so far trade ROC-AUC against F₁ without closing the gap to gradient boosting.

Target Audience

Machine learning researchers working on financial risk and tabular prediction; quantitative analysts and data scientists at banks, insurers, and credit bureaus evaluating model choices; regulators and risk officers who need calibrated probabilities rather than rankings; and practitioners deciding whether foundation models justify their inference cost. Readers most likely to benefit are those already familiar with imbalanced classification metrics and model selection, since the paper's central conclusions depend on interpreting ROC-AUC and F₁ together under severe class imbalance.

Authors’ abstract

Foundation models have shown promise across various financial applications, yet their effectiveness for corporate bankruptcy prediction remains systematically unevaluated against established methods. We study bankruptcy forecasting using Llama-3.3-70B-Instruct and TabPFN, evaluated on large, highly imbalanced datasets of over one million company records from the Visegrád Group. We provide the first systematic comparison of foundation models against classical machine learning baselines for this task. Our results show that models such as XGBoost and CatBoost consistently outperform foundation models across all prediction horizons. LLM-based approaches suffer from unreliable probability estimates, undermining their use in risk-sensitive financial settings. TabPFN, while competitive with simpler baselines, requires substantial computational resources with costs not justified by performance gains. These findings suggest that, despite their generality, current foundation models remain less effective than specialized methods for bankruptcy forecasting.