NSR-Boost: A Neuro-Symbolic Residual Boosting Framework for Industrial Legacy Models

AI summary

Overview

• Research area: Neuro-symbolic AI for tabular data, applied to industrial machine learning systems; combines LLM-based code generation, symbolic regression, Bayesian optimization, and gradient-boosted decision trees.
• Technical level: Advanced. The paper assumes familiarity with GBDTs, symbolic regression, bi-level optimization, Bayesian optimization (TPE), and industrial model deployment constraints.
• Scope: The paper proposes a non-intrusive framework that repairs frozen legacy GBDT models by generating interpretable symbolic "expert" functions with an LLM, validates it on six public datasets and one private financial dataset, and reports a production deployment at Qfin Holdings.

What This Paper Is About

Industrial tabular systems rely on Gradient Boosted Decision Trees (GBDTs) such as XGBoost and LightGBM, but these models eventually hit a ceiling in representation capability. The usual fixes — swapping in a new architecture (TabNet, FT-Transformer, CatBoost) or adding generated features (AutoFeat, OpenFE, CAAFE, FeatLLM) — all require changing the input schema or the model itself, which means retraining and refactoring live pipelines. NSR-Boost instead keeps the deployed model completely frozen and learns an additive correction term only in the regions where that model fails.

Key Contributions

1. A non-intrusive neuro-symbolic framework. NSR-Boost is presented as the first framework that uses LLM-generated code to improve a legacy model while keeping that model frozen, preserving low-latency inference and full logical interpretability because the final artifact is executable Python code with no LLM in the inference path.
2. A bi-level optimization mechanism that explicitly decouples symbolic structure search (outer loop, LLM-driven with negative/positive constraints) from parameter tuning (inner loop, gradient-free Bayesian optimization via Tree-structured Parzen Estimator). This is designed to overcome the non-differentiability bottleneck of symbolic regression and to refine coarse-grained CART-derived decision boundaries into precise numerical thresholds.
3. Empirical validation at two scales. The framework is reported to outperform SOTA baselines on six public benchmarks and one private dataset, and to improve a production financial risk-control model at Qfin Holdings on real online traffic.
4. An interpretability and safety design in which generated experts use explicit if-else logic and output the legacy model's original prediction outside their target region, ensuring zero interference for non-target groups and allowing business personnel to review decision logic directly.

Main Findings

• Public benchmark performance. On six public datasets spanning 699 to 48,000 samples, NSR-Boost achieves a mean rank of 1.0. On credit-g, XGBoost accuracy rises from 0.751 to 0.815 (+6.4%), and on blood-transfusion from 0.742 to 0.820 (+7.8%). The paper attributes these gains on data-scarce datasets to the LLM's semantic priors acting as regularization against noise.
• Private financial dataset. On the private Qfin Holdings risk-control dataset, evaluated with AUC, NSR-Boost raises the base XGBoost score from 0.688 to 0.694. The paper reports that a simple residual boosting baseline overfit high-frequency industrial noise and regressed from 0.688 to 0.659, while symbolic experts filtered the noise.
• Backbone generalizability. Figure 2 reports that NSR-Boost consistently improves accuracy regardless of the underlying gradient boosting backbone; on credit-g, LightGBM improves by +4.3%. The paper argues this shows the mined symbolic residuals capture intrinsic data properties rather than a specific algorithm's biases.
• Ablation results on credit-g. Removing Bayesian optimization causes the largest drop, from 0.815 to 0.766. Removing either context-aware aggregation or region identification drops accuracy to 0.773. Removing raw feature inputs at the aggregation stage drops performance to 0.783. Removing statistical prompting costs 0.035 accuracy. Boundary refinement contributes a smaller but real 0.5% gain.
• Interpretability demonstration. Figure 3 shows generated symbolic expert code for credit-g using explicit if-else structures that activate correction only within defined regions and otherwise return the legacy model's original output.
• Online deployment scale. The "NSR-Platform" infrastructure comprises 2 groups totaling 16 H20 accelerator cards and processes 8 core business models daily, equivalent to 96 parallel chains. Each chain is allocated a 12-core Intel Xeon Silver 4214 @ 2.20GHz CPU with 60GB memory. Generating a full suite of 12 chains for one business model consumes approximately 48 GPU-hours. The platform has cumulatively optimized models approximately 130 times across Risk Control, Fund Drawdown, and User Login, with average improvements exceeding 0.3% in both AUC and KS.
• Online result: Emerging Customer Segment (B-Card). Over approximately 2.4 million real user samples, against a legacy XGBoost model tuned on 300–400 expert features, NSR-Boost achieved an average AUC improvement of 1.17% and an average KS improvement of 1.81%, and reduced the bad rate from 3.04% to 2.86%.
• Online result: Portfolio Management Segment (B-Card). Over approximately 1.1 million real user samples, during the out-of-time deployment phase, NSR-Boost achieved an AUC improvement of 0.37% against a baseline that degraded due to concept drift. (The corresponding KS figure is not reported in the available content.)

Methodology in Plain English

The approach proceeds in three stages, all operating on a legacy model that is never modified.

1. Find the weak spots. The framework computes residuals — the difference between the true label and the legacy model's prediction — on the training set. It then trains a shallow CART tree to fit the absolute residuals and scores each leaf by cumulative error and sample coverage. Leaves above a threshold form the "hard region." Because CART leaves are mutually exclusive, these regions are mathematically disjoint, which means parallel optimization chains do not interfere and only one expert activates per inference query.

2. Grow symbolic experts with a bi-level loop. For each hard region, an LLM (locally deployed Seed-OSS-36B-Instruct in the experiments) iteratively modifies a "seed function" that starts as a constant bias constrained by the region rule. The prompt includes the task description, feature statistics such as IV and PSI, a few sampled real instances annotated with error types like False Negatives or High Uncertainty, and summaries of prior successful and failed modifications as positive/negative constraints. An annealing threshold on AUC allows tolerated early fluctuations to avoid local optima. Once a candidate structure is chosen, the inner loop handles numbers: the LLM acts only as a "Configurator" generating parameter and search-space snippets, which are injected into a sandbox, with runtime errors fed back for correction until the script runs. A TPE optimizer then performs sampling and evaluation. An incremental freezing strategy keeps inherited parameters fixed and searches only newly added coefficients, and the first successful run also folds the CART tree's hard thresholds into the search space so boundaries can be numerically refined (the paper's example is adjusting a split point from 50 to 48.5).

3. Aggregate with context. Each expert produces a Symbolic Interaction Vector containing its raw score, its residual relative to the legacy output, and their ratio. A lightweight XGBoost aggregator takes the original business features concatenated with the legacy output and the active experts' interaction vectors, allowing it to arbitrate corrections based on raw feature context rather than expert scores alone.

Why This Matters

• Research impact. The paper reframes residual correction as a code-generation problem and argues that decoupling discrete structure search from continuous parameter search sidesteps the combinatorial explosion and differentiability constraints that limit prior symbolic regression and automated feature engineering work. It also positions LLM-generated code as a deployment-time artifact rather than an inference-time component.
• Real-world applications:
  • Financial risk control and credit scoring, where the reported online deployment lives and where bad-rate reduction translates directly into avoided losses.
  • Fund drawdown and portfolio management, listed among the core scenarios the NSR-Platform has optimized.
  • User login and other high-concurrency online services where constant inference latency is mandatory.
  • Any regulated tabular pipeline where data cannot leave on-premise infrastructure and where API-based LLM methods cannot be used — the paper notes that privacy compliance prevented evaluating CAAFE, FeatLLM, OCTree, and LLM-FE on its private dataset.
• Industry relevance. The core selling point is organizational rather than purely statistical: the framework requires no retraining of the existing model, no changes to the input feature schema, and no refactoring of online feature extraction pipelines. Because the deployed output is Python code, the LLM inference cost disappears after generation, and the paper reports that the compute cost is negligible relative to business gains on assets "of a scale of hundreds of billions."

Future Directions

• Scaling beyond the reported configuration. The experiments fix K = 5 experts, a temperature of 0.1, 20 sampled instances per prompt, and termination after 5 successful iterations. How these choices trade off against performance and the reported 48 GPU-hours per 12-chain suite is not explored.
• Generalization across model families and domains. Backbone variation is tested on gradient boosting models; whether the same symbolic residual approach transfers to other legacy model types remains an open question.
• Broader aggregation designs. The current aggregator is a lightweight XGBoost over interaction vectors; alternative gating mechanisms and their latency-accuracy tradeoffs are not examined.
• Longitudinal robustness. The Portfolio Management Segment result shows concept drift hurting the baseline during the out-of-time phase; how often experts must be regenerated as data distributions shift, and at what cost, is not addressed.

Target Audience

Readers who will benefit most are applied machine learning engineers and researchers working on industrial tabular systems, particularly those in risk control, fintech, and other regulated domains where retraining a deployed model is expensive or politically infeasible. It is also relevant to researchers in neuro-symbolic AI, symbolic regression, and LLM-driven program synthesis who are interested in using LLMs as offline artifact generators rather than as predictors. The paper is dense with optimization machinery and assumes a working knowledge of GBDTs and Bayesian optimization, so it is not suited to beginners.

Authors’ abstract

Although the Gradient Boosted Decision Trees (GBDTs) dominate industrial tabular applications, upgrading legacy models in high-concurrency production environments still faces prohibitive retraining costs and systemic risks. To address this problem, we present NSR-Boost, a neuro-symbolic residual boosting framework designed specifically for industrial scenarios. Its core advantage lies in being ``non-intrusive''. It treats the legacy model as a frozen model and performs targeted repairs on "hard regions" where predictions fail. The framework comprises three key stages: First, finding hard regions through residuals, then generating interpretable experts by generating symbolic code structures using Large Language Model (LLM) and fine-tuning parameters using Bayesian optimization, and finally dynamically integrating experts with legacy model output through a lightweight aggregator. Experimental results demonstrate that the framework significantly outperforms state-of-the-art (SOTA) baselines across six public datasets and one private dataset. More importantly, we report the successful deployment of NSR-Boost within the core financial risk control system of Qfin Holdings, where empirical results on real-world online traffic exhibit superior performance improvements and a significant reduction in the bad rate. In conclusion, it effectively captures long-tail risks missed by traditional models and offers a safe, low-cost evolutionary paradigm for industry.