How Data Quality Affects Machine Learning Models for Credit Risk Assessment

AI summary

Overview

Research area: Data-centric AI, applied to machine learning for credit risk assessment (financial ML).

Technical level: Intermediate. The paper assumes familiarity with credit scoring, common classifier families, and classification metrics (accuracy, precision, recall, F1), but its central methodology and conclusions can be understood without deep statistical background.

Scope: The paper measures how five categories of deliberately injected training-data errors (wrong labels, duplicates, missing values, outliers, and noise) at two corruption levels (30% and 50%) change the F1 score of 10 standard classifiers on a public credit risk dataset.

What This Paper Is About

Machine learning models are widely used to predict whether a borrower will default, yet they are trained on real-world data that is frequently incomplete, duplicated, mislabeled, or noisy. This paper asks a simple but under-examined question: how much does each type of data defect in the training set actually change a credit risk model's predictive performance, and does the answer depend on which model you choose?

To answer this, the author takes a clean open-source credit risk dataset, deliberately corrupts it in controlled ways using the Pucktrick library, and retrains 10 classifiers on the corrupted data while always testing on clean data — isolating the effect of training-time data quality from test-time variability.

Key Contributions

1. A systematic robustness benchmark for credit risk models. The study corrupts one Kaggle credit risk dataset (32.581 rows, 12 columns) across five error categories at two severity levels, producing more than 80 corrupted datasets, and evaluates 10 classifiers on each.

2. A formal "error model" abstraction in the Pucktrick library. The paper introduces the error model as a six-element specification — 𝔼 = (𝔻, ε, σ, ρ, p, φ), covering the dataset, error type, set of features, predicates, percentage modified, and corruption mode ("new" or "extended") — making corruption modular, configurable, and reproducible.

3. Evidence that some data defects help rather than hurt. The paper documents counterintuitive gains, most notably the lda model reaching an F1 score of 0.9675 when 50% duplicates are introduced — an improvement of 17% over the same model trained on the original dataset.

4. A demonstration that error impact is model-specific, not feature-specific. The same corrupted feature can leave one model unaffected and severely damage another, which the author argues means preprocessing strategy must be tied to the chosen algorithm rather than applied generically.

Main Findings

• Wrong labels are unambiguously destructive. No model trained on mislabeled data outperformed its clean-data counterpart. At 30% label corruption, F1 scores ranged from 0.6237 (lda) down to 0.0003 (svm); at 50%, et, lr and svm collapsed to 0.000, 0.0013 and 0.00036 respectively.

• Duplicated rows generally improved performance. At 30% duplication, every model except Random Forest improved (lda 0.9626, et 0.9396, lr 0.9554, knn 0.6023, mlp 0.6774, nb 0.4437, dt 0.5514, svm 0.5000). At 50%, all models improved except Random Forest and MLP, which degraded slightly.

• Missing values were tolerated by most models but not all. On average, every model except Naive Bayes and MLP improved at both 30% and 50% corruption. MLP effectively collapsed (F1 of 0.0003 and 0.0287), and Naive Bayes declined (0.4102 and 0.3838).

• Outliers produced the most striking single gain. Almost all models improved with outlier contamination, with SVM nearly doubling its F1 score relative to its clean-data result of 0.1646 (reaching 0.3188 at 30% and 0.4026 at 50%). The author attributes this to outliers altering which points become support vectors and thus shifting the decision boundary.

• Noise effects were inconsistent. Only Logistic Regression (0.9065 at 30%, 0.8822 at 50%), SVM (0.4257, 0.4233) and Extra Trees (0.8642, 0.8254) improved at both noise levels. Random Forest, Naive Bayes and MLP declined at both levels, while KNN, LDA and Decision Tree varied by severity.

• Two models are consistently fragile. MLP and Naive Bayes are the only classifiers that showed significant decline across the error categories, which the author attributes to Naive Bayes' distributional assumptions and MLP's sensitivity to outliers through gradient-based optimization.

• Per-feature noise effects are not uniform. For the feature person_income under noise, the paper reports that for 40% of the models examined the F1 score diminished, showing that sensitivity is model-specific rather than driven by a single bad feature.

• Severity matters in a non-monotonic way. Increasing corruption from 30% to 50% sometimes made performance worse (e.g., lda under mislabeling fell from 0.6237 to 0.2360) and sometimes better (e.g., lda under duplicates rose from 0.9626 to 0.9675).

Methodology in Plain English

The workflow has five steps.

First, the author takes a public credit risk dataset with 12 columns and 32.581 rows. After removing rows with null values (under 10% of the dataset) and binning variables including person_age, income and loan_amount, the final training dataset contains 14 features, 1 target variable, and 28.637 rows. The dataset is unbalanced: 25473 rows have loan_status equal to 0 (78,2% no-default) and 7.108 rows have loan_status equal to 1 (21,8% default); the original data also contains 165 duplicated rows.

Second, the data is split 80/20 into training and test sets. The test set stays clean throughout.

Third, the training set is deliberately corrupted using Pucktrick, which supports five error categories:

• Duplicates — copies of existing rows, including modified duplicates with altered column values.
• Noise — values replaced with numbers, dates, or categories sampled uniformly within the feature's original range, or random strings for string features.
• Outliers — anomalous values injected in the range [3*|σ|, 5*|σ|] for high values or [−5*|σ|, −3*|σ|] for low values.
• Missing values — values set to NaN.
• Wrong labels — target labels swapped between classes.

Fourth, five error models are defined. 𝔼1 corrupts the target variable Loan_status across the whole dataset at 30% (mode "new") and 50% (mode "extended"). 𝔼2 introduces duplicates across all features simultaneously. 𝔼3, 𝔼4 and 𝔼5 each inject missing values, noise, or outliers into one feature at a time, producing 14 distinct datasets per model and more than 80 corrupted datasets in total.

Fifth, 10 classifiers — Linear Discriminant Analysis (lda), Logistic Regression (lr), Extra Trees (et), Random Forest (rf), K-Nearest Neighbors (knn), Multi-Layer Perceptron (mlp), Naive Bayes (nb), Decision Tree (dt), Support Vector Machine (svm), and Quadratic Discriminant Analysis (qda) — are trained on each corrupted training set via the PyCaret framework using default hyperparameters, then evaluated on the clean test set. F1 score is the primary metric because of the class imbalance; accuracy, precision and recall are also reported for the clean-data baseline. Note that results tables report nine models; qda results are not reported.

Why This Matters

The paper reframes data quality as a model-selection criterion rather than a purely upstream preprocessing problem. Its most provocative implication is that aggressive cleaning — removing duplicates, imputing values, and trimming outliers — may in some cases remove signal that certain models exploit, while label noise, which practitioners often treat as a lower-priority concern, is the one defect that reliably destroys performance.

Real-world applications:

• Credit scoring pipelines: Institutions can prioritize investment in labeling quality and label auditing over blanket deduplication, since mislabeling caused universal degradation while duplicates frequently helped.
• Model risk management and stress testing: The error models provide a repeatable way to stress-test a candidate model against realistic data degradation before deployment.
• Regulatory and Basel-aligned model validation: The framework offers a structured robustness argument for models used in capital and risk assessment under regulatory scrutiny.
• MLOps and retraining monitoring: The results motivate tracking noise and label errors as live pipeline health signals, particularly during periodic model retraining.

Industry relevance: The paper targets practitioners who need to know which algorithms survive imperfect historical data, and it positions Pucktrick as a practical tool for pipeline testing rather than a purely academic artifact.

Future Directions

• Replicate across other credit datasets. The author explicitly calls for applying the same methodology to different credit datasets to test whether the observed patterns generalize beyond this single 32.581-row source.

• Extend beyond binary classification. The paper proposes covering multiclass tasks and regression scenarios, since credit risk is currently treated only as a binary default/no-default problem.

• Investigate fairness impacts of corruption. The author identifies the effect of data corruption on fairness as an intriguing avenue, connecting to the related work showing 7–9% lower AUC for underserved groups in credit scoring.

• Exploit the error model abstraction further. Because corruption is expressed as a configurable, reproducible transformation, the framework could be reused to compare preprocessing strategies or automated data-cleaning pipelines against one another under identical degradation conditions.

Target Audience

Practitioners who build or validate credit risk and financial ML pipelines, including data scientists, ML engineers, and model risk or validation analysts, will find the practical guidance most directly useful — especially the model-by-model robustness tables and the finding that label noise deserves the most attention.

Researchers in data-centric AI and data quality will find value in the formalized error model and the Pucktrick-based methodology, which is designed to be reused as a flexible experimental framework. The paper is less suited to readers seeking a new predictive algorithm; its contribution is empirical and methodological rather than architectural.

Authors’ abstract

Machine Learning (ML) models are being increasingly employed for credit risk evaluation, with their effectiveness largely hinging on the quality of the input data. In this paper we investigate the impact of several data quality issues, including missing values, noisy attributes, outliers, and label errors, on the predictive accuracy of the machine learning model used in credit risk assessment. Utilizing an open-source dataset, we introduce controlled data corruption using the Pucktrick library to assess the robustness of 10 frequently used models like Random Forest, SVM, and Logistic Regression and so on. Our experiments show significant differences in model robustness based on the nature and severity of the data degradation. Moreover, the proposed methodology and accompanying tools offer practical support for practitioners seeking to enhance data pipeline robustness, and provide researchers with a flexible framework for further experimentation in data-centric AI contexts.