RAISE: A Unified Framework for Responsible AI Scoring and Evaluation

AI summary

Overview

Research area: Responsible / trustworthy AI evaluation for deep learning models on structured (tabular) data, spanning explainability, fairness, robustness, and sustainability.

Technical level: Intermediate. The framework is conceptually accessible, but the paper assumes familiarity with classification metrics, SHAP-style feature attributions, group fairness measures, adversarial attacks, and carbon accounting for model training.

Scope (one sentence): The paper proposes RAISE, a unified scoring framework that normalizes 21 metrics across four responsibility dimensions into per-dimension scores plus an aggregated Responsibility Score, and uses it to compare an MLP, a Tabular ResNet, and a Feature Tokenizer Transformer across three public tabular datasets.

What This Paper Is About

Regulatory frameworks such as the EU AI Act say what responsible AI should achieve, but they do not say how to measure it, and the tools that exist for explainability, fairness, robustness, and sustainability have matured separately rather than together. The authors build RAISE (Responsible AI Scoring and Evaluation) to close that gap for tabular models — the model type used in highly regulated sectors like finance and healthcare — by producing a multi-dimensional "responsibility profile" for each model rather than a single accuracy number. The goal is to make architectural trade-offs visible so that model selection becomes an explicit, documented choice about which trade-offs fit a given high-stakes context.

Key Contributions

1. A unified, four-pillar evaluation framework. RAISE quantifies model behavior across explainability, fairness, robustness, and sustainability using 21 quantitative metrics drawn from established literature, then aggregates them into a Dimension Score (DS) per pillar and a single overall Responsibility Score (RS).
2. A performance-controlled comparison protocol. All models were trained to a comparable F1-Score threshold on each dataset, so that differences in responsibility profiles are not simply explained by differences in predictive accuracy.
3. An empirical study of architectural trade-offs. Three architectures (Multilayer Perceptron, Tabular ResNet, Feature Tokenizer Transformer) were evaluated on three high-stakes tabular datasets (German Credit, Diabetes 130-Hospitals, Census Income / ACSIncome), with each experiment run on an 80/20 split and repeated five times.
4. A reproducible, modular implementation. The framework and full hyperparameter details are released at https://github.com/raise-framework/raise.

Main Findings

• Accuracy hides responsibility differences: All models reached similar F1-Scores — for example, 0.7683 (MLP), 0.7715 (TabResNet), and 0.7708 (Transformer) on German Credit, and 0.8374, 0.8378, and 0.8379 respectively on Diabetes — yet their Responsibility Scores diverged sharply (0.8352, 0.7461, 0.6402 on German Credit; 0.8796, 0.8716, 0.6222 on Diabetes). The paper states that predictive accuracy is a weak and often misleading stand-in for a model's operational and ethical fitness.
• No architecture dominates: The paper reports that no single model is universally superior across all responsibility criteria, which it presents as the central argument for multi-dimensional evaluation.
• The Transformer's sustainability cost is extreme: Its Sustainability Score was 0.2480 on German Credit, 0.4575 on ACSIncome, and 0.0071 on Diabetes, against 0.9855 / 0.9899 / 0.9833 for the MLP and 0.9689 / 0.9766 / 0.9677 for the Tabular ResNet. The raw components reflect this: normalized kgCO2e for the Transformer was 0.0000 on Diabetes, and its FLOPs and MACs scores were 0.0000 on German Credit and Diabetes.
• The MLP is the efficiency-and-robustness profile: The MLP recorded the highest Sustainability Score on all three datasets and the highest Robustness Score on all three (0.9139 German Credit, 0.9898 ACSIncome, 0.9988 Diabetes), but the paper describes its explanations as broad and less faithful.
• The Tabular ResNet is the balanced middle ground: It delivered steady results across conditions and posted the highest Responsibility Score on ACSIncome (0.8676), while remaining close to the MLP on German Credit (0.7461) and Diabetes (0.8716).
• The Transformer's fairness advantage is context-dependent: It had the highest Fairness Score on German Credit (0.9399 versus 0.9003 for MLP and 0.8996 for TabResNet), but on Diabetes the MLP scored highest (0.9770 versus 0.9636 and 0.9231).
• Explainability differences are narrow: Explainability Scores clustered in a small band across datasets — 0.5412 / 0.5024 / 0.5562 (German Credit), 0.4620 / 0.5730 / 0.4799 (ACSIncome), and 0.5594 / 0.5589 / 0.5666 (Diabetes) for MLP / TabResNet / Transformer. The paper nonetheless describes the Transformer as strong on explainability, especially in the difficult low-data setting.
• Robustness is uneven: On German Credit the Robustness Score spread from 0.1300 (Transformer) to 0.6133 (TabResNet) and 0.9139 (MLP), whereas all three models scored above 0.98 on ACSIncome and Diabetes.
• Fairness metrics are reported per-group-difference: Using gender as the sensitive attribute, the framework reports normalized differences in Accuracy, Precision, TPR, FPR, Demographic Parity, and Equalized Odds. For instance, the Accuracy Difference score for the Transformer on Diabetes was 0.9609 versus 0.9541 for the MLP.

Methodology in Plain English

The authors take four things people care about when deploying a model — can we understand it, is it fair across groups, does it hold up under adversarially perturbed inputs, and what does it cost environmentally — and turn each into a numeric score.

For explainability, they generate model-agnostic feature attributions with SHAP, then grade those attributions with eight metrics from the Quantus framework grouped into four categories: explanation robustness (Local Lipschitz Estimate, Consistency), faithfulness (Faithfulness Correlation, Faithfulness Estimate), randomization sanity checks (Model Parameter Randomization Test, Random Logit Test), and complexity (Sparseness, Complexity).

For fairness, they measure the absolute difference in standard classification metrics (Accuracy, Precision, Recall, False Positive Rate) between gender subgroups, and add two formal group fairness measures from AIF360 and Fairlearn: Demographic Parity and Equalized Odds.

For sustainability, they estimate carbon emissions (CO2e) with the Lacoste Score, which accounts for hardware power consumption and regional emission factors, and add three computational-efficiency metrics: parameter count, FLOPs, and MACs. All sustainability metrics are max-norm scaled across models and datasets to make the comparison fair.

For robustness, they use the Adversarial Robustness Toolbox (ART) to measure the drop in test accuracy under Fast Gradient Sign Method attacks (FGSM Accuracy Gap), alongside two attack-independent metrics: the CLEVER-u Score, which estimates the minimum perturbation needed to cause misclassification, and Loss Sensitivity, which measures local change in the model's loss under input variation.

Each raw metric is then normalized to a common scale, with lower-is-better metrics (marked with an asterisk in the paper's table) inverted via 1 − raw so that a score of 1 always represents ideal behavior. Metrics are averaged within each pillar to produce four Dimension Scores, which together form the responsibility profile — the paper's primary output — with an aggregated Responsibility Score provided only as a high-level summary. Predictive accuracy is kept separate deliberately.

The experimental setup uses three public tabular datasets — German Credit (finance), Diabetes 130-Hospitals (healthcare), and Census Income / ACSIncome (socioeconomics) — with an 80/20 train/test split, five repeated runs per experiment, and all models implemented in PyTorch. Dataset sizes are not reported in the paper. Full hyperparameter details are provided through the GitHub repository rather than in the text.

Why This Matters

Impact on research. The paper argues that isolated tooling for individual responsibility dimensions is insufficient: fairness toolkits like AIF360 can mitigate bias but may impose unsustainable computational costs, and SHAP-style explanations can faithfully describe a model that is nonetheless biased. RAISE proposes that these dimensions be measured jointly and reported as a profile, and its performance-controlled design offers a template for comparing architectures without letting accuracy differences confound the comparison. It also highlights that robustness benchmarks such as WILDS and RobustBench have focused largely on computer vision, leaving tabular robustness comparatively underexplored.

Real-world applications:

• Consumer credit scoring: Choosing among candidate models for credit risk prediction, where transparency and non-discrimination obligations are legally enforced and where the model's carbon footprint of retraining is a real operational cost.
• Clinical risk and readmission models: Diabetes readmission forecasting is a setting where subgroup performance disparities can translate directly into unequal care, making the fairness profile a first-order selection criterion.
• Socioeconomic and public-policy modeling: Income classification illustrates the civil-rights stakes of automated decisions, where documentation of why a model was chosen supports public accountability.
• Regulatory and internal audit documentation: The responsibility profile acts as a defensible record explaining why a particular architecture was selected, aligning technical evidence with the kind of justification regulators request.

Industry relevance. The paper targets tabular deep learning specifically because it underpins automated decision-making in the most heavily regulated sectors. For organizations already using accuracy leaderboards for model selection, RAISE offers a concrete, reproducible scoring scheme and an open implementation, and its central practical message is that selection should be reframed as choosing the trade-off profile that matches an application's ethical and operational requirements.

Future Directions

1. Extend coverage to classical models. The authors plan to add non-neural models such as boosted trees, which remain workhorses in many industries, so the framework reflects the models practitioners actually deploy.
2. Add privacy as a core dimension. Privacy would measure how well a model protects sensitive data and would become a fifth pillar alongside explainability, fairness, robustness, and sustainability.
3. Refine normalization for cross-dataset comparability. The authors flag normalization refinement as a goal for making scores comparable across datasets and settings.
4. Move from lab to practice. They plan usability studies and direct work with stakeholders to test whether the framework genuinely helps people make better, safer decisions in daily work, rather than remaining a technical artifact.

Target Audience

ML practitioners and applied researchers selecting or auditing models for regulated tabular domains such as finance and healthcare; governance, risk, and compliance teams who need quantitative evidence for model-selection decisions; and researchers working on responsible-AI evaluation frameworks who want a worked example of jointly measuring explainability, fairness, robustness, and sustainability — along with the architectural trade-offs those joint measurements expose.

Authors’ abstract

As AI systems enter high-stakes domains, evaluation must extend beyond predictive accuracy to include explainability, fairness, robustness, and sustainability. We introduce RAISE (Responsible AI Scoring and Evaluation), a unified framework that quantifies model performance across these four dimensions and aggregates them into a single, holistic Responsibility Score. We evaluated three deep learning models: a Multilayer Perceptron (MLP), a Tabular ResNet, and a Feature Tokenizer Transformer, on structured datasets from finance, healthcare, and socioeconomics. Our findings reveal critical trade-offs: the MLP demonstrated strong sustainability and robustness, the Transformer excelled in explainability and fairness at a very high environmental cost, and the Tabular ResNet offered a balanced profile. These results underscore that no single model dominates across all responsibility criteria, highlighting the necessity of multi-dimensional evaluation for responsible model selection. Our implementation is available at: https://github.com/raise-framework/raise.