Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoring

AI summary

Overview

Research area: Explainable AI (XAI) for online time series models, with clinical and other safety-critical monitoring as the motivating setting.

Technical level: Intermediate. The paper is readable without deep XAI background, but its problem formulation, integration-path construction, and evaluation metrics involve moderate mathematical notation.

Scope: A framework (Delta-XAI) plus a new attribution method (SWING) for explaining changes in a model's predictions between two time steps, together with an evaluation suite for that setting.

What This Paper Is About

Existing time series XAI methods mostly explain one prediction at one time step, treating time steps independently and ignoring how a prediction evolved. In online monitoring, however, practitioners usually care about the difference between adjacent predictions — the same predicted probability can mean improvement or deterioration depending on where it came from, and subtracting single-step attributions across steps produces implausible explanations. Delta-XAI formalizes the problem of attributing prediction changes to input features, adapts 14 existing XAI methods to that problem via a wrapper, adds an evaluation suite for the online setting, and introduces SWING (Shifted Window Integrated Gradients), which builds temporal history into the integration path.

Key Contributions

1. Problem formulation and unified framework. The authors define the task of explaining prediction changes in online time series monitoring — attributing feature contributions to the difference between predictions at two time steps (T_1 < T_2) — and adapt 14 existing XAI methods to it through a prediction-difference wrapper, alongside an evaluation suite tailored to the online setting.

2. SWING, a new attribution method. They propose Shifted Window Integrated Gradients, which extends Integrated Gradients by using the retrospective prediction window as a baseline (retrospective baseline selection), integrating over all baseline–input pairs (dual-path integration), and following a piecewise-linear sliding path through intermediate historical windows (piecewise-linear historical integration).

3. Theoretical guarantees. They prove the Attribution Decomposition Theorem for Online Completeness, plus three properties of SWING: Online Completeness, Implementation Invariance, and Skew-Symmetry.

4. Systematic empirical analysis. Across diverse benchmarks and backbone architectures, they show SWING surpasses state-of-the-art alternatives under diverse evaluation metrics, and they report that classical gradient-based methods (notably IG) can outperform recent time series-specific alternatives once adapted for temporal analysis.

Main Findings

• Gradient methods hold up better than expected. When adapted for prediction-change explanation, conventional gradient-based methods such as Integrated Gradients typically outperform recent alternatives — a pattern the authors note is consistent with recent findings in single-time settings (TIMING).

• SWING leads on clinical attribution benchmarks. On the MIMIC-III decompensation benchmark with an LSTM backbone, SWING scored Corr. 23.87 ± 0.16, CPD 16.23 ± 0.10, AUPD 22.27 ± 0.19, MPD 15.52 ± 0.12, AUMPD 17.76 ± 0.04, CPP 5.85 ± 0.04, AUPP 18.20 ± 0.06, MPP 6.06 ± 0.05, and AUMPP 0.40 ± 0.00, the best result on most metrics. The strongest competing numbers included WinIT (Corr. 19.64 ± 0.07, CPD 12.25 ± 0.04) and TimeX (Corr. 16.99 ± 0.09, CPD 11.45 ± 0.06).

• Surrogate-based explainers lag on preservation metrics. LIME, TimeX, and TimeX++ show lower performance on preservation metrics, which the authors attribute to the data- and hyperparameter-sensitivity of their surrogate models.

• Robustness across datasets and backbones. On the Activity human activity recognition dataset, SWING achieves the highest scores across all metrics; on the synthetic Delayed Spike and Switch-Feature benchmarks it performs best or second-best on most metrics. On MIMIC-III with CNN and Transformer backbones, SWING outperforms competing methods across most metrics.

• Longer time gaps narrow the advantage. At (T_2 - T_1 = 6) and (24), SWING achieves best or near-best scores on most metrics with particularly strong preservation-metric gains, though gaps narrow as CPD and AUPD converge across methods. WinIT is reported only for 6 due to generator limits.

• Substitution baselines distort evaluation. Zero and average imputation for removed features produce out-of-distribution samples. On MIMIC-III with an LSTM backbone and SWING attributions, zero substitution gave CPD 28.12 and an OOD score of 0.840, average substitution gave CPD 14.85 and OOD 0.222, while forward-fill gave CPD 12.98 and OOD 0.093 — the rationale for adopting forward-filling.

• All three SWING components matter. Ablating retrospective baseline selection and piecewise-linear historical integration together causes substantial degradation, and removing either module individually also reduces performance, indicating complementary roles. The provided text is truncated mid-sentence at the finding that, without dual-path integration, the model attains the best faithfulness scores (CPD, AUPD, ...); the remainder of that result and the full ablation numbers are not available in the excerpt.

• Adapted single-step attributions are not enough. The authors state that neither computing attributions on differenced inputs (as (f) is generally nonlinear) nor subtracting attributions across outputs (as attribution algorithms are also nonlinear) yields valid explanations, and they report that Dynamask in particular produces misleading explanations when its attributions are subtracted across two time steps.

Methodology in Plain English

The paper first sets up the task: a model reads a fixed lookback window of (W) time steps and (D) features and outputs class probabilities. Instead of explaining one window, the goal is to explain the change in probability between two windows at times (T_1) and (T_2), where the target class is the one with the largest probability increase. The authors restrict to (T_2 - T_1 < W) so windows overlap and share temporal context.

To reuse existing tools, they wrap the model so that its output becomes the difference between the two predictions: (g(X) = f(X_{T_2-W+1:T_2}) - f(X_{T_1-W+1:T_1})). Any single-prediction explainer can then be pointed at this wrapped function. For explainers that are linear and complete — SHAP variants, IG, DeepLIFT — the wrapper reduces to simply subtracting attributions from the two time steps, which yields an exact decomposition of the prediction change into newly added features, delayed effects of intermediate features, and removed oldest features.

SWING starts from Integrated Gradients, which walks a straight line from a zero baseline to the input. That path ignores temporal structure and can leave the data manifold. SWING instead uses the immediately preceding window as the baseline, averages integrations over both combinations of baselines and targets, and, when the temporal gap is large, walks through intermediate windows step by step so the path stays near realistic data. In practice the path integral is approximated by uniformly sampling 50 points.

For evaluation, the authors replace removed features with forward-fill rather than zeros or averages, and they build on TIMING's sequential removal idea (CPD, CPP) with area-based metrics (AUPD, AUPP), macro-level aggregation over a centered sliding window (MPD, MPP, AUMPD, AUMPP), and a correlation metric. Experiments use LSTM backbones mainly, plus CNN and Transformer; datasets are MIMIC-III, PhysioNet 2019, Activity, Delayed Spike, and Switch-Feature; 50 feature points are removed per time step and results are reported as mean ± standard error over five runs.

Why This Matters

Impact on research. The paper reframes time series XAI around prediction changes rather than static predictions, supplies a wrapper that makes 14 existing methods comparable on that task, and argues that current faithfulness/sufficiency protocols are unreliable for evolving predictions. It also reports that a well-tuned classical gradient method can beat newer, more elaborate alternatives, which is a useful corrective for the subfield.

Real-world applications (as discussed in the paper):

• Healthcare monitoring: a sepsis risk drop from 90% to 50% signals improvement while a rise from 10% to 50% signals deterioration; explanations must identify which signals drove the recovery or the deterioration.
• Finance: analysts wanting the factors behind a credit score drop.
• Transportation and climate monitoring: the paper lists these as safety- and mission-critical domains where online time series are gathered.
• Handling irregular sampling and imputation: in a scenario where all values at (T_3) are forward-filled, standard attribution wrongly assigns high importance to the imputed readings, whereas the paper's time-wise attribution from (T_1 \to T_2) and (T_2 \to T_3) de-emphasizes imputed values.

Industry relevance. Because SWING satisfies online completeness, implementation invariance, and skew-symmetry, its outputs are internally consistent and auditable — a practical requirement for deploying explanations in regulated or high-stakes monitoring. The paper also frames computational efficiency as part of the evaluation suite for real-time feasibility.

Future Directions

• Extending beyond adjacent steps at scale. The headline setting is (T_2 - T_1 = 1); performance gaps narrow at gaps of 6 and 24, raising the question of how to preserve attribution quality over long horizons, and WinIT could only be evaluated at a gap of 6 because of generator limits.
• Replacing or auditing evaluation substitutions. The authors show zero/average substitution creates out-of-distribution samples and adopt forward-fill; whether other substitution strategies or generative counterfactuals would change the ranking of methods is left open.
• Broader evaluation coverage. The suite adds coherence via a case study and time/memory complexity empirically, but the paper's own framing leaves room for a more principled coherence metric beyond case studies.
• Generalizing the wrapper's guarantees. The exact decomposition result requires a linear and complete explainer with a fixed baseline; clarifying how far the framework extends to non-linear or non-complete explainers remains an open question.

Target Audience

Researchers and practitioners in explainable AI for time series, especially those working on clinical or other online monitoring systems; machine learning engineers evaluating attribution methods for deployment in regulated domains; and readers familiar with Integrated Gradients, SHAP-style attribution, or time series XAI who want a framework for attributing prediction changes rather than single predictions.

Authors’ abstract

Explaining online time series monitoring models is crucial across sensitive domains such as healthcare and finance, where temporal and contextual prediction dynamics underpin critical decisions. While recent XAI methods have improved the explainability of time series models, they mostly analyze each time step independently, overlooking temporal dependencies. This results in further challenges: explaining prediction changes is non-trivial, methods fail to leverage online dynamics, and evaluation remains difficult. To address these challenges, we propose Delta-XAI, which adapts 14 existing XAI methods through a wrapper function and introduces a principled evaluation suite for the online setting, assessing diverse aspects, such as faithfulness, sufficiency, and coherence. Experiments reveal that classical gradient-based methods, such as Integrated Gradients (IG), can outperform recent approaches when adapted for temporal analysis. Building on this, we propose Shifted Window Integrated Gradients (SWING), which incorporates past observations in the integration path to systematically capture temporal dependencies and mitigate out-of-distribution effects. Extensive experiments consistently demonstrate the effectiveness of SWING across diverse settings with respect to diverse metrics. Our code is publicly available at https://github.com/AITRICS/Delta-XAI.