Similar Predictive Fit but Different Latent Dynamics: Characterizing Learned Dynamical Structure in Personalized Models of Brain Disorders

AI summary

Overview

Research area: Machine learning for clinical neuroscience — interpretability and evaluation of personalized latent-dynamical models built on EEG foundation-model representations, with epilepsy as the clinical testbed. The paper sits at the intersection of representation learning, latent time-series modeling, and model evaluation.

Technical level: Intermediate. The framing is accessible, but the pipeline involves a pretrained CNN–Transformer encoder, global k-means state discretization, multinomial logistic transition distributions (mLTD) with group-sparse regularization, and AUROC / Mann–Whitney evaluation.

Scope (one sentence): The paper tests whether two clinical groups (epilepsy vs. non-epilepsy) that are equally predictable under their own personalized latent-transition models nonetheless learn substantially different internal dependency structures.

What This Paper Is About

Personalized clinical AI models are usually judged by how well they predict — but good prediction does not automatically tell you what the model learned internally. The authors ask a simple, targeted question: if two groups of subjects are equally predictable under their own personalized latent-state models, does that mean those models have learned the same temporal dynamics? Using scalp EEG from the Temple University Epilepsy Corpus (TUEP), they show the answer is no: epilepsy and non-epilepsy subjects have nearly identical held-out predictive fit, yet their learned latent transition-dependency graphs differ sharply in density.

Key Contributions

1. Separating two evaluation axes. The paper formally distinguishes predictive validity (can the model predict that same subject's unseen trajectory?) from characterization of learned dynamical structure (how does the model organize temporal dependencies within that individual?).
2. Demonstrating a fit–structure dissociation. Comparable within-subject predictive fit coexists with substantially denser learned dependency structure in epilepsy subjects, and this pattern holds at two latent-state resolutions, k = 4 and k = 6.
3. Showing that personalized dependency structure carries clinically associated information. Graph-derived features from the subject-specific dependency matrix Wₙ give moderate epilepsy vs. non-epilepsy discrimination under stratified 5-fold subject-wise cross-validation.
4. Providing sensitivity and robustness checks. The authors report null-model prevalence, non-null re-analysis, group-mean heatmaps, ROC curves, and two distinct lag-wise checks to test whether the structural difference is an artifact.

Main Findings

• Predictive fit is matched across groups. At k = 4, within-subject held-out cross-validation log-likelihood was nearly identical between groups (−0.992 vs. −0.991; Mann–Whitney p = 0.95), and within-subject next-state AUROC was similarly matched (0.855 vs. 0.861; p = 0.54). At k = 6, held-out CV log-likelihood also did not differ (−1.399 vs. −1.370; p = 0.18).

• Learned dependency structure is denser in epilepsy. At k = 4, epilepsy subjects retained 10.01 nonzero dependencies on average versus 5.67 for non-epilepsy subjects (p = 1.1 × 10⁻⁷). Normalized dependency densities were 0.626 vs. 0.354 — roughly 62.6% versus 35.4% of all possible state-to-state dependencies.

• The densification persists at the finer resolution. At k = 6, nonzero dependencies were 19.90 vs. 13.46 (p = 5.2 × 10⁻⁵), with normalized densities of 0.553 vs. 0.374. The difference therefore survives normalization by the larger number of possible dependencies at k = 6.

• Self-dependency differs only at the coarser resolution. Mean diagonal self-dependency was higher in epilepsy at k = 4 (0.219 vs. 0.152; p = 1.7 × 10⁻⁴) but not at k = 6 (0.201 vs. 0.212; p = 0.18).

• Graph features retain moderate group discrimination. At k = 4, group-classification AUROCs were 0.697 (scalar summaries), 0.679 (graph-derived features), and 0.615 (vectorized full matrix). At k = 6, the corresponding values were 0.677, 0.645, and 0.654. The abstract highlights graph-derived features at AUROC 0.68 (k = 4) and 0.65 (k = 6). These are distinct from the within-subject next-state AUROC of about 0.86.

• Null models are unevenly distributed but do not explain the effect. At k = 4, all-zero Wₙ graphs occurred in 29/99 non-epilepsy vs. 8/99 epilepsy subjects; at k = 6, 10/99 vs. 8/99. Excluding null models, scalar group AUROC remained above chance (0.665 at k = 4; 0.722 at k = 6).

• Predictive fit stays matched across the lag grid. In the full-cohort analysis at k = 4 (n = 198), held-out CV log-likelihood did not differ between groups at any lag (p ≥ 0.37), so the structural difference is not explained by one group systematically selecting a more or less predictive lag.

• Motivating preliminary results (Appendix A, explicitly not independent validation). The frozen encoder achieved TUAB AUC 85.24 ± 5.61 (accuracy 80.25 ± 5.91) and TUEP AUC 67.35 ± 5.91 (accuracy 69.00 ± 3.24; sensitivity 75.00 ± 14.14; F1 70.37 ± 4.01). A cross-encoder occupancy comparison with an independently pretrained LUNA-Base encoder found essentially uncorrelated absolute group-level occupancy (r = −0.06 non-epilepsy; r = 0.03 epilepsy), but after Hungarian alignment, epilepsy-associated occupancy changes showed descriptive concordance r_Δ = 0.96 (exploratory, four matched states).

Methodology in Plain English

The authors reuse a small EEG foundation model rather than training a new one. The encoder is a CNN–Transformer of roughly 1.46M parameters, pretrained by self-supervised reconstruction on 2,259,998 non-overlapping 30-second segments from the Temple University EEG Corpus (TUEG), about 18,800 hours of EEG. All EEG is band-pass filtered 0.3–75 Hz, notch filtered at 60 Hz, mapped to the standard 19-channel 10–20 montage, resampled to 200 Hz, normalized to approximately ±100 μV, and scaled to [−1, 1].

The encoder is frozen for every TUEP analysis — no fine-tuning on the clinical task. Each 30-second segment becomes a 128-dimensional vector. A single global k-means model, fit across all subjects, defines a shared latent-state space, so state labels mean the same thing for everyone. k = 4 is the primary resolution (chosen from an elbow in a preliminary analysis of the TUEP representation space), with k = 6 as a robustness check; the authors note that individual states are not assumed to correspond across the two resolutions.

Each subject then becomes a discrete state sequence, one label per 30-second segment. The temporal model is fit independently per subject: mLTD uses group-sparse regularization across temporal lags to estimate which directed state-to-state dependencies actually help predict the next state, producing a sparse, directed, weighted k × k matrix Wₙ. Entries are predictive dependency strengths, not Markov transition probabilities, and do not sum to one; diagonal entries are within-state self-dependencies. Maximum lag L ∈ {1, …, 10} (30 s to 300 s of history) and sparsity λ ∈ {10⁻⁶, 10⁻⁴, 10⁻², 1} are chosen per subject by 5-fold time-series cross-validation maximizing mean held-out log-likelihood. Graphs regularized down to all zeros are labelled null models and kept in the primary analysis.

Evaluation runs on two axes. Predictive validity uses mean held-out CV log-likelihood plus within-subject next-state AUROC. Learned dynamical structure uses the count of retained nonzero dependencies Eₙ and the normalized density Dₙ = Eₙ / k². Group comparisons use Mann–Whitney tests. As a secondary analysis, Wₙ is represented at three levels of detail (scalar summaries, graph-derived features, and the vectorized full matrix) and each is fed to L2-regularized, class-balanced logistic regression with stratified 5-fold subject-wise cross-validation to see whether structure alone carries epilepsy-associated information. Clinical labels are never used during representation extraction, clustering, or personalized model fitting — only in post-estimation evaluation.

Why This Matters

Impact on research. The paper argues that predictive equivalence is not sufficient evidence that personalized models have learned equivalent internal dynamics. This is a model-evaluation claim rather than a classification result: the authors explicitly say the goal is not to classify epilepsy. It motivates treating structural characterization as a distinct, legitimate evaluation step — one that can be performed before individual latent states have any established clinical interpretation. It also connects to the broader foundation-model concern that large pretrained models transferred across populations may achieve good performance without transparent internal semantics.

Potential real-world applications (as directions the paper motivates, not validated uses):

• Closed-loop brain stimulation. The introduction cites machine learning for personalized adaptive interventions including closed-loop brain stimulation as a motivating clinical direction for personalized dynamical models.
• Intervention-aware patient world models. The proposed evaluation step is framed as a precursor to models that predict how an individual's trajectory changes under intervention, conditioned on a perturbation uₜ and characterized by ΔWₙ = Wₙᵖᵒˢᵗ − Wₙᵖʳᵉ.
• Cross-encoder and cross-population model auditing. The framework gives a concrete way to probe whether a frozen encoder's downstream dynamics are robust or encoder-dependent, building on the exploratory cross-encoder occupancy comparison with LUNA-Base.
• Structured clinical research on dynamical disorders. Because epilepsy is conceptualized as a dynamical disorder involving transitions among brain states, a per-subject dependency graph offers a research descriptor of how a subject moves between states, complementing occupancy-based measures of where and how often the subject visits them.

Industry relevance. For teams building clinical foundation models or regulatory-facing evaluation pipelines, the takeaway is directly operational: reporting AUROC and held-out likelihood alone can mask substantial structural differences in what personalized models learn. The paper's stated responsible-use position is that these dependency graphs are research descriptors of model-learned latent dynamics — not diagnostic biomarkers, causal brain graphs, or physiological connectivity maps — and should not be used for autonomous diagnosis, treatment selection, or intervention planning without prospective clinical validation.

Future Directions

1. Condition dynamics on interventions. Extend the framework to P(x_{t+1} | x_{t:t−L+1}, uₜ) and quantify intervention-associated structural change via ΔWₙ, which requires longitudinal or intervention-conditioned data and explicit causal assumptions.
2. Establish cross-encoder concordance of subject-specific Wₙ. The personalized analyses use representations from a single frozen EEG foundation model; whether subject-level dependency graphs agree across independently pretrained encoders remains to be established (the existing cross-encoder comparison is only group-occupancy-level, exploratory, and matches only four states).
3. Test stability and generalizability. Prospective, test–retest, and longitudinal analyses are needed to establish whether the learned dynamical structure is stable and clinically generalizable.
4. Resolve the null-model asymmetry and assign clinical semantics. Higher null-model prevalence among non-epilepsy subjects at k = 4 may partly contribute to the density difference (though non-null sensitivity analyses remain above chance); and the unsupervised k-means states still lack established clinical or physiological semantics, so state-level interpretation remains an open validation problem.

Target Audience

This paper is most useful to machine learning researchers working on clinical or health-related foundation models, especially those interested in model interpretability and evaluation beyond predictive metrics. It will also appeal to computational neuroscience and EEG researchers studying epilepsy as a dynamical disorder, to methodologists interested in latent-state and transition-dependency modeling (mLTD, group-sparse regularization), and to clinical AI teams who need a concrete argument for why predictive validity and structural characterization should be reported separately. Readers should be comfortable with latent-space modeling, cross-validation design, and AUROC interpretation; the paper's framing is accessible, but the methods are not introductory.

Authors’ abstract

As AI models move toward clinical decision-making and personalized treatment, understanding \emph{what} a model learns is important beyond predictive accuracy alone. We investigate whether personalized latent dynamics reveal clinically associated differences even when predictive fit is similar. A lightweight CNN--Transformer EEG foundation model pretrained on the Temple University EEG Corpus (TUEG) extracts segment-level representations. Using the Temple University Epilepsy Corpus (TUEP), representations are mapped to a shared latent-state space, and sparse multinomial logistic transition distributions (mLTD) are fit independently to each subject to obtain personalized transition-dependency graphs $W_n$. Analyses include $n{=}198$ subjects (99 epilepsy / 99 non-epilepsy). At $k{=}4$, epilepsy subjects exhibit substantially denser learned dependency structure ($p{=}1.1\times10^{-7}$), with the same pattern at $k{=}6$ (19.90 vs. 13.46; $p{=}5.2\times10^{-5}$). Graph-derived features provide moderate group discrimination under 5-fold subject-wise cross-validation (AUROC 0.68 at $k{=}4$; 0.65 at $k{=}6$). In contrast, held-out log-likelihood is nearly identical between groups at $k{=}4$ ($-0.992$ vs. $-0.991$; $p{=}0.95$), with similarly matched next-state prediction (AUROC 0.855 vs. 0.861; $p{=}0.54$). Thus, similar predictive fit does not imply similar learned dynamics: groups can be comparably predictable while differing substantially in the internal dynamical structure learned by personalized models. This distinction motivates evaluating learned structure alongside predictive performance in personalized clinical models.