Probabilistic Textual Time Series Depression Detection

AI summary

Overview

Research area: Natural Language Processing for mental health — specifically, probabilistic regression of depression severity (PHQ-8 scores) from clinical interview transcripts, framed as a textual time-series problem.

Technical level: Intermediate. The paper is readable by anyone familiar with basic neural sequence modeling, but it assumes comfort with concepts such as likelihood-based training, calibration, and predictive distributions.

Scope: The paper introduces PTTSD, a probabilistic LSTM-plus-attention framework that predicts PHQ-8 depression severity with calibrated uncertainty from utterance sequences, evaluated on the DAIC-WOZ and E-DAIC interview corpora.

What This Paper Is About

Most automated systems that estimate depression severity from interview transcripts output a single number — for example, "PHQ-8 = 12" — with no indication of how confident the model is. That is a serious weakness in a clinical setting, where a prediction is far more useful if it comes with a trustworthy measure of uncertainty. PTTSD addresses this by predicting a full probability distribution over the PHQ-8 score (Gaussian or Student's-t) rather than a point estimate, and by adding a sequence-to-sequence variant that shows how the model's belief and confidence evolve utterance by utterance across an interview.

Key Contributions

1. A fully probabilistic sequence model. PTTSD jointly predicts the PHQ-8 score and its calibrated uncertainty from utterance-level textual time series, replacing point predictions with distributional outputs trained by negative log-likelihood.

2. Two model formulations. A sequence-to-one variant that predicts one distribution from the full transcript, and a sequence-to-sequence variant that produces a per-utterance distribution for every prefix, trained against the same session-level PHQ-8 label. The latter enables temporal interpretability even though the target itself does not change over time.

3. Calibration, temporal, and sensitivity analyses. The paper provides a three-part calibration assessment, an analysis of how uncertainty and error evolve across utterance positions, and a sensitivity study over the NLL loss weighting parameters.

4. Competitive text-only benchmark results plus qualitative case studies. PTTSD is evaluated on E-DAIC and DAIC-WOZ using both original and WhisperX re-transcribed interviews, and four participant case studies illustrate distinct uncertainty regimes.

Main Findings

• Competitive text-only accuracy on E-DAIC. Among the text-only systems compared, PTTSD (seq-to-seq with all-MiniLM-L6-v2) achieves the lowest test MAE of 3.85 and RMSE of 4.52. This variant also reports dev MAE 3.47 and RMSE 4.57. Results were obtained on WhisperX re-transcriptions of the original E-DAIC audio, similar to the approach of Sadeghi et al. (2024).

• Competitive accuracy on DAIC-WOZ. PTTSD seq-to-one with all-MiniLM-L6-v2 achieves the lowest test MAE of 3.55 and matches the best test RMSE of 4.77 reported by Fang et al.

• Performance drops on original E-DAIC transcripts. Using seq-to-seq with Gaussian NLL, all-MiniLM-L6-v2 reaches test MAE 4.63 and RMSE 5.56, while MentalBERT reaches 4.60 MAE and 5.58 RMSE. The authors note that the calibration and temporal-interpretability properties hold regardless of transcript source.

• Gaussian NLL is the most reliable loss. On E-DAIC (single run), Gaussian NLL gives test MAE 3.8603 and RMSE 5.0219; Student's-t NLL gives 3.9294 and 5.1488; MSE gives the lowest test MAE (3.6694) but worse dev behavior and the highest test RMSE among probabilistic losses (4.8760); MAE loss underperforms across all metrics (test MAE 4.1885, RMSE 5.4407).

• Attention and residual connections matter most. Against a full model with MAE 3.85, RMSE 4.99, and NLL 1.05, removing attention raises MAE by 53.29% (5.91) and RMSE by 48.28% (7.40), with NLL 1.56. Removing residual connections raises MAE by 38.52% (5.34) and RMSE by 33.42% (6.66), with NLL 1.43. Removing the variance head still degrades point accuracy (MAE +3.57% to 3.99, RMSE +5.04% to 5.24) and removes calibrated uncertainty altogether.

• Uncertainty is well calibrated. The Gaussian NLL model achieves an Expected Calibration Error of 0.0220 and near-ideal 68% coverage (66.2%). The MSE-based model is underconfident, with 84.0% coverage and ECE of 0.0675.

• Uncertainty tracks error. Predicted standard deviation correlates strongly with absolute error (r = 0.88, ρ = 0.64, p < 0.001).

• Uncertainty resolves as context accumulates. Average predicted uncertainty and absolute error are both high early in the interview, decrease and stabilize around timestep 250, and error rises again after timestep 300, which the authors attribute to data sparsity since few training sessions exceed that length.

• Loss weighting changes the shape of predicted uncertainty. Under the standard setting (α = β = γ = 1), predicted σ values span a moderate range (μ_σ = 4.0, x̃ = 3.8, σ_σ = 1.9) with the five severity categories staying within [0, 10]. Doubling β compresses predictions (σ_σ = 0.6) as does halving γ (σ_σ = 0.8), collapsing severity-level distinctions; doubling γ produces a wide distribution (μ_σ = 9.5, σ_σ = 6.2) that separates severity groups better but is miscalibrated. The standard configuration has the lowest test NLL (1.2439) versus 1.4519 (β = 2), 1.3766 (γ = 2), and 1.4459 (γ = 0.5).

• Four qualitative uncertainty regimes appear in case studies. Participant 634 shows Accurate & Confident behavior with a normalized error of 0.12 and narrow ±1σ bands. Participant 716 shows Calibrated Uncertainty with an error-to-uncertainty ratio of 0.41 and initial difficulty until utterance 70. Participant 640 is an Overconfident Error case: a true score of 17 is predicted as low severity with tight bands, giving a ratio of 7.68, outside 2σ. Participant 710 is Ambiguous but Correct: the predicted score peaks at ŷ = 8.63 at utterance 125 before declining to |y − ŷ| = 0.32 at utterance 140, with broad intervals (σ = 4.53).

• Seq-to-seq vs. seq-to-one is dataset-dependent. The seq-to-seq MiniLM variant outperforms seq-to-one on E-DAIC, while the pattern reverses on DAIC. The authors explicitly do not claim one formulation is strictly superior, describing them as complementary.

Methodology in Plain English

The researchers treat each interview transcript as an ordered sequence of utterances. Every utterance is converted into a vector using a pretrained sentence encoder — either the compact all-MiniLM-L6-v2 Sentence Transformer or the domain-adapted MentalBERT. Those vectors pass through an LSTM that produces hidden states, followed by multi-head self-attention with a residual connection, so each utterance representation can draw on the full conversational context.

Instead of outputting a single number, two small output heads predict the parameters of a probability distribution over the PHQ-8 score: a mean (the predicted severity) and a spread (the uncertainty). The model is trained by minimizing the negative log-likelihood of the true score under that predicted distribution, using either a Gaussian or a Student's-t form.

Two variants are trained. In seq-to-one, the attended representations are averaged over time and one distribution is produced. In seq-to-seq, a distribution is produced at every timestep and each is trained against the same session-level PHQ-8 label, which yields a trajectory of predicted means and uncertainties across the interview.

Data comes from DAIC-WOZ (189 sessions) and E-DAIC (275 sessions), using the official splits: 163/56/56 for E-DAIC and 107/35/56 for DAIC-WOZ. E-DAIC audio is re-transcribed with WhisperX, with results also reported on the original transcripts. Training uses Adam with a cosine-annealed learning rate, 50 epochs, early stopping on development MAE with patience 15, and log-transformed targets. Models are trained on a single NVIDIA A100-SXM4-80GB GPU, taking roughly 2 hours 23 minutes (about 172 seconds per epoch), with 2,703,403 trainable parameters. Results are averaged over three seeds and code is publicly released.

Why This Matters

Impact on research. The paper shifts the conversation in text-based depression detection from raw point accuracy toward calibrated, interpretable uncertainty. It shows that probabilistic training can serve as a form of regularization that slightly improves point estimates, and that per-utterance distributions can be extracted from data whose label is only available at the session level — an idea transferable to other session-level prediction problems with sequential inputs.

Real-world applications:

• Digital therapy and remote care, where clinician access is limited and automated severity estimates need explicit confidence signals before they inform triage.
• Clinical decision support, letting a practitioner distinguish a prediction the model stands behind from one that warrants caution.
• Interview review and training, using the temporal trajectories to highlight which utterances drove a prediction shift or where the model remained ambiguous.
• Screening at scale, where calibrated intervals allow downstream systems to flag only the cases with sufficiently confident estimates.

Industry relevance. The framework is end-to-end and needs no prompt engineering or handcrafted features, and the released implementation makes it straightforward to adapt to other transcript-based regression tasks. The finding that uncertainty modeling can improve point accuracy as well as add a confidence signal is a practical argument for adopting probabilistic heads in production mental-health NLP pipelines.

Future Directions

1. Multimodal extension. The authors propose complementing text with prosodic and visual signals, which the current text-only framework cannot use.

2. Clinical validation with human users. The paper explicitly does not assess whether uncertainty estimates help practitioners make decisions, and calls for human-centered evaluations with therapists or end users.

3. Context-aware utterance encoding. Utterances are currently encoded independently by pretrained language models without context-aware finetuning, potentially missing local coherence and discourse-level cues.

4. Better aggregation for seq-to-one. The current variant uses masked average pooling over time; the authors suggest attention-weighted pooling as a viable direction.

Target Audience

This paper is most useful for NLP and machine learning researchers working on clinical or mental-health text applications, particularly those interested in uncertainty quantification, calibration, and interpretability. It is also relevant to applied data scientists building decision-support or triage systems who need to know when a model's prediction can be trusted, and to clinical informatics researchers evaluating whether distributional outputs could inform real practitioner workflows. Some familiarity with sequence models and probabilistic regression is helpful.

Authors’ abstract

Accurate and interpretable predictions of depression severity are essential for clinical decision support, yet existing models often lack uncertainty estimates and temporal interpretability. We propose PTTSD, a Probabilistic framework for Depression Detection from clinical interview utterance sequences that predicts PHQ-8 scores while modeling calibrated uncertainty. PTTSD includes sequence-to-sequence and sequence-to-one variants, both combining LSTMs, self-attention, and residual connections with Gaussian or Student's-t output heads trained via negative log-likelihood. The sequence-to-sequence variant enables temporal analysis of how predictive confidence evolves over an interview, despite the target being a single session-level score. Evaluated on E-DAIC and DAIC-WOZ, PTTSD achieves competitive performance among text-only systems (e.g., MAE = 3.85 on E-DAIC, 3.55 on DAIC) and produces well-calibrated prediction intervals. Ablations confirm the value of attention and probabilistic modeling, while a three-part calibration analysis and qualitative case studies highlight the clinical relevance of uncertainty-aware prediction.