ITPP: Learning Disentangled Event Dynamics in Marked Temporal Point Processes

AI summary

Overview

Research area: machine learning for continuous-time event sequence modeling, specifically Marked Temporal Point Processes (MTPPs).

Technical level: Advanced. The paper assumes familiarity with neural ordinary differential equations, self-attention, and point process likelihoods.

Scope: The paper proposes ITPP, a channel-independent encoder-decoder architecture with an ODE backbone and a type-aware inverted self-attention layer, and evaluates it on four real-world and two synthetic datasets against 10 baselines.

What This Paper Is About

Most Marked Temporal Point Process models squeeze information from every event type into one shared latent vector, a strategy the authors call "channel mixing." When a user's history spans unrelated categories such as clothing, smartphones, and food, that single entangled representation can blur the distinct dynamics of each type, hurting accuracy and inviting overfitting. ITPP instead separates each event type into its own channel, simulates each channel's evolution independently, then explicitly re-links the channels to capture how one event type excites or inhibits another.

Key Contributions

1. A channel-independent architecture for MTPP modeling, named ITPP, built as an encoder-decoder framework with an ODE-based backbone, which disentangles the heterogeneous dynamics of different event types.
2. A type-aware inverted self-attention module that captures correlations between event channels, using channel-specific biases alongside attention to preserve inherent connections between event types while also incorporating state-based correlations.
3. A reformulation of the channel-independence idea from multivariate time series forecasting into the MTPP setting, which the authors argue differs due to asynchronization and semantic discretization.
4. Extensive experiments across probabilistic evaluation, prediction evaluation, intensity recovery, and ablation studies on four real-world and two synthetic datasets, compared against 10 baselines.

Main Findings

• Best joint likelihood on every real-world dataset: ITPP achieves the best joint NLL of arrival times and marks (TM-NLL) on StackOverflow, MIMIC, Taobao, and Earthquake. Its scores are 2.106 (StackOverflow), 1.054 (MIMIC), -1.189 (Taobao), and 1.692 (Earthquake).
• Trade-off between time and mark losses: On datasets with many event types (StackOverflow with 22 types, MIMIC with 75, Taobao with 17), ITPP improves mark NLL (M-NLL) substantially at a slight cost in time NLL (T-NLL). On Earthquake, which has only 7 event types, it focuses more on the time component. The authors attribute this to the channel-independent design making marks easier to distinguish as type count grows.
• Self-attention baselines underperform RNN baselines: THP, SAHP, and AttNHP do not beat strong RNN-based models such as LogNormMix and NHP, even on long-sequence datasets (StackOverflow, Taobao) where long-range dependency capture should help.
• Channel-mixing ODE models are inconsistent: NeuralODE and ODE-GRU show competitive probabilistic fits only on StackOverflow, the dataset with the largest number of events (142,777), suggesting vulnerability to overfitting without large data.
• Prediction performance: Using RMSE for time and F1 for marks, ITPP secures top performance on five of the six indicators across StackOverflow, MIMIC, and Earthquake, falling only marginally short on F1 for Earthquake. ITPP reaches RMSE 1.007 on StackOverflow, 0.831 on MIMIC, and 1.226 on Earthquake, with F1 of 0.313, 0.856, and 0.319 respectively.
• Intensity recovery: On the synthetic Poisson and Hawkes datasets, where ground-truth intensities are analytically known, ITPP significantly outperforms existing intensity-based MTPP models in Mean Absolute Percentage Error (MAPE). Figure 5 visualizes the top three models on Hawkes and shows ITPP's predicted intensity tracking the ground truth more closely across all three event types.
• Ablation on channel independence: Removing the channel-independent design (a variant called ITPP w/o CI) causes a significant performance drop across the five metrics used in the probabilistic and prediction analyses, with the effect especially pronounced on the smaller-scale Earthquake dataset.
• Ablation on inverted self-attention: Removing the inverted self-attention module causes a substantial drop in performance, suggesting event types in these datasets are highly correlated and missing those dependencies undermines predictive power.
• Resistance to overfitting: Training and testing loss curves on the Earthquake dataset show severe overfitting in LogNormMix and ODE-GRU, whose testing loss surges while training loss keeps decreasing. ITPP shows stronger resistance.
• Synthetic supplementary results: On the Hawkes dataset, ITPP reports TM-NLL of 0.316 versus 0.323 for the next best (ODE-GRU), RMSE of 0.315, and F1 of 0.423.

Methodology in Plain English

The model works in three stages under an "encoding-correlation-decoding" design.

First, encoding: instead of merging all event types into one hidden state, ITPP treats each event type as its own channel. Within a channel, two kinds of state change are simulated. Between events, the state drifts smoothly according to an ordinary differential equation governed by a learned drift function; at an event, the state jumps abruptly via a learned update network. Channels are simulated simultaneously with shared parameters, so each channel keeps its own time-varying context embedding without interference from others.

Second, correlation: a type-aware inverted self-attention layer treats each channel as a token and applies attention across channels rather than across time steps. Query, key, and value projections share mapping matrices across channels but use channel-specific bias terms, so the model keeps type identity while still learning which event types influence which others.

Third, decoding: each attention output is decoded independently into a type-specific intensity value, using a decoding network whose parameters are shared across channels. The total intensity is the sum over channels, and the joint probability of the next event combines the mark's intensity with the integral of the total intensity since the last event.

Training uses maximum log-likelihood estimation with a negative log-likelihood loss, with gradients computed by backpropagation using the adjoint sensitivity method. Implementation is in JAX 0.5.3 with Python 3.11.5, CUDA 11.6, Equinox 0.12.2, the Tsit5 solver from Diffrax 0.7.0, and the AdamW optimizer from Optax 0.2.4 with an initial learning rate of 0.001, on an Intel Xeon Gold 5218 CPU and NVIDIA GeForce RTX 4090 GPUs. All results are averaged across 5 runs.

Why This Matters

Impact on research: The paper argues that channel independence, already influential in multivariate time series forecasting, transfers productively to MTPP modeling once reformulated for asynchronous, semantically discrete event data. It also suggests that ODE-based MTPP models may not be inherently weak, but rather have been held back by channel-mixing encoders.

Real-world applications:

• Healthcare, where MIMIC provides sequences of ICU clinical visits and disease diagnoses across 75 disease types.
• Finance, where event timing and type both carry signal.
• Traffic and other continuous-time sensor-driven systems.
• E-commerce user behavior, illustrated with the Taobao dataset of timestamped browsing and purchasing across 17 product categories.
• Seismic monitoring, illustrated with the Earthquake dataset covering events from 1996 to 2023 across 7 magnitude-based types.

Industry relevance: Any system that logs asynchronous, typed events and needs calibrated next-event timing and type predictions can use the framework. The paper reports a public implementation at github.com/AnthonyChouGit/ITPP, and the overfitting-resistance results matter most for practitioners working with small datasets, where the authors show competing models degrading badly on test loss.

Future Directions

• Determining more precisely when channel independence helps most: the paper observes the benefit is especially pronounced on smaller-scale data such as Earthquake, and that ITPP shifts focus toward mark loss as type count grows, but leaves the underlying mechanism to be characterized.
• Exploring the time-versus-mark trade-off: ITPP improves M-NLL at slight cost to T-NLL on high-type datasets, and it is not established how to balance these components deliberately.
• Extending the framework beyond intensity-based decoding, given the paper notes that alternative decoders such as mixtures of log-normal distributions (LogNormMix) and diffusion-based approaches have been applied to MTPP settings.
• Combining channel independence with architectures beyond the ODE backbone tested here, since the paper frames its contribution as an architectural principle rather than a single fixed model.

Target Audience

Researchers and graduate students in machine learning working on point processes, asynchronous event sequence modeling, or neural differential equations; practitioners building prediction systems for healthcare records, e-commerce activity, traffic, or seismic monitoring; and anyone interested in how channel independence, developed in time series forecasting, carries over to irregular event data.

Authors’ abstract

Marked Temporal Point Processes (MTPPs) provide a principled framework for modeling asynchronous event sequences by conditioning on the history of past events. However, most existing MTPP models rely on channel-mixing strategies that encode information from different event types into a single, fixed-size latent representation. This entanglement can obscure type-specific dynamics, leading to performance degradation and increased risk of overfitting. In this work, we introduce ITPP, a novel channel-independent architecture for MTPP modeling that decouples event type information using an encoder-decoder framework with an ODE-based backbone. Central to ITPP is a type-aware inverted self-attention mechanism, designed to explicitly model inter-channel correlations among heterogeneous event types. This architecture enhances effectiveness and robustness while reducing overfitting. Comprehensive experiments on multiple real-world and synthetic datasets demonstrate that ITPP consistently outperforms state-of-the-art MTPP models in both predictive accuracy and generalization.