Addressing Mark Imbalance in Integration-free Neural Marked Temporal Point Processes

AI summary

Overview

• Research area: Machine learning for event-stream modelling — specifically neural Marked Temporal Point Processes (MTPPs) and the treatment of class (mark) imbalance in next-event prediction.
• Technical level: Advanced. The paper assumes familiarity with conditional intensity functions, conditional joint probability density functions, improper integration, cumulative distribution functions, inverse transform sampling, and negative log-likelihood training of point processes.
• Scope: The paper proposes a thresholding scheme for imbalanced marks in MTPPs, paired with a new neural model (IFNMTPP) that predicts the mark first and the time second without expensive numerical improper integration.

What This Paper Is About

Real-world event streams often contain some marks (event types) that occur constantly and others that occur very rarely, and existing MTPP models are biased toward the frequent ones. The authors show this imbalance badly degrades prediction of rare marks — for example, using the SAHP model, macro-F1 on frequent marks reaches 0.6183 ± 0.0010 on Retweet versus 0.0266 ± 0.0135 on rare marks. Their goal is to improve next-event mark and time prediction, especially for rare marks, by learning per-mark thresholds that rescale mark probabilities by each mark's prior probability, and by building a neural model that predicts the mark before the time so that those thresholds can be applied.

Key Contributions

1. Identifying an overlooked problem: The study investigates how mark imbalance affects MTPP next-event prediction, an issue the authors state existing MTPP studies overlook.
2. A thresholding solution for MTPP: It introduces what the authors describe as the first solution to address mark imbalance in MTPP, learning thresholds that tune the mark probability normalized by the mark's prior probability, rather than predicting the mark directly from the mark probability.
3. A unified integral formulation: It shows that two improper integrations — one for the mark probability p*(m) and one for time prediction — can be unified into a single function Γ*(m, t).
4. A new neural model (IFNMTPP): It proposes the Integration-free Neural Marked Temporal Point Process, which approximates that unified improper integration instead of using computationally expensive numerical methods, and supports both time sampling and mark-probability estimation.

Main Findings

• Thresholding strongly helps rare marks: On Retweet, the full method (ours) reaches macro-F1 0.2010 ± 0.0082 on rare marks versus 0.0333 ± 0.0082 for the same model without thresholding. The authors report that "ours" performs much better on rare and all-mark prediction, and shows comparable performance on frequent marks, which they say implies improved recall for frequent marks.
• Mark-first prediction order is necessary for thresholding: "ours" beats "time-mark-with-thresholding" on rare-mark prediction on all datasets (for example, USearthquake rare marks 0.0339 ± 0.0051 versus 0.0298 ± 0.0093; StackOverflow rare marks 0.1476 ± 0.0041 versus 0.1023 ± 0.0094). The authors also note that "time-mark-w/o-thresholding" is lower than "time-mark-with-thresholding", concluding that predicting the mark first is more suitable for handling mark imbalance with thresholding.
• Time prediction benefits from conditioning on the mark: The paper reports that predicting time from p*(t|m) slightly outperforms predicting it from p*(t). The reported MAE values are mixed by dataset — on StackOverflow, ours gives 0.5212 ± 0.0142 for all marks versus 0.6417 ± 0.0127 for time-mark-with-thresholding, and on USearthquake 0.6856 ± 0.0063 versus 0.8516 ± 0.2378, while on Retweet all-mark MAE is 2515.1 ± 6.5029 for ours versus 2504.5 ± 4.4738.
• Thresholding beats resampling: The authors compare against oversampling and undersampling and report that "ours" consistently outperforms both. On USearthquake all marks, macro-F1 is 0.1382 ± 0.0071 for ours, 0.0647 ± 0.0165 for oversampling, and 0.0576 ± 0.0121 for undersampling. They observe that the resampling ratio affects performance but is hard to set correctly for different marks on different datasets.
• Competitive time prediction against existing MTPP models: The paper compares against FullyNN, THP, SAHP, AttNHP, and Marked-LNM, and reports the best MAE on Retweet (2515.1 ± 6.5029 versus FullyNN 5126.0 ± 854.88, SAHP 3320.0 ± 242.70, THP 3601.1 ± 231.52, AttNHP 3551.1 ± 12.611, Marked-LNM 2559.8 ± 5.9380) and on USearthquake (0.6856 ± 0.0063 versus FullyNN 1.2684 ± 0.3715, SAHP 0.7608 ± 0.0588, THP 0.7322 ± 0.0078, AttNHP 6.4583 ± 2.2939, Marked-LNM 0.7646 ± 0.0026). On Taobao, however, Marked-LNM reports lower MAE than ours for all marks (0.2058 ± 0.0079 versus 0.3324 ± 0.0579) and rare marks (0.2043 ± 0.0091 versus 0.3385 ± 0.0627); the Marked-LNM frequent-mark value is not present in the provided content.
• Model fidelity checked on synthetic data: The authors evaluate IFNMTPP's fidelity using five synthetic datasets; they state the synthetic results (Section E.2) demonstrate high fidelity compared with other MTPP models, but those numbers are not included in the provided content.
• Imbalance is visible in the probability surface: Figure 2 shows mark frequency distributions across Retweet, USearthquake, and StackOverflow and the corresponding p*(m, t) per mark, illustrating that if mark k1 is frequent and k2 is rare, p*(k1, t) > p*(k2, t) for most of the time.

Methodology in Plain English

The authors start from the standard MTPP setup, where the joint distribution of the next event's mark and time is written as p*(m, t) = λ*(m, t)F*(t), with F*(t) the probability that nothing has happened since the last event.

Predict the mark first, then the time. Most prior work predicts time first from p*(t) and then picks the mark from p*(m | t) at that predicted time. The authors argue this is bad for imbalance handling, because thresholds would have to be re-learned for every possible time. Instead, they predict the mark from p*(m), which does not depend on time, and only then predict the time from p*(t | m).

Thresholding on prior-normalized probability. For each mark m they compute the ratio r_m = p*(m) / p̄*(m), where p̄*(m) is simply the proportion of mark m in the training set. A rare mark can have a low p*(m) but still a high r_m, signalling that it is unusually likely relative to its own base rate. The predicted mark is then the one maximizing (r_m − ε_m), where the threshold ε_m is learned to maximize the F1 score of the pairwise comparison of mark m against all other marks.

Unifying two integrals into one. Computing p*(m) and the mean time both involve integrating to infinity, which has no analytic solution and is expensive numerically. The authors define Γ*(m, t) as the integral of p*(m, τ) from time t to positive infinity, then rewrite p*(m) = Γ*(m, t_l) and F*(t | m) = (Γ*(m, t_l) − Γ*(m, t)) / Γ*(m, t_l). One function now serves both the mark probability and time sampling.

The IFNMTPP model. Because Γ*(m, t) is an improper integral over an infinite interval, they approximate it with a neural network. A history encoder (an LSTM in the reported architecture) produces context; each mark gets a non-negative vector v_m, and the input to the Integral Estimation Module is f(m, t) = v_m(t − t_l) + b_m. The module uses fully connected layers with non-negative weights and monotonic-increasing, unbounded activations, ending with σ(x) = 1/(1 + e^x), so the model is intrinsically monotonically decreasing in t. Outputs s*(m, t) are normalized by a partition function Z(H_{t_l}) = Σ_m s*(m, t_l), giving Γ*(m, t) = s*(m, t) / Z(H_{t_l}). Proposition 3.1 states that the output is Γ*(m, t) when its gradient is −p*(m, t).

Sampling time. To draw times from p*(t | m), the authors use Inverse Transform Sampling, solving F*(t^i | m) = u^i where u^i is a uniform sample, using the bisection method since the CDF is monotonic. They note Thinning Algorithm is unsuitable here because it explicitly requires the expression of p*(t | m), which is typically unknown. The expected time is then approximated by the average of N samples.

Training. IFNMTPP is trained with the negative log-l

Authors’ abstract

Marked Temporal Point Process (MTPP) has been well studied to model the event distribution in marked event streams, which can be used to predict the mark and arrival time of the next event. However, existing studies overlook that the distribution of event marks is highly imbalanced in many real-world applications, with some marks being frequent but others rare. The imbalance poses a significant challenge to the performance of the next event prediction, especially for events of rare marks. To address this issue, we propose a thresholding method, which learns thresholds to tune the mark probability normalized by the mark's prior probability to optimize mark prediction, rather than predicting the mark directly based on the mark probability as in existing studies. In conjunction with this method, we predict the mark first and then the time. In particular, we develop a novel neural MTPP model to support effective time sampling and estimation of mark probability without computationally expensive numerical improper integration. Extensive experiments on real-world datasets demonstrate the superior performance of our solution against various baselines for the next event mark and time prediction. The code is available at https://github.com/undes1red/IFNMTPP.