Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective

AI summary

Overview

Research area: Generative machine learning for event sequence forecasting, with a benchmark spanning statistical baselines, diffusion models, and temporal point process methods.

Technical level: Advanced. The paper assumes familiarity with diffusion models, latent variable models, optimal transport, discriminator-based two-sample tests, and the Hungarian assignment algorithm.

Scope: A benchmark and a new aggregation algorithm for judging and improving multi-horizon forecasts of heterogeneous event sequences along two axes — prediction accuracy and distributional realism.

What This Paper Is About

Event sequences are chains of timestamped events, each carrying a mix of numerical features (such as a transaction amount) and categorical features (such as a merchant category code), and they arise in finance, healthcare, and business process intelligence. Existing work on such data, mostly temporal point processes, evaluates forecasts almost exclusively on how close a single predicted continuation is to the actual one, ignoring whether generated futures look like genuine samples from the underlying distribution. The paper reframes forecasting as conditional generative modeling, builds a benchmark that scores both accuracy and realism, shows that no method currently does well at both, and introduces a procedure that converts a realism-strong diffusion model into a state-of-the-art predictor.

Key Contributions

1. A unified evaluation setup for heterogeneous event sequences. The authors implement statistical baselines (Repeat, HistSampler, Mode), GPT-2-style autoregressive and non-autoregressive models (AR, MT), a latent diffusion model (LDM) adapted from TabSyn, a data-space diffusion model (DDM) adapted from CDiff, and the state-of-the-art non-diffusion method DEF, all adapted to forecasting events with mixed numerical and categorical features.

2. A benchmark with two new metrics. Alongside three curated real-world card transaction datasets (Gender, Age, Alphabattle), the paper introduces TMS (Temporal Matching Score) for prediction accuracy and cDFS (Conditional Detection Fooling Score) for distributional realism. TMS replaces hand-tuned edit costs with baseline-rescaled per-feature normalization and a permutation matched by a Hungarian assignment; cDFS is a history-conditioned variant of the classifier two-sample test.

3. Evidence of a fundamental accuracy–realism trade-off. No evaluated method is strong on both TMS and cDFS; the paper shows the Pareto front explicitly per dataset.

4. Wasserstein BaryBooster. An inference-time aggregation procedure that draws samples from a base generator and returns the continuation minimizing the sum of TMS transport costs to them, in the spirit of a discrete Wasserstein barycenter. Applied on top of LDM, it produces the highest TMS across all three datasets without retraining and without collapsing realism entirely.

Main Findings

• The two quality axes are nearly disjoint. At horizon L=32, LDM is the only generative model with a non-trivial cDFS (0.856 on Alphabattle, 0.907 on Gender, 1.000 on Age), while every other learned model — including the state-of-the-art DEF — collapses to cDFS near zero, meaning a simple discriminator separates its continuations from real ones with near-perfect accuracy.

• Accuracy rankings invert the realism rankings. DEF is the best or second-best method on TMS across all three datasets (0.424 on Alphabattle, 0.413 on Gender, 0.508 on Age) yet has cDFS of 0.000, 0.002, and 0.000 respectively.

• A trivial baseline is competitive. Repeat, which simply returns the last L events of the history with their inter-arrival times, is the strongest baseline on Alphabattle (TMS 0.440) and second-best on Gender (0.361) and Age (0.454), beating every learned method on Alphabattle. It also lies on the Pareto front of the baseline pool on all three datasets.

• GPT-style models underperform. AR and MT stay below HistSampler on both metrics, and their numerical features collapse toward the conditional median, losing nearly all variance — the failure mode the paper illustrates with its roulette toy example.

• Repeat beats the diffusion models on accuracy. LDM scores TMS 0.405, 0.318, and 0.426; DDM scores 0.251, 0.107, and 0.441 — all below Repeat except DDM on Age.

• BaryBoosting lifts diffusion models to state of the art. LDM-WB achieves the highest TMS on every dataset (0.495 on Alphabattle, 0.436 on Gender, 0.522 on Age) while keeping a non-trivial cDFS (0.177, 0.256, 0.224). DDM-WB improves more modestly (0.274, 0.136, 0.480), reflecting DDM's weaker base generations.

• The benefit is not restricted to the optimized metric. BaryBoosting LDM also improves metrics it does not optimize directly: the amount R1 rises from 0.479 to 0.583 on Alphabattle, 0.180 to 0.334 on Gender, and 0.216 to 0.438 on Age; time R1 rises from 0.148 to 0.337, 0.144 to 0.331, and 0.074 to 0.331.

• Boosting helps diffusion models most. Gains are largest for diffusion models, absent for DEF, and small for the remaining non-diffusion methods. The effect is strongest at horizon L=4, where boosting moves LDM from near-bottom TMS to best-in-class.

• Cross-horizon rollouts do not pay off. Models trained at a shorter horizon and rolled out semi-autoregressively to a longer one are almost never better than models trained directly at the target horizon.

• The trade-off transfers outside finance. On the two temporal point process corpora Retweet and Amazon (Appendix L), which have far fewer and much shorter sequences and only a timestamp plus a categorical mark per event, the same trade-off and the same leaders on each axis carry over.

• cDFS ranking is stable. Varying the discriminator's architecture and capacity gives a Spearman correlation of at least 0.91, so the relative ordering is robust even though absolute values depend on the discriminator.

Methodology in Plain English

The authors treat forecasting as conditional generation: given a history, a model should be able to produce many plausible continuations, not one. To judge quality they build a pair of scores. For accuracy, TMS compares each predicted event to the matching ground-truth event and rescales the error by how well a naive baseline (the historical median for features, evenly spaced timestamps from the last event for time) would have done, which makes features measured in different units comparable. It then searches over permutations between predicted and true horizon positions, solving the assignment with the Hungarian algorithm, so that "wrong content" and "right content in the wrong order" are penalized differently.

For realism, cDFS trains a discriminator to tell a real continuation apart from a generated one, given the same history, and reports 2·(1 − AUROC), clipped at 0.5 for AUROC, so the score lies in [0, 1] and reaches 1 only when the discriminator cannot beat chance. Because the discriminator sees the history, it can flag continuations that look plausible in isolation but contradict the preceding events.

The Wasserstein BaryBooster is an inference-time wrapper: draw N samples from any base generator (N=30 in the main experiments), then return the continuation that minimizes the total TMS transport cost to those samples. This is the discrete TMS analogue of a Wasserstein barycenter, and it can be swapped for any other metric in the same minimization.

Why This Matters

Impact on research. The paper argues that generative evaluation of event sequences has been one-sided: accuracy metrics were the only ones reported, and they can reward forecasts that match no real trajectory at all. The roulette example makes this concrete — averaging the amounts of a winning and a losing evening produces an unrealistic pairing like an $18 bus ride that scores well on MSE and per-event accuracy while matching neither outcome. By making realism a first-class citizen and showing the trade-off quantitatively, the paper sets a new baseline for how these models should be judged. The finding that a one-line non-parametric method sits on the Pareto front also raises the bar for claims of progress in this area.

Real-world applications (as discussed or implied by the paper):

• Customer transaction forecasting — predicting the next purchases on a card, the primary setting of all three benchmark datasets.
• Clinical record modeling — anticipating upcoming entries in a patient's record, a motivating example the authors cite.
• Online gaming — forecasting the next actions of a player, another cited application domain.
• Business process intelligence — the general class of chained event processes the paper positions its work within.

Industry relevance. The BaryBooster result matters practically because it converts an existing generative model into a stronger predictor with no retraining — a post-hoc change to inference rather than a new architecture. The authors also note the overhead is a tunable choice: N=5 samples retain roughly 75 percent of the TMS gain at one sixth of the sampling cost, and sampling parallelizes across GPUs. Conversely, they flag that realistic synthetic transactions could be misused to evade fraud detection, though all models here are trained on public, anonymized data.

Future Directions

• Foundation models for event sequences. The authors propose a single generative model trained once, from which many downstream tasks can be solved, as the natural extension of their finding that realism can be converted into accuracy.

• Reducing the BaryBooster's cost. The procedure currently requires generating multiple candidate trajectories and solving an iterative barycentric aggregation at inference time, which the authors identify as potentially prohibitive in latency-sensitive or resource-constrained deployments.

• Broadening the data landscape. The benchmark is limited to a small number of card transaction datasets because no non-financial public corpus met the size, length, and structure criteria simultaneously. The authors frame this as a limitation of public data availability in privacy-sensitive industries rather than of the benchmark design, and list rejected corpora in the appendix.

• Closing the accuracy–realism gap. Since no method excels on both TMS and cDFS, and Repeat sits on the Pareto front, the authors point to room for specialized generative models that surpass Repeat's accuracy without sacrificing realism.

Target Audience

Researchers and practitioners in generative modeling, temporal point processes, and sequence prediction who need to evaluate whether their forecasts are realistic as well as accurate. It is also relevant to applied scientists in finance, healthcare, and e-commerce who work with heterogeneous timestamped event data, and to anyone designing benchmarks where distributional fidelity matters alongside point accuracy. The paper is demanding: readers should be comfortable with diffusion models, optimal transport, and discriminator-based distributional tests.

Authors’ abstract

Event sequences are chains of random events, each described by heterogeneous (numerical and categorical) features. The problem of event sequence forecasting occurs in finance, healthcare, business process intelligence, and other fields. In this work, we study multi-horizon forecasting of event sequences as a conditional generative modeling problem. We evaluate forecasts not only for prediction accuracy, but also for distributional realism. To this end, we present a benchmark comprising metrics for both quality aspects, strong statistical baselines, recent models adapted from adjacent fields, and curated real-world datasets. Careful evaluation reveals that no method excels at both accuracy and realism, highlighting a fundamental trade-off between the two. Finally, we introduce Wasserstein BaryBooster, an algorithm that aggregates samples from a realism-strong diffusion model into a state-of-the-art predictor, demonstrating that distributional realism can be converted into prediction accuracy. Code is available at https://github.com/xaosina/event-sequence-forecasting.