EventGPT: Next-Event Pretraining on Banking Customer Histories

AI summary

Overview

Research area: Machine learning for financial event sequences — self-supervised pretraining of a transformer on banking customer histories, evaluated on the public Multimodal Banking Dataset (MBD-mini).

Technical level: Intermediate. The paper assumes familiarity with transformers and causal language modeling, but its framing (predict the next event, reuse the representation) is explained in plain terms.

Scope: A single-author paper describing the architecture, training recipe, and evaluation of EventGPT, an open-weights hierarchical causal transformer trained and tested exclusively on publicly available banking data.

What This Paper Is About

Banks usually model customer outcomes by hand-building a table of counts, sums, and recencies per customer, then training a supervised model that needs labels for every new question. This paper asks whether a model pretrained only to predict a customer's next event can (a) estimate the probability of future events with no labels at all, and (b) supply one reusable per-customer representation for supervised tasks. The authors build EventGPT, train it on 100,224 clients from the public MBD-mini dataset, and compare it against hand-engineered features, schema-free token counts, a flat GPT-style decoder, and a Markov-chain baseline.

Key Contributions

1. An open reference model for banking event sequences. To the authors' knowledge, this is the first open-weights generative model for banking event sequences trained and evaluated on a public benchmark with reproducible baselines; the tokenization, model, training and evaluation code, and checkpoints are released.

2. A hierarchical architecture for event streams. An event encoder pools each event's tokens into one state, a history decoder attends causally over event states using temporal rotary embeddings (RoPE derived from elapsed time), and a token decoder generates the next event. Over a window of n tokens, token layers attend within events at cost O(n) and history layers attend over roughly n/8 events at O((n/8)²), against O(n²) for attention over all tokens.

3. Zero-shot scoring of event targets. The token decoder's next-event distribution scores five "first occurrence next month" targets at ROC-AUC 0.79 to 0.92 with no labels, no classification head, and no training on the targets.

4. A careful evaluation of frozen and fine-tuned embeddings. The paper benchmarks the embedding against 114 engineered features and a schema-free 605-feature token-count table, with paired standard errors clustered by client, and shows where the embedding wins, ties, and loses.

5. Point-in-time embeddings. An embedding at an as-of date uses only events before that date, so it can be stored and reused across tasks without leaking the future (the paper notes this guarantee does not extend to the pretraining population; an out-of-time run is referenced).

Main Findings

• Zero-shot event scoring works without labels. From its next-event distribution alone, EventGPT scores five transaction kinds at ROC-AUC 0.79 to 0.92 on both seeds; on aligned windows, seed 42 gives 0.916 (first 48/31), 0.918 (first 23/49), 0.829 (first 26/28), 0.791 (first 48/10) and 0.906 (first 50/19).

• Labels needed to match it are in the tens to hundreds of positives. CatBoost on 114 engineered features needs 10,000 to 30,000 randomly drawn labeled client-months to pass the zero-shot score on 50/19, 30,000 to 300,000 on three others, and is still below it at 300,000 on 23/49 (passing only with every positive of the pretraining folds, 0.930). With positive-focused draws, the features pass at 50 positives on 48/31, 100 on 26/28, 48/10 and 50/19, and 400 on 23/49.

• It beats a label-free Markov baseline on some targets, not all. A Markov chain over event kinds scaled by recent activity scores 0.79 to 0.88. EventGPT is higher on 23/49 (+0.057) and 50/19 (+0.095), and within two standard errors on the other three. Across all 74 transaction kinds with at least 50 test positives, EventGPT is higher by more than two paired standard errors on 48 kinds, lower on 7, and level on 19 (median difference 0.042).

• Frozen embeddings lose to engineered features on product prediction. Mean ROC-AUC is 0.787 for the frozen EventGPT embedding against 0.808 for the engineered features over five test folds. Adding profile fields and full-history token counts narrows the gap to 0.004, which is not statistically significant on any fold.

• Ensembling is where the embedding pays off. Rank-averaging EventGPT with the engineered features adds 0.014 to 0.017 ROC-AUC on each of five test folds. The same ensemble built without EventGPT adds only 0.001 to 0.008, so 0.007 to 0.016 of the gain is EventGPT's own (p < 0.01 on each of ten fold and seed pairs). Used as a feature inside the table, the embedding leaves ROC-AUC unchanged or lower.

• Late fusion of static attributes helps product targets. Adding the profile fields as a late-fusion path adds 0.013 to 0.019 ROC-AUC to the product targets and brings target_4 level with the engineered features.

• Sampling whole months beats the one-step score. Rolling out 16 sampled continuations per row scores 0.010 to 0.036 above the one-step score on all five event targets. The sampled months expect 15.5 transactions per row against 15.5 observed, with a Spearman correlation of 0.81 across rows. As literal forecasts of next month they are weaker: matched by kind, a sampled month and the real one share 0.459 of their transactions, against 0.545 for repeating the previous month.

• The zero-shot advantage is not unique to the hierarchical design. A flat GPT-style decoder (23.5M parameters vs EventGPT's 22.9M), scored the same way, also reaches 0.78 to 0.92 — so zero-shot scoring needs only a model trained to predict the next event.

• The earlier-occurrence rule matters. Without setting the score to zero for clients who already had the kind, the score on 50/19 falls from 0.91 to 0.72.

Methodology in Plain English

Data. The authors use MBD-mini, a 10% sample of the Multimodal Banking Dataset (CC BY 4.0) covering 100,224 clients in five folds, with 38.8 million transactions, 66.3 million geolocation pings and 0.22 million support dialogues. Each client's stream starts with a user token and a profile event, then interleaves transactions and geolocation in time order. Time gaps and amounts are bucketed into tokens, giving about 8 tokens per event and 53.4 million events overall.

Model. Three stacked stages, trained jointly by next-token cross-entropy. The event encoder reads one event's tokens bidirectionally and mean-pools them into a single vector. The history decoder runs six causal layers over those vectors, placing each event not at position 1, 2, 3, but at a rotary position proportional to the logarithm of elapsed time, so two events a year apart stay far apart. The token decoder then predicts the next event token by token, conditioned on the previous event's hidden state. The last hidden state at the as-of date is the customer embedding. Because every history ends with a "now" event whose gap runs to the as-of date, two clients with identical histories but different idle periods get different embeddings.

Training. Width 384, 12 heads, two event-encoder layers, six history layers, two token-decoder layers: 22.9 million parameters. Fold 0 is never seen in pretraining or vocabulary fitting. A run of 655 million tokens takes 1.5 hours on one NVIDIA L4 in bfloat16 (about 118,000 tokens per second), against 3.9 hours for the flat decoder. The final configuration adds query-key normalization; without it a seed-43 run diverged at step 4,400.

Evaluation. A row is a client and a month from February 2022 to January 2023. The test fold holds 243,936 rows, of which 242,604 are scored after dropping 111 clients with no stream. Baselines: CatBoost on 114 hand-built features, CatBoost on a schema-free 605-feature token-count table, a flat decoder, and a Markov chain over event kinds. Standard errors use DeLong's method with variance clustered by client, which also gives paired comparisons between models ranking the same clients.

Why This Matters

Impact on research. The paper provides a public, reproducible reference point in an area where the closest industrial work (Nubank's nuFormer, Revolut's PRAGMA, Visa's TransactionGPT, Starling Bank's announced model trained on 15 billion transactions) releases no checkpoints and evaluates on private data. The authors argue that even a released checkpoint would transfer poorly, since each institution's event vocabulary is its own. They also report their own negative results plainly — the embedding loses to hand-built features on product prediction, and zero-shot scoring is matched by a flat decoder.

Real-world applications:

• Cold-start propensity scoring: ranking which customers will take a product or make a first transaction in the next month before any labels exist for that target.
• Label-efficient targeting: the paper's framing of "50 to 400 labeled positives" is directly actionable for teams that can only hand-label a small sample.
• Reusable customer representations: one stored point-in-time embedding serving many downstream tasks through small logistic-regression heads, rather than a new feature table per question.
• Second ranker in production ensembles: adding the embedding's ranks to an existing feature-table model contributed 0.007 to 0.016 ROC-AUC of its own.

Industry relevance. The architectural lessons (per-event encoding, time-based rotary positions, two-level attention) are cheap to adopt: training takes about two hours on a single L4 GPU. The paper's honest accounting of where the approach does not beat feature engineering is as useful to practitioners as the wins.

Future Directions

• Comparing against published results on full MBD. The paper reports the MBD paper's numbers (aggregation 0.783, CoLES 0.773, TabBERT 0.762, TabGPT 0.802, supervised RNN 0.819) only as context, noting that MBD-mini is a different sample and that the comparison on full MBD is planned.
• Quantifying pretraining leakage. Pretraining uses events from 2021 and 2022 for the pretraining clients, including events from the test clients' label months; the effect is to be measured by an out-of-time run, which is referenced but not reported in the available content.
• Reproducing the choice of the five event targets. The authors state they chose the five kinds by hand and did not keep the script, so they cannot establish which data the choice used; the fold-level ranking shows the selected kinds are strongly enriched but does not reproduce the choice, since two unselected kinds rank above two selected ones.
• Better generative forecasting. Sampled months calibrated well on transaction volume (15.5 expected against 15.5 observed, Spearman 0.81) but were weak as literal forecasts of composition (0.459 overlap), leaving room to improve the rollout beyond the one-step distribution.

Target Audience

Machine learning researchers and practitioners working on event-sequence modeling, tabular-versus-representation-learning comparisons, or foundation models for financial services. It is most valuable to data scientists at banks and lenders deciding whether to invest in pretraining a sequence backbone rather than building feature tables, and to academic researchers who need a reproducible, open-weights baseline on public banking data. Readers should be comfortable with transformer architecture; the evaluation design (clustered standard errors, paired comparisons) will also interest applied statisticians.

Authors’ abstract

A model pretrained to predict a customer's next event can estimate the probability of future events without training on their labels and provide a reusable representation for supervised downstream banking tasks. We pretrain EventGPT, a hierarchical causal transformer, by next-token maximum likelihood estimation on the public Multimodal Banking Dataset (MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings). Keeping one state per event, EventGPT trains in about two hours on one L4 GPU at windows up to 8,192 tokens. From its next-event distribution, with no labels, EventGPT scores five targets of the form "the client makes a transaction of a given kind for the first time next month" at ROC-AUC 0.79 to 0.92. Gradient boosting on 114 engineered features needs 10,000 to 300,000 labeled client-months to match it on four targets and stays below it on the fifth. For product prediction, frozen EventGPT embeddings score below the engineered features (mean ROC-AUC 0.787 against 0.808 over five test folds), and a rank-averaged ensemble of EventGPT, profile fields and token counts with the engineered features adds 0.014 to 0.017, of which 0.007 to 0.016 is EventGPT's own. The code and pretrained weights are released.