Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings

AI summary

Overview

Research area: Natural Language Processing applied to financial transaction modeling — specifically, bridging LLM semantic understanding with tabular sequential foundation models used in payment networks.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, embedding layers, LLM hidden-state extraction, and multi-task learning. The conceptual motivation is accessible, but the implementation details and metrics presume a machine learning engineering background.

Scope: This paper presents and empirically validates a hybrid framework that replaces randomly initialized categorical embeddings in a production transaction foundation model with precomputed LLM-generated sentence embeddings, achieving semantic enrichment without incurring runtime LLM inference costs.

What This Paper Is About

Transaction foundation models process payment records — amounts, timestamps, merchant names, category codes, and locations — to predict things like a user's next purchase or to flag anomalous activity. These models typically represent categorical fields such as merchant names by mapping them to arbitrary integer indices and learning embeddings from scratch, which discards rich semantic information: the index for "Costco" carries no signal that it is a wholesale retailer with a membership model. The paper's goal is to inject that missing semantic knowledge using LLM-generated embeddings, while preserving the low latency and scalability required for real-time financial deployment.

Key Contributions

1. A practical hybrid framework that integrates LLM-based sentence embeddings into a sequential tabular foundation model, replacing index-based categorical representations for merchant name, MCC, and location fields while leaving the rest of the architecture and training procedure untouched.

2. A preprocessing and prompt generation pipeline built on multi-source data fusion and an "Explicit One-word Limitation" prompt design principle, which constrains LLM outputs to produce consistent, focused, noise-resistant semantic representations across different LLM architectures.

3. An offline precomputation strategy that decouples semantic enrichment from inference — embeddings are generated once and used to initialize embedding layers, so no LLM inference occurs at serving time, preserving production latency and cost profiles.

4. Empirical validation on one billion real-world transactions across four LLM architectures (Llama2-7b, Llama2-13b, Llama3-8b, Mistral-7b), showing consistent improvements across multiple transaction understanding tasks, including a proprietary production-relevant metric.

Main Findings

• Merchant and MCC prediction benefit most consistently. LLM-based initialization improved MCC prediction in 82% of experimental configurations and merchant prediction in 100%, indicating that semantic embeddings effectively capture relationships between merchant categories and specific merchants — the task where index-based representations were weakest.

• Amount prediction improves with most strategies, except MCC-only. Geographic and holistic (all-fields) representations helped forecast next transaction amounts; MCC information in isolation did not, suggesting spending patterns are driven more by location and overall merchant semantics than by category codes alone.

• Location prediction gains are modest. Because location is relatively stable within a transaction sequence, the vanilla model already performs well, and only all-fields initialization produced clear improvements.

• Llama3-8b is the most versatile embedding source. Across initialization strategies and tasks, it consistently achieved top-tier performance, suggesting better architectural suitability for transaction understanding.

• Production transaction metrics improved. On the confidential transaction metrics assessment, relative improvement over the deployed baseline system reached +3.93% (Mistral-7b, all fields) and was positive for nearly every configuration tested.

• Combining fields generally outperforms single-field initialization. Joint MCC + merchant and all-fields initialization tended to outperform isolated fields, though some architectures showed reduced performance on specific metrics, hinting at optimization challenges when integrating multiple embedding types simultaneously.

• Enriched prompts outperform naive ones. Early experiments with simple prompts (e.g., "provide the embedding of MCC 5044") produced poor results, motivating the multi-source fusion approach that supplies category descriptions, related merchant examples, and geographic context.

Methodology in Plain English

The researchers start with raw transaction data and identify three categorical fields worth enriching: merchant category codes (MCC), merchant names, and locations.

Step 1 — Enrich the data. Each field is augmented from external sources. An MCC gets its official title, business description, and examples of similar categories. A merchant gets its location and its MCC description attached. A location gets economic and demographic context. This addresses the sparsity of raw transactional records.

Step 2 — Build structured prompts. The enriched information is turned into natural language prompts designed for LLM consumption. A key design choice — the "one-word limitation" — constrains outputs so that different LLMs produce comparable, focused representations rather than verbose and inconsistent responses that would add noise.

Step 3 — Extract embeddings. Each prompt is fed to an open-source LLM, and the representation of the last non-padding token from the final hidden layer is taken as the sentence embedding. This is done offline for all categorical values.

Step 4 — Initialize and fine-tune. These embeddings replace the randomly initialized embedding layers for merchant name, MCC, and location in the existing transaction foundation model. The rest of the architecture, multi-task loss, and training procedure remain unchanged; the model fine-tunes from this semantically informed starting point.

Step 5 — Evaluate. The team trains on 20 months of data, validates on the 21st month, and tests on the final three months, comparing against the vanilla index-based baseline across next-amount, next-MCC, next-city, next-merchant, and a proprietary transaction metrics task.

Why This Matters

Impact on research: The paper demonstrates a middle path between two extremes in structured-data modeling — pure ID-based embeddings (efficient but semantically blind) and end-to-end LLM inference (semantically rich but computationally prohibitive). It also shows that offline semantic initialization is a viable transfer mechanism, and that prompt design principles like output-length constraints can standardize embeddings across heterogeneous LLM backbones.

Real-world applications:

• Fraud and anomaly detection: Semantically informed merchant representations can help distinguish legitimate from suspicious transaction patterns, particularly for merchants the model has seen few examples of.
• Cold-start merchant handling: New merchants with no transaction history can inherit meaningful representations from their name, category, and location, rather than starting from a random initialization.
• Personalized financial services: Better next-transaction prediction underpins recommendations, budgeting tools, and spend forecasting for cardholders.
• Risk and credit modeling: Enriched categorical representations can feed downstream underwriting and portfolio risk systems that rely on merchant-level signals.

Industry relevance: The work originates from Visa Research and targets payment networks directly, where latency, throughput, and cost constraints are non-negotiable. Because embeddings are precomputed offline, the approach slots into existing serving infrastructure without requiring GPU-backed LLM inference per request — a prerequisite for any production deployment at transaction volume.

Future Directions

• More sophisticated prompt engineering. The one-word constraint is effective but simple; domain-specific LLM fine-tuning or automated prompt optimization for financial contexts could yield stronger representations.

• Broader model coverage. The study evaluated established architectures but not newer or purpose-built embedding models such as NV-Embed or Qwen3-embedding, which may perform better on this task.

• Extension to additional categorical fields. Transaction channels, payment methods, and temporal patterns are candidates for similar semantic enrichment, but their generalizability is untested.

• Dynamic, time-aware embeddings. Current embeddings are static and generated offline, so they cannot reflect seasonal business shifts, evolving merchant characteristics, or changing market conditions. A mechanism for periodically refreshing or conditioning embeddings on time could capture these dynamics.

Target Audience

This paper is most valuable to machine learning engineers and applied researchers working on tabular foundation models, sequential recommendation, or production financial ML systems — particularly those who need semantic enrichment but cannot afford LLM inference at serving time. It will also interest NLP researchers studying how LLM knowledge can be distilled into smaller task-specific models through embedding initialization rather than end-to-end distillation. Readers without grounding in transformer architectures and embedding layers will find the experimental sections difficult, though the problem framing and framework overview are broadly accessible.

Authors’ abstract

The ubiquity of payment networks generates vast transactional data encoding rich consumer and merchant behavioral patterns. Recent foundation models for transaction analysis process tabular data sequentially but rely on index-based representations for categorical merchant fields, causing substantial semantic information loss by converting rich textual data into discrete tokens. While Large Language Models (LLMs) can address this limitation through superior semantic understanding, their computational overhead challenges real-time financial deployment. We introduce a hybrid framework that uses LLM-generated embeddings as semantic initializations for lightweight transaction models, balancing interpretability with operational efficiency. Our approach employs multi-source data fusion to enrich merchant categorical fields and a one-word constraint principle for consistent embedding generation across LLM architectures. We systematically address data quality through noise filtering and context-aware enrichment. Experiments on large-scale transaction datasets demonstrate significant performance improvements across multiple transaction understanding tasks.