Can LLMs Access their World Knowledge for Event Prediction?

AI summary

Overview

Research area: Natural language processing, specifically large language model (LLM) reasoning, temporal event prediction, and benchmark design.

Technical level: Intermediate. Readers will benefit from familiarity with LLM prompting, temporal point processes, and standard classification metrics, though the paper's central idea (descriptions plus symbolic labels) is explained accessibly.

Scope: The paper argues that LLMs already encode useful world knowledge for predicting what happens next in an event sequence, and that the reason they often underperform specialized models is a flaw in how the prediction task is presented to them, not a flaw in the knowledge itself.

What This Paper Is About

Temporal event prediction asks what event will happen next, and when, given a history of timestamped events such as commits in a code repository, earthquakes, or diplomatic actions. Prior work typically represents each event as only a type name and a timestamp, and recent studies suggest LLMs add little over simpler specialized models on this task. This paper tests the opposite explanation: that LLM world knowledge is accessible, but current interfaces strip away the semantics and add label-name biases that prevent models from using it.

Key Contributions

1. A reframing of the problem as an interface question. The authors show that LLM knowledge becomes usable for event prediction when the prediction interface is designed to expose it, rather than treating the model's underperformance as evidence that the knowledge is absent.

2. Two concrete interface choices. Grounding every event in a natural-language description, and replacing semantic event-type names with non-semantic symbolic identifiers (such as type_0) to remove what the authors call "semantic bias."

3. The DescribeEvent benchmark. A benchmark of approximately 150K events (148,185 events in 1,940 sequences) across seven datasets and six domains, in which every single event is paired with its own natural-language description — unlike prior datasets that attach at most one description per sequence.

4. An empirical link between reasoning ability and description benefit. The paper shows that models with strong reasoning performance benefit from descriptions, while weaker models gain little or even degrade when descriptions are added.

Main Findings

• Descriptions help even with semantic labels. Adding natural-language descriptions to semantic event types increases F1 and lowers MAE for nearly every evaluated LLM, indicating the descriptions supply information beyond types and timestamps.

• Descriptions plus symbolic labels produce the strongest results. This combination achieves the best per-dataset F1 and MAE on most datasets and the highest average F1 with the lowest average MAE across the seven datasets, supporting the idea that removing label-name signal pushes the model to rely on descriptions.

• Symbolic labels alone are not enough. Without descriptions, symbolic labels perform similarly to semantic labels for most models, because removing semantics also removes the cues the model could otherwise use.

• Semantic bias is real and measurable. In a label-permutation test where descriptions are omitted, symbolic-label accuracy is more stable across random relabelings than semantic-label accuracy, indicating label names themselves carry predictive signal.

• Description benefit tracks reasoning capability. Models that benefit from descriptions score at least 77% on GPQA Diamond and at least 7% on Humanity's Last Exam, while GPT-4o Mini and Llama-3.1-8B perform significantly worse on both and see overall F1 drop when descriptions are added.

• Symbolic labels mainly improve precision on rare events. Across the 46 event types analyzed (Amazon excluded due to its large label space), the precision gain from symbolic labels increases as event frequency decreases, with a Pearson correlation of r = −0.51 (p ≈ 0.0003). Recall shows almost no relationship with frequency (r = −0.01).

• The strongest LLM setting beats non-LLM baselines on most datasets. Among non-LLM methods, KNN is the strongest baseline for event-type prediction and description-aware NHP is strongest for time prediction. Adding Sentence-BERT description embeddings to non-LLM baselines often improves MAE but does not consistently improve F1.

• Descriptions carry content, not just paraphrased labels. A validator LLM recovers the event type from description alone well above random chance on every dataset (for example, 0.992 versus 0.111 random on NBA), while the cosine-similarity gap between a description and its true label name versus other label names stays small (for example, 0.073 on GitHub Repo), showing the descriptions are not simply restating labels.

Methodology in Plain English

The authors start from the standard event-prediction format: each event is a type plus a timestamp. They modify this in two ways. First, they attach a natural-language description to every event, taken from the raw source text recorded at the time the event happened, so no future information leaks backward. For a GitHub event this might be a commit message; for an earthquake, magnitude, depth, and location.

Second, they address a specific failure mode. Because event type names are ordinary words, an LLM may have learned associations between them from pretraining — for example expecting "merge" to follow "push" in code text — and predict from those associations rather than from the actual event history. The authors call this semantic bias. Their fix is to rename every event type to an anonymous token like type_0, keeping semantics only in the descriptions.

To test this, they build DescribeEvent by collecting public event data from seven sources, applying three universal curation filters to each dataset (a cap on how many events share one type, a cap on consecutive repeats of the same type, and a minimum number of distinct types per sequence), plus domain-specific filters such as a magnitude threshold for earthquakes and bot-account exclusion for GitHub and Wikipedia.

They then evaluate six LLMs — Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, GPT-5 Mini, Grok-4.1 Fast, and Gemini-2.5 Flash — under four settings formed by combining semantic or symbolic labels with descriptions present or absent, and compare against six non-LLM baselines: ARIMA, KNN, Hawkes, NHP, RMTPP, and MLP. Prediction quality is measured with macro-averaged F1 for event type and mean absolute error on the inter-event gap for timing.

Why This Matters

Impact on research. The paper redirects a live debate. Rather than asking whether LLMs are useful for temporal event prediction, it shows the answer depends heavily on interface design, and it supplies a benchmark with per-event descriptions that prior datasets lacked. It also extends the label-symbolization idea from general NLP classification into event prediction and connects it to a measurable precision effect on rare events.

Real-world applications:

• Software engineering: anticipating the next action in a repository or user session (issue opened, PR merged, push) from commit messages and activity descriptions.
• Disaster monitoring: forecasting the next seismic event by tectonic zone, where descriptions carry magnitude, depth, and location.
• Geopolitics and risk: predicting the next diplomatic action from GDELT-style event descriptions, relevant to analysts tracking escalation or cooperation.
• Commerce and content platforms: predicting the next purchase category, edit type, or play type, where descriptions exist as review text, edit summaries, or play-by-play annotations.

Industry relevance. The authors are affiliated with RBC Borealis and the University of Alberta. Off-the-shelf LLMs being competitive with, or better than, specialized models on most datasets suggests a practical path for organizations that already have LLMs deployed and event logs with associated text, without training a bespoke model. The paper is explicit about the trade-off: LLM inference may take seconds per prediction versus tens of milliseconds for specialized event-prediction models, which the authors frame as the cost of better performance and natural-language reasoning traces.

Future Directions

• Fine-tuning on DescribeEvent. The authors note the benchmark is currently designed for evaluation only and that using it as training data via parameter-efficient adaptation is promising future work.

• Establishing causal mechanism. The correlation between reasoning capability and description utilization is observed, but the authors state that a causal mechanism requires further investigation.

• Reducing inference latency. Closing the gap between seconds-per-prediction for LLMs and tens of milliseconds for specialized models is an open practical question.

• Better numeric time representations. Inter-event gaps are currently written as raw floating-point numbers with standard tokenizers, which the authors flag as worth improving.

Target Audience

Researchers and practitioners working on temporal event prediction, temporal point processes, or LLM benchmarking will get the most from this paper. It is also relevant to applied teams considering whether to replace or augment specialized forecasting models with general-purpose LLMs, and to researchers interested in how prompt and label design choices shape apparent LLM capability.

Authors’ abstract

Recent work has questioned whether large language models (LLMs) can be useful for temporal event prediction. Despite vast pretrained world knowledge, LLMs often underperform simpler specialized models. We ask whether this knowledge is truly inaccessible, or whether current methods fail to present event prediction in a form LLMs can use. We argue for the latter and show that LLMs become effective event forecasters when the interface is designed around two choices. The first grounds each event in a natural-language description, and the second separates event semantics from the prediction space to prevent semantic bias from label associations. To test this, we introduce DescribeEvent, a benchmark of ~150K events across seven datasets and six domains with event-level descriptions. We find that capable LLMs substantially benefit from descriptions and symbolic labels, outperforming description-free variants and non-LLM baselines, while weaker models gain little. These results suggest that accessing LLM world knowledge for event prediction is an interface problem, and that event descriptions offer a practical way to elicit the knowledge LLMs already encode.