LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

AI summary

Overview

Research area: Natural language processing / financial AI evaluation — specifically temporal knowledge boundaries and hindsight leakage in LLM backtesting.

Technical level: Intermediate. The paper is readable without deep ML background, but assumes familiarity with multiple-choice benchmarking, accuracy baselines, and the idea of a training-data cutoff.

Scope (one sentence): The paper builds a benchmark of 2,226 dated U.S. financial facts and uses it to test whether the three common safeguards against hindsight leakage in LLM backtests — vendor-stated cutoffs, model self-reports, and "answer as of a past date" instructions — actually work.

What This Paper Is About

Financial backtests of LLMs only mean something if the model did not already know the outcomes it is being asked to predict. Practitioners assume one of three safeguards protects them: trusting the vendor's stated knowledge cutoff, asking the model what it knows, or telling it to answer as if it were a past date. The paper tests all three against measured model behavior on the same set of dated facts, and finds that none of them reliably separates known outcomes from genuine forecasts.

Key Contributions

1. A dated financial benchmark. 2,226 dated U.S. facts spanning 1999 to 2026 — drawn from a pool of 20,972 — covering twelve FRED macroeconomic series, month-end closing prices for 50 large U.S. companies, and Federal Reserve target-rate decisions. Every fact is scored against the value published at the time (first-release macro data, split-adjusted-away stock prices), not later revisions.

2. A fact-level pairing of answers and knowledge claims. Each fact yields four prompts that share the same candidate values: a plain value question (Task 1), a value-plus-claim question in Recall and Predict wordings with no declared date (Task 2), and a Predict question prefixed with "Assume that today is [date]. Answer as you would on that date." (Task 3). Because the answer and the claim come from a single choice, accuracy and self-reported knowledge can be compared fact by fact.

3. Behavioral estimation of two boundaries. A consistent procedure for estimating the actual cutoff (last quarter whose two-quarter rolling accuracy stays above the midpoint between the model's 2015–2020 level and chance) and the model-identified cutoff (last quarter whose claim rate stays above half its 2015–2020 level), plus comparison against self-reported and vendor-stated cutoffs.

4. Direct tests of the three safeguards, including a simulated self-report-filtered backtest and a simulated dated backtest, showing how much historical knowledge survives each filter.

Main Findings

• Models know more than they say. GPT-5 answers 76% of the 1,340 controlled facts correctly (chance ≈ 0.27) but claims to know only 38%. GPT-4o scores 50% accuracy against a 20% claim rate; DeepSeek-V3.2 scores 54% against 28%. The direction reverses for Llama-3.3-70B (0.48 claims vs. 0.43 accuracy) and especially for the two small Qwen models, which claim to know roughly 70% of facts despite accuracies of 0.36–0.37.

• "I don't know" does not mean "I can't answer." GPT-5 answers 61% of the facts it disclaims correctly across the full sample, more than twice chance. Within its measured knowledge window, that figure is 0.75 (on 1,109 facts, 46% denied), falling to 0.33 after the window (231 facts, 100% denied). DeepSeek-V3.2 shows the same pattern (0.53 inside vs. 0.34 after). GPT-4o (0.37 vs. 0.32) and Llama-3.3-70B (0.31 vs. 0.27) are closer to chance on disclaimed facts, but the Llama filter retains only 22% of in-window facts.

• Claims do carry some signal. Within-period gaps between accuracy on claimed-known and disclaimed facts range from 0.08 (DeepSeek-V3.2) to 0.39 (GPT-4o), and all 95% bootstrap intervals exclude zero. The problem is the level, not the direction.

• Knowledge claims outlast knowledge. Llama-3.3-70B's accuracy falls to chance after 2021Q2, but it keeps claiming knowledge until 2023Q1 — a seven-quarter gap — and its vendor-stated cutoff of December 2023 sits ten quarters past its measured knowledge boundary. GPT-4o and DeepSeek-V3.2 show one-quarter gaps. GPT-5's boundaries coincide at 2024Q3, matching its stated cutoff. The two Qwen models stay near chance throughout but claim knowledge until 2024. The abstract reports claim persistence of up to seven quarters beyond the measured knowledge boundary.

• A declared date changes the words, not the knowledge. Moving GPT-5 from Task 2 Predict to Task 3 drops claims of prior knowledge from 0.39 to 0.01 while accuracy moves only from 0.76 to 0.75. Llama-3.3-70B goes 0.63 → 0.00 and GPT-4o 0.33 → 0.04, again with essentially unchanged accuracy. DeepSeek-V3.2 moves the other way (0.37 → 0.46) at unchanged accuracy. In the simulated dated backtest, GPT-5 labels 99% of its in-window answers as predictions and reaches 0.82 accuracy on those answers (0.39 after its boundary); Llama-3.3-70B labels 100% as predictions at 0.51 (0.29 after).

• The leakage is concentrated in precise numbers. Stock prices produce the largest under-claiming gaps: GPT-5 +0.54 (59% accuracy vs. 5% claimed), DeepSeek-V3.2 +0.47 (49% vs. 2%), GPT-4o +0.33 (33% vs. 0%). Llama-3.3-70B reverses on macro facts (−0.21; 53% claimed vs. 32% correct). Federal Reserve decisions show the tightest agreement (−0.16 to +0.04).

• Individual examples make the failure concrete. GPT-5 says it does not know Intel's January 2021 close and selects 55.51; it says it does not know Chevron's January 2006 close and selects 59.38; it says it does not know the February 2024 USD/JPY average and selects 149.62 — in both the plain Recall wording and the dated Predict wording.

• Controls and sensitivity. A persistence strategy answers 47% of numerical facts correctly without recall, so numerical knowledge is measured only on the remaining 1,007 facts. "Unchanged" is correct in 78% of policy months, so balanced accuracy is used, putting that strategy at one third. Combined chance reference is ≈ 0.27. Tightening the threshold from one-half to one-third moves GPT-5's and DeepSeek-V3.2's boundaries by at most one quarter and leaves Llama-3.3-70B with a six-to-seven-quarter gap, but widens GPT-4o's gap from one to six quarters; more permissive thresholds give unstable estimates.

• Point-in-time models are not yet a control. TiMaGPT, ChronoGPT-Instruct, and DatedGPT (1–1.5B parameters) often pick the first option in multiple-choice format, so they were scored by candidate likelihood across five cutoffs from 2014 to 2022 plus DatedGPT-2014. Accuracy stays near chance (0.25–0.30) on both sides of each cutoff, with all differences ≤ 0.041 and every 95% interval including zero.

Methodology in Plain English

The authors fix one economy and one calendar so that any change over time reflects the model rather than a change in topic. They collect dated U.S. facts, of which 2,226 are sampled — sparse before 2021 (two macro facts and one stock per month, plus every FOMC month) and dense from 2021 onward (every series and four stocks per month), because current model cutoffs fall between 2021 and 2025.

Each fact becomes a multiple-choice question with four candidate values spaced roughly one typical monthly change apart (three outcomes for Fed decisions). The same candidates are reused across four prompts, so the model's answer and its claim about that answer come from one choice: a plain accuracy question; a version where each candidate appears twice, once tagged "I know this value" and once tagged "I do not know this value" (eight options, six for policy); a Predict version using "I already know the outcome" versus "I do not know the outcome"; and a dated version of the Predict prompt with the date set to the last day of the month before the fact.

To keep a model from scoring without real recall, they discard numerical facts where simply repeating last month's value would land on the right candidate, and they use balanced accuracy for policy decisions so that always answering "unchanged" earns one third rather than 78%. Accuracy over time then traces the actual knowledge boundary; claim rates over time trace the model-identified boundary. Comparing the two, and comparing both to published cutoffs, shows which safeguard holds.

They evaluate GPT-5, DeepSeek-V3.2, GPT-4o, and Llama-3.3-70B-Instruct via API at temperature 0 (up to 1,024 output tokens, 4,096 for GPT-5), plus Qwen3-8B and Qwen3.5-9B locally with greedy decoding and thinking disabled, plus the three point-in-time families. That is 8,904 prompts per model.

Why This Matters

Impact on research. Published LLM forecasting and trading results are routinely defended with a vendor cutoff or a prompt instruction. This paper shows that both can be wrong in the same direction — the vendor cutoff for Llama-3.3-70B sits ten quarters past where the model's accuracy actually decays to chance, and a date instruction converts 99% of GPT-5's in-window answers into "predictions" without removing a single correct answer. Any backtest whose evaluation window falls inside a model's training data cannot be certified leak-free by self-report, instruction, or stated cutoff; it needs a behavioral audit on the model itself.

Real-world applications:

• Model selection and due diligence for asset managers: replace the vendor cutoff with a measured boundary for the specific domain (equities vs. macro vs. policy) before deciding which historical period is credible.
• Vendor and model-developer documentation: publish behavioral knowledge assessments on dated, domain-specific facts alongside the stated training cutoff.
• Compliance and audit of AI-driven investment research: a measurable procedure for documenting whether a reported result could have been contaminated by recalled outcomes.
• Academic replication: re-check the reported edge of previously published LLM trading or forecasting systems against measured knowledge boundaries.

Industry relevance. The cost is low — 8,904 single-letter multiple-choice prompts per model — and the finding that under-claiming is concentrated in precise numerical facts (stock prices) is exactly the fact family most financial backtests depend on. The paper notes that vendor cutoffs tend to be conservative relative to measured boundaries in their experiments, but DeepSeek-V3.2 publishes no cutoff at all, leaving behavioral measurement as the only option.

Future Directions

• Extend past one economy. All 2,226 facts come from U.S. series; whether the claim–knowledge gap holds for other markets and currencies is untested here.
• Build point-in-time models big enough to serve as controls. Current 1–1.5B point-in-time models sit at chance on both sides of their cutoffs, so they cannot yet validate behavioral boundary estimation. The conclusion states this directly: the benchmark supplies the procedure, but current point-in-time models are still too small to be the alternative.
• Improve calibration of self-reports, especially for precise numerical facts, where under-claiming is largest — and understand why Llama-3.3-70B over-claims on macro while under-claiming elsewhere.
• Standardize the audit. Reporting measured boundaries alongside backtest results and reassessing them at every model update, with explicit margins for boundary uncertainty given the observed sensitivity to threshold choice.

Target Audience

Researchers and practitioners who evaluate LLMs on financial forecasting or trading tasks; quantitative researchers and model-risk staff at asset managers and banks who need defensible backtest windows; benchmark designers working on temporal knowledge and model self-knowledge; and model developers who report training cutoffs. Readers with a basic grasp of accuracy-versus-chance comparisons and multiple-choice evaluation will follow the paper comfortably; no specialized econometrics is required.

Note on the text: the benchmark's name appears in the supplied content as a run of LaTeX color-code arguments interleaved with characters (rendering the letters F-i-n-D-e-n-d-r-o), so it is referred to here only by description. The paper's Limitations and Ethics section is truncated in the supplied content and is therefore not summarized.

Authors’ abstract

Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic…