Are LLMs Good Financial User Simulators? A Preliminary Study

AI summary

Overview

Research area: LLM-based user simulation, benchmark design, financial AI / behavioral finance.

Technical level: Intermediate. The setup and findings are conceptually accessible, though familiarity with classification metrics (recall, F1, MAE) and LLM evaluation protocols helps.

Scope: This paper introduces AInvestor, a pilot benchmark built from 120 paper-trading volunteers, to test whether state-of-the-art LLMs can reproduce an individual investor's next-day trading decisions from point-in-time information.

What This Paper Is About

LLMs are increasingly proposed as "user simulators" — stand-ins for real people used to test financial advice, recommendations, and risk warnings before exposing real clients to them. This paper asks whether those simulators are actually faithful: given only what was knowable before a cutoff date, can a model predict what a specific volunteer did on the next trading day, including which security they traded and how much? The answer, based on a controlled four-month study, is largely no — models produce plausible-looking behavior that is systematically unlike the real thing.

Key Contributions

1. A new pilot benchmark (AInvestor). A dataset aligning de-identified participant profiles, cross-scenario platform interactions, virtual portfolio states, simulated transactions, and time-aligned market observations from 120 volunteers over roughly four months (230K+ interactions).

2. A hierarchical evaluation protocol. Rather than scoring only next-action accuracy, the paper decomposes the task into whether to trade, what to trade, in which direction, and at what size — plus temporal consistency and trajectory-level fidelity.

3. A controlled ablation on market context. The same simulator is evaluated with and without point-in-time market information, isolating how much external context actually helps.

4. Identification of "behavioral compression" as a failure mode. The paper names and characterizes three specific distortions (inertia bias, basket compression, temporal accumulation) that aggregate metrics typically hide.

Main Findings

• All models fall below a trivial baseline. Action accuracy across six LLMs (GPT-5.6-Sol, GPT-5.6-Luna, GPT-5.5, Gemini-3-Flash, DeepSeek-V4, GLM-5.2) ranges from 42.60% to 47.53%, entirely under the always-predict-hold baseline of 48.27%.

• Inertia bias dominates. Real users buy, sell, and hold in roughly balanced proportions. Models predict hold for the vast majority of instances — hold recall reaches 75.99%–97.83%, while sell recall collapses to 0.00%–6.57% and buy recall to 0.79%–14.68%. This is described as behavioral inertia, not risk aversion.

• Basket compression. Real active days often involve multiple trades — sometimes over a hundred, and frequently the same symbol multiple times. Simulators overwhelmingly emit a single ticker, so Ticker Exact Match stays low and set-valued portfolio allocation is lost.

• Market context helps only partially. In the ablation, adding point-in-time market data raised action accuracy (28.13% → 40.63%) and ticker exact match (16.67% → 20.00%), but worsened quantity estimation (MAPE 66.07% → 89.95%). Cumulative behavioral deviation improved (78.26% → 60.87%) but stayed high.

• Real users respond to markets; models barely do. Users trade more when markets have recently fallen — a negative correlation over one day (−0.063) and one week (−0.112), turning positive over one month (+0.023). Weekly trading rate falls from 58.3% in the worst fifth of days to 43.9% in the best, surviving a within-user test (t = 4.60 over 82 users). Models reproduce roughly a tenth of this swing and mention market declines in only a small fraction of explanations.

• Models rely on a fixed impression of each user. For about two-thirds of users, every date receives the same prediction. Accuracy becomes a property of which user is being predicted rather than which day — an offset of roughly 14 points of real market response versus about 1 point extracted by the model.

• Trajectory similarity is not behavioral fidelity. A simulator can post safer or better returns simply by omitting a user's losing trades. Return-level agreement therefore cannot validate a user simulator and may actively reward inaction.

• Temporal drift compounds. Over the rolling horizon, the gap in cumulative active trading points between real users and simulators widens continuously, producing artificially low-turnover, high-inertia synthetic investors.

Methodology in Plain English

The researchers deliberately avoided real brokerage data, citing privacy and regulatory risk. Instead they recruited 120 volunteers onto a paper-trading platform where participants traded non-redeemable virtual funds under real-time market conditions, with no connection to real accounts or assets. The platform logged everything: news browsing, stock and ETF views, chatbot interactions, simulated orders, and virtual portfolio states.

They then built a rolling prediction task. For each participant on each trading day, the simulator receives only information available before that day — a stable profile derived from earlier history, the current virtual portfolio, long-term trade history, the past seven days of behavior, and (in the market-enhanced condition) contemporaneous market data. It must then predict the next trading day's action.

The prediction is factored into stages: first, does a trade happen at all; second, which securities; third, direction and quantity for each. Evaluation is conditionally applied — ticker and quantity metrics only count when the model actually predicted a buy or sell. The authors ran six LLMs through this protocol, compared market-free against market-enhanced settings, and then dug into the error structure by examining action composition, transaction structure, and cumulative behavior over time.

Why This Matters

Impact on research. The paper argues that treating a user simulator as a fixed, unexamined component confounds downstream conclusions. An advisory agent can look effective simply because its simulated counterpart is unrealistically passive or compliant. This work gives the field a way to measure that confound before it contaminates results, and reframes simulator evaluation away from aggregate accuracy toward action, security, quantity, and trajectory fidelity jointly.

Real-world applications:

• Pre-deployment testing of robo-advisors — checking how different user types respond to a rebalancing proposal or risk warning before any client is exposed.
• Stress-testing recommendation engines — detecting whether a system's apparent success depends on simulated users who never push back or trade actively.
• Regulatory and compliance scenarios — modeling likely retail behavior under market shocks without harvesting real customer data.
• Product design for retail brokerages — understanding where a proposed feature might trigger behavior the simulator systematically fails to anticipate.

Industry relevance. The finding that models ignore supplied market context while a well-designed agent would condition on it is directly actionable for anyone building financial LLM systems. The structural biases — suppressed liquidity provision, overstated stability, missed path-dependence — are exactly the failure modes that would distort portfolio risk estimates or advisory backtests.

Future Directions

• Scale up participants and horizon. The study is explicitly a pilot; 120 volunteers over four months limits how confidently the behavioral patterns generalize across market regimes.

• Model evolving portfolio constraints explicitly. Quantity prediction failed most sharply, and the authors attribute this to private constraints — available cash, existing positions, risk budget — that external market context cannot substitute for.

• Support multi-order and set-valued prediction. Current architectures emit one ticker and one quantity; real trading days are baskets. This requires rethinking the output space, not just better prompting.

• Separate contextual availability from contextual use. The core open question is not whether market data is supplied, but whether an agent conditions on it rather than on a frozen per-user impression — a distinction the paper quantifies as roughly 14 points of real response versus about 1 point extracted.

Target Audience

Researchers working on LLM-based user simulation, human-behavior modeling, or agent benchmarking will find the protocol design and failure taxonomy most useful. Financial AI practitioners building advisory, recommendation, or risk systems should read the structural-bias section closely, since it identifies failure modes that aggregate metrics conceal. Behavioral economists and HCI researchers interested in longitudinal individual decision-making will also benefit, though the paper assumes some familiarity with classification metrics and evaluation ablations.

Authors’ abstract

Large language models (LLMs) are increasingly used as user simulators, but their ability to reproduce evolving individual financial decisions remains unclear. We present a preliminary study in a controlled paper-trading environment with 120 volunteers. Participants used non-redeemable virtual funds under real-time market conditions; no real brokerage accounts, real-money positions, or real transaction records were accessed. Given only information available before a prediction cutoff, a simulator predicts the participant's next-trading-day action, traded security, and transaction quantity. We evaluate temporally aligned rolling predictions and compare settings with and without point-in-time market information. Market context improves action and ticker prediction in the controlled ablation, while transaction sizing remains difficult. We also observe systematic behavioral compression: models overproduce hold actions, underpredict sell decisions, and simplify multi-security transactions. These results provide an initial empirical characterization and motivate larger-scale evaluation of individual, temporal, and portfolio-level behavioral fidelity.