FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

AI summary

Overview

Research area: AI agent evaluation, with a specific focus on financial workflows, LLM-based coding agents, and open-world benchmarking.

Technical level: Intermediate. The paper is readable without a finance background, but it assumes familiarity with LLM agents, tool use, backtesting, and benchmark design.

Scope (one sentence): The paper introduces FinCosmos, an evaluation paradigm in which a simulated investor hides a grammar-generated trading strategy, the agent must source its own data from the open Web under a fixed cutoff, and programs verify the delivered strategy, data, and metrics without any reference answer.

What This Paper Is About

Financial agents can fail silently: they produce a polished report that looks exactly as convincing as a correct one, while getting the user's intent wrong, using unretrieved or future-dated data, or reporting returns nobody can reproduce. Existing benchmarks eliminate these failures by construction — they hand the agent curated data, state the task in full, and enforce the timeline themselves — so the failures are removed rather than measured. FinCosmos instead leaves all three open at once (intent, data, and time) and grades what the agent actually delivers.

Key Contributions

1. An open-world evaluation framework for financial agents. A simulated investor holds a hidden, grammar-generated strategy; the agent receives no data and must acquire it from the Web; and programs verify the delivery without a reference answer, target state, or realized returns.

2. A released benchmark and tooling. Ten frozen tasks (seeds 0–9), plus the task generator, the evaluator, and a public live leaderboard at https://leaderboard.fincosmos.com.co. New tasks can be drawn on demand rather than reusing a fixed pool.

3. An empirical case for each design choice. The paper shows that closed benchmarks remove silent failures instead of measuring them, and that grading by user satisfaction or by an LLM judge alone would count those failures as successes and would reorder the agents.

4. An evaluation of four coding agents plus a tool-free baseline, showing that none completes the workflow, that they break most often at the workflow's two ends (pinning down the investor's strategy and delivering verifiable results), and that they can learn the tasks from FinCosmos feedback.

Main Findings

• No session is accepted. Across 150 sessions (four agents × ten tasks × three sessions, plus a bare model), the acceptance rate is 0 and pass^k is 0 for every agent and every k. Pre-veto quality ranges only from 20.2 to 37.2 out of 100.

• Satisfaction and correctness diverge sharply. The simulated user was satisfied in 35 of the 150 sessions (23%), and all 35 fail at least one veto, every one caught by a program check. 29 delivered no trade log or no prices, 15 used data after the cutoff or filled trades before their signal, and 15 delivered a strategy other than the user's; 33 of the 35 fail at least one of these three.

• Standard grading would call these successes. Using only the four vetoes an LLM judge decides (dimensions 2.7, 4.1, 5.1, 5.6), 17 of the 35 sessions (49%) would be accepted, giving an 11% acceptance rate instead of 0. The grader also reorders the agents: by user sign-off, Gemini CLI ties for first with 10 of 30 sessions, but none of them passes the LLM-judged vetoes (Fisher's exact test, p < 0.001), while Codex keeps 8 (p = 0.005 against Gemini CLI).

• Convincing reports hide unreproducible numbers. In 29 of the 35 satisfied sessions the agent delivered no trade log or no prices, so reported returns cannot be rebuilt; in the 5 sessions where the program could partly rebuild them, none matches the report. Against the agent's own equity curve, reported metrics match in only 10 of the 35 sessions, contradict the curve in 11, and are reported without any curve in 12. The Kimi Code session in Figure 1 reported a maximum drawdown of −18.96% where its own delivered curve shows −21.91%, past the user's 20% stop, and ranked stocks by 21-day momentum where the user asked for short-term reversal.

• Quality rankings are tight and partly non-significant. Codex (37.2) and Claude Code (37.1) are indistinguishable (p = 0.99, paired permutation test over tasks). Codex scores above Gemini CLI (22.2) and the bare model (20.2), both surviving Holm correction (adjusted p = 0.02); Claude Code's gap to the bare model does not (adjusted p = 0.10); Gemini CLI (22.2) and Kimi Code (22.3) do not differ from the tool-free model (p ≥ 0.44).

• Fluency is not correctness. The bare model earns its quality from the strategy (49.8) and user-experience (44.6) categories but scores 0.0 on data and 4.6 on the backtest. It delivers the user's strategy in 9 sessions, as often as the agents, yet passes no later workflow step.

• Agents break at both ends of the workflow. Of the 120 agent sessions, only 25 deliver the investor's strategy; 15 of those also keep data within the cutoff; 14 also avoid look-ahead; none delivers trades that reproduce its reported returns.

• Elicitation is weak. A task's belief voices 6.7 items on average, the opening request states 2.6–3.7 of them, and by session end the user has stated 4.5–5.4. Agents therefore raise only 54–64% of the details they were not told, and the delivered strategy fails to match 66–83% of the details never stated. Clarification coverage correlates with pre-veto quality (Spearman ρ = 0.25, p = 0.007). The ambiguity veto fails in 68% of sessions; stock selection is the most frequent fidelity failure (52%); transaction cost fails in only 9%.

• Two completeness checks dominate the veto count. Process completeness (dimension 3.8) and the verifier audit (3.7) fail in 119–120 of the 120 agent sessions and do not separate the agents; excluding them, 2 of 150 sessions would be accepted, both Codex.

• Difficulty and time bite. Quality falls with task difficulty (Spearman ρ = −0.24, p = 0.007 over the 120 agent sessions), from 41.6 on tasks of difficulty up to 10 to 26.6 on tasks of difficulty 15 and above. 70% of agent turns hit the 300-second limit; only 58 of 120 sessions reach a fourth turn, and of the 62 that end earlier the user walks away in 51. Failed program vetoes fall from 5.3 after turn one to 3.8 after turn six.

• Cost varies more than quality. Codex reads 3.20 million input tokens per session for the same quality as Claude Code at 1.13 million; Claude Code is the most expensive at API-equivalent prices ($2.24 per session versus about $0.80 for the others).

• Human experts back the machine grades. Two independent financial experts annotated 60 sessions (13 per agent, 8 for the bare model). Human experts attribute 88.33% of failures to the agents, and agree with the LLM judge at κ = 0.87. All 150 sessions were valid under the simulator's decision validator; grading failed for 2 sittings, which were rerun.

• The benchmark is solvable. With FinCosmos's leak-free feedback, a "Claude + Memory" configuration learned all ten tasks, none of which it passed on the first attempt.

Methodology in Plain English

The framework has three moving parts.

The Strategy Generator. A hidden trading strategy is assembled by picking one option for each of six components — stock selection, market timing, portfolio construction, risk control, execution, and transaction cost. Coherence rules reject nonsensical combinations (for example, risk parity weighting without a covariance estimator). This produces roughly 3.8 million distinct strategy combinations before parameter values are drawn, so fresh tasks can be sampled whenever a released set saturates or leaks. Sampling is seeded, so every strategy is reproducible; the released benchmark freezes the first ten sampled tasks.

The User Simulator. A language model plays one of three personas (retail investor, generalist analyst, biotech specialist) and holds the hidden strategy as plain-language convictions with exact numbers, such as "cut to cash if the drawdown reaches 20%". Its disclosure policy is strictly reactive: it states only what the agent asks about or proposes. The opening request is deliberately vague, and the persona prompts forbid evaluation vocabulary like "point-in-time" or "cutoff". A structured decision is validated by a program before each message is sent, and the simulator may only end the session when every core goal is met with cited evidence.

The Evaluator. A session is graded on 45 dimensions in seven categories. Five categories are scored — data (6 dimensions, 1 veto), strategy (12, 7), backtest (8, 4), assessment (5, 1), and user experience (6, 2) — for 37 scored dimensions; cost (3) and process transparency (5) are reported but not scored. Programs judge 17 dimensions, an LLM judge 19, and both together 6. Where an LLM grade has a program counterpart, the program caps it. The pre-veto quality Q combines category scores on a 0–100 scale with equal weight per category; the final score is Q multiplied by a pass indicator. Fifteen dimensions are vetoes, and a run is accepted only when the user is satisfied, no veto fails, and grading is complete.

Four groups of checks target the three silent failures. Data provenance requires every delivered series to trace to a URL or an API call with its query parameters. Point-in-time correctness requires every dated row to be no later than the cutoff — a single late row fails the veto — every fill to come on a later date than its signal, and fundamentals used at entry to have been announced no later than the entry date. Independent recomputation rebuilds returns from the delivered trade log and compares every reported metric within fixed tolerances (0.02 for returns and drawdowns, 0.5 for the Sharpe ratio). Strategy fidelity checks every item of the hidden specification against the delivered files, requiring a supporting quote, and caps the fidelity score by whether the user actually stated the requirement, so an agent is not failed for details that never came up.

Feedback is leak-free: for the data and strategy categories, the agent gets scores without the reasons that would reveal the hidden strategy. The environment gives agents web search, code execution, and a "Financial Modeling Prep" MCP server with 247 financial APIs, over up to six rounds of 300 seconds each, with all ten tasks on U.S. stock trading.

Why This Matters

Impact on research. The paper reframes benchmark design as a question of which failures you choose to keep rather than which tasks you choose to pose. It gives a concrete, ref-free alternative to LLM-judge grading and realized-return scoring, and it shows quantitatively that the grader you pick changes both the score and the ranking of agents (0% versus 11% versus 23% acceptance for the same 150 sessions). The compositional grammar is also a direct answer to task-set staleness and leakage.

Real-world applications:

• Investment research automation. Firms deploying agents to build and report backtests can require point-in-time compliance and independent metric recomputation before any result reaches a portfolio manager.

• Vendor and model procurement. Quality differences between the four agents on the same ten tasks, plus the token and dollar cost per session, support concrete build-versus-buy and cost-benefit decisions.

• Deliverable auditing and compliance. The provenance and recomputation checks are directly reusable as an internal review gate for any research artifact an agent hands to a client or regulator.

• Agent development and regression testing. The diagnostic breakdown by workflow step (25 → 15 → 14 → 0 sessions) tells engineering teams exactly which stage to fix, and the leak-free feedback loop supports training and self-improvement.

Industry relevance. The failure modes measured here — unpinned user intent, undated or future-leaking data, unreproducible reported metrics — are exactly the ones that turn into capital misallocation or regulatory exposure. A 23% satisfaction rate combined with a 0% acceptance rate is a concrete warning that "the user was happy" is not a usable quality bar for financial agents.

Future Directions

• The two completeness checks need calibration. The process-completeness (3.8) and verifier-audit (3.7) vetoes fail in 119–120 of 120 agent sessions and do not discriminate between agents; without them, 2 of 150 sessions would have been accepted. Whether they should be softened, split, or kept as hard gates is unresolved.

• The reproducibility gap is the deepest open problem. No agent, including the memory-equipped one at first attempt, delivered trades that reproduce its reported returns. Understanding whether this is a tooling, context-management, or reasoning limit is a natural next study.

• Improving elicitation. Agents raise only 54–64% of the details they were not told and get 66–83% of the rest wrong, while clarification coverage correlates with quality. Better question-asking policies — and whether they generalize — are an obvious follow-up.

• Generalization of learning and domain coverage. The learning result covers ten frozen tasks with one memory configuration; whether memory transfers to newly sampled strategies from the 3.8-million-strategy space, and whether the paradigm extends beyond U.S. equities (the only asset class in this study), remain untested.

• Simulator and judge fidelity under model change. The paper cites literature that success rates shift with the simulator model and that stronger assistants make weaker simulators; it validates its own simulator with experts (κ = 0.87 on the four LLM-judged vetoes over 30 sessions), but the stability of that agreement across future model versions is an open question.

Target Audience

Researchers building or evaluating agents for high-stakes domains; benchmark designers interested in simulated users, generative task spaces, and reference-free grading; practitioners deploying LLM agents in financial research, trading, or advisory roles; and risk, compliance, and model-validation teams who need to decide when an agent's deliverable is safe to act on. Readers who want a concrete demonstration of why user sign-off and LLM-judge grading are insufficient quality bars will get the most immediate value.

Authors’ abstract

Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and…