LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

AI summary

Overview

Research area: Evaluation methodology for LLM agents (agentic benchmarks), applied to live financial markets.

Technical level: Intermediate to Advanced. Readers should be comfortable with agent scaffolding (ReAct), tool calling, retrieval/memory, and metric design; no deep finance background is required because all rules and metrics are defined in the paper.

Scope: The paper introduces LiveMACEBench, a process-aware benchmark that runs five frontier LLMs as persistent paper-trading agents in live cryptocurrency and U.S. equity markets for roughly 30 days, and evaluates them through both realized returns and trace-derived, capability-specific diagnostics across tool use, persistent memory, rule following, and multi-agent collaboration.

What This Paper Is About

Most agent benchmarks score only the final outcome, which hides which capability produced that outcome. This problem gets worse in evolving environments, where results reflect a closed loop between agent behavior and changing external conditions that the agent does not control. The authors use live markets as a naturally evolving testbed and build matched agent configurations so that mechanism access, effective mechanism use, and realized task benefit can be measured separately rather than conflated.

Key Contributions

1. Process-aware evaluation framing. The authors formulate agent evaluation around the distinction between realized outcomes and the mechanisms that produce them, combining three signals: task outcomes, controlled mechanism effects, and trace-derived process diagnostics.
2. Controlled diagnosis of agentic mechanisms. LiveMACEBench evaluates Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration under matched backbones, environments, and interaction protocols, with a Base ReAct configuration as reference, so capability differences and failure modes can be localized rather than inferred from aggregate performance.
3. A naturally evolving persistent testbed. The framework is instantiated in live cryptocurrency and U.S. equity markets with non-resettable account state and decision rounds every four hours, so earlier actions carry consequences into later rounds and the interaction has no task-defined terminal state.
4. Capability-specific metric suites. The paper defines compact composite scores and their components, including the Tool Call Score (TCS) from Tool Relevance, Invocation Efficiency, Information Coverage, Evidence Faithfulness, Routing Quality, Valid Call Rate, and Hallucination-Free Rate; the Memory Score (MS) from Memory Content Quality, Memory Usage Quality, change in maximum drawdown, and change in tail loss; the Rule Satisfaction Score (RSS) and Rule Auditability Score (RAS) from Hard-rule Pass Rate, Soft-rule Satisfaction, Rule Coverage, and Conflict Handling; and the Observable Collaboration Score (OCS) from Role Diversity and Evidence Integration.

Main Findings

• A pronounced outcome–capability gap. As the evaluation horizon expands, cumulative-return rankings fluctuate substantially, with 21, 19, 19, and 29 pairwise reversals under the tool use, memory, rule following, and collaboration configurations respectively. The corresponding capability rankings show only 0, 5, 8, and 2 reversals.
• No universally dominant model, and capability leaders differ from return leaders. GPT-5.4 leads tool use, Grok-4.20 memory utilization, Gemini-3.1-Pro rule following, and DeepSeek-V3.2 collaboration. Grok-4.20 achieves the highest return in the tool-use configuration despite the lowest Tool Call Score, and DeepSeek-V3.2 leads rule-condition returns while Gemini-3.1-Pro achieves the highest rule satisfaction.
• High information coverage masks inefficient tool use. Information Coverage is above 0.8 in every case (0.809–0.911), while Invocation Efficiency is consistently the weakest judge-rated dimension (0.581–0.731). Qwen3-Max has the highest Invocation Efficiency (0.731) and judge mean (0.855), while GPT-5.4 has slightly higher Tool Relevance (0.899 vs. 0.896) and Information Coverage (0.911 vs. 0.901).
• Tool discovery and tool execution are separate failure modes. Qwen3-Max has the lowest Routing Quality (0.736) but a near-perfect Valid Call Rate (0.978); Grok-4.20 shows the converse, with Routing Quality of 0.805 but a Valid Call Rate of 0.666. GPT-5.4 is strong at both stages (Routing Quality 0.920, Valid Call Rate 0.998). Hallucination-Free Rate is 1 for all four models and is omitted from the figure.
• Persistent access does not imply persistent learning. GPT-5.4 performs 332 memory retrievals but stores only one memory, whereas Gemini accumulates 93 entries; DeepSeek, Grok, and Qwen retain 41, 12, and 11. GPT repeatedly reuses a single broadly applicable rule and rarely identifies new experience worth storing.
• Memory accumulation, content quality, and use quality diverge. Gemini has the largest store with high Memory Content Quality (0.938) but the lowest Memory Usage Quality (0.497); Grok stores only 12 entries with lower Memory Content Quality (0.760) but the highest Memory Usage Quality (0.855); GPT-5.4's single stored memory receives the highest Memory Content Quality (0.978) while the store fails to grow.
• Memory consistently reduces downside risk but has heterogeneous return effects. All five memory-augmented agents reduce tail loss relative to matched memory-free counterparts, and four of five also improve maximum drawdown and consecutive-loss streaks. The memory return advantage grows in the late phase (April 23 onward) for DeepSeek and Grok, and Gemini shifts from a negative full-period differential (−1.32) to a positive late-period differential (+0.53); GPT-5.4 shows no corresponding improvement (−1.46 to −1.80).
• Rule execution and rule auditability diverge. DeepSeek-V3.2 reaches a Rule Satisfaction Score of 0.847 but a Rule Auditability Score of 0.623, while Gemini-3.1-Pro scores strongly on both (0.872 and 0.882). A recurring execution bottleneck is R1-02, the single-asset concentration limit, particularly for GPT-5.4 and Qwen3-Max, often arising from incorrect estimation of post-trade exposure rather than from ignoring the rule.
• Return does not identify rule-following capability. DeepSeek-V3.2 and GPT-5.4 achieve nearly identical final returns (+5.765% and +5.738%) but differ substantially in Hard-rule Pass Rate (0.877 vs. 0.682) and Rule Satisfaction Score (0.847 vs. 0.634); Gemini-3.1-Pro has a lower return (+3.503%) but the highest Hard-rule Pass Rate, Rule Satisfaction Score, and Rule Auditability Score.
• Multi-agent gains are model-dependent and carry a common risk cost. Multi-Agent improves cumulative return over matched ReAct accounts by 2.45, 3.72, and 2.57 percentage points for GPT-5.4, DeepSeek-V3.2, and Gemini-3.1-Pro, but reduces Qwen3-Max's return by 4.39 points. All four models record more sharp gains but also larger maximum drawdowns and more negative average tail losses; Qwen3-Max adds six sharp gains offset by a 2.85-point increase in maximum drawdown.
• The same harness induces distinct collaboration patterns. DeepSeek-V3.2 is the most balanced (Role Diversity 0.760, Evidence Integration 0.852); Gemini-3.1-Pro is evidence-centric (Role Diversity 0.492, Evidence Integration 0.984); Qwen3-Max reuses substantial evidence (Evidence Integration 0.751) with a narrow role set (Role Diversity 0.282); GPT-5.4 nearly collapses to a single-role workflow (Role Diversity 0.012, Evidence Integration 0.518).
• Observable collaboration and realized performance are complementary. DeepSeek-V3.2 combines the highest Observable Collaboration Score with the largest return gain, but the remaining models break any simple monotonic relationship: GPT-5.4 improves return despite almost no role diversity, while Qwen3-Max retains non-trivial evidence integration yet underperforms ReAct.

Methodology in Plain English

The authors give each of five frontier models (GPT-5.4, DeepSeek-V3.2, Gemini-3.1-Pro, Grok-4.20, and Qwen3-Max) a persistent simulated brokerage account and let it trade continuously for roughly 30 days, from April 13 to May 13, 2026, across six cryptocurrencies and seventeen U.S. equities. Market data come from Alpaca's consolidated Securities Information Processor feed for equities and from Hyperliquid for crypto. Every four hours, each agent starts a fresh "decision round" in which it can reason, call tools, and place zero or more trades; the account state carries forward, so a trade changes the portfolio exposed to every later price move. Trading outcomes are tracked at finer intervals than the four-hour rounds using account-equity checkpoints.

To isolate capabilities, the team builds one shared ReAct scaffold and then creates matched variants: a Base agent, a Tool Use agent with a large tool space accessed through dynamic routing, a Persistent Memory agent that stores and retrieves experience across rounds, a Rule Following agent given a hierarchical rule set (hard system constraints R0, hard client policies R1, and soft preferences R2, prioritized R0 > R1 > R2) plus a requirement to state which rules it checked and how conflicts were resolved, and a Multi-Agent Collaboration agent that replaces single-agent reasoning with a manager-led hierarchy of specialists plus a separate execution stage. Because backbone, environment, and protocol are held fixed, differences between variants can be attributed to the mechanism.

Evaluation then combines three layers: task outcomes such as cumulative return; controlled mechanism effects, meaning what changes when a mechanism is added relative to its matched counterpart; and process diagnostics computed from complete decision traces of model responses, tool calls, tool outputs, and actions. Diagnostics mix programmatic checks extracted from execution records (for example Routing Quality, Valid Call Rate, Hard-rule Pass Rate, Role Diversity) with LLM-as-a-judge rubrics (for example Invocation Efficiency, Memory Usage Quality, Conflict Handling), scored over 170 traces per evaluated backbone by Qwen3.5-397B-A17B.

Why This Matters

Impact on research. The paper reframes live environments from leaderboards into diagnostic instruments. It provides a concrete demonstration that outcome metrics, mechanism availability, and mechanism utilization measure different things, and it supplies reusable metric designs and a matched-configuration protocol that other benchmark builders can adapt to non-financial evolving environments.

Real-world applications:

• Model selection for agent deployments: teams can compare models on the capability they actually need (for example economical tool use or auditable rule compliance) instead of a single end metric.
• Compliance and audit tooling in finance: the separation of executed rule compliance from demonstrated rule understanding maps directly onto regulated workflows that require explainable constraint handling.
• Risk monitoring for autonomous trading agents: the finding that memory-augmented agents consistently reduced tail loss while returns varied is directly relevant to risk-control design.
• Memory system design: the multi-stage memory diagnostics (construction, retrieval, application) help engineers locate where a memory subsystem fails rather than only measuring end performance.
• Multi-agent architecture decisions: evidence that multi-agent setups can add return while increasing drawdown, or improve return with almost no role diversity, informs when orchestration is worth its cost.

Industry relevance. Firms deploying LLM agents in long-horizon, stateful settings need to know whether an agent succeeds because of a capability or despite lacking it. The paper's trace-level diagnostics and matched comparisons give practitioners an evaluation pattern that distinguishes capability from luck under shifting conditions.

Future Directions

• Longer horizons and broader regimes. The authors explicitly name extending process-aware evaluation across longer horizons, broader market regimes, and other continuously evolving environments as a natural next step; the current window is approximately 30 days.
• Diagnosing and fixing the identified bottlenecks. Redundant tool interactions, memory construction stagnation, state-grounded exposure estimation for concentration limits (R1-02), and weak cross-agent evidence integration are concrete failure modes that invite targeted intervention studies.
• Calibrating and validating judge-based metrics. Several capability scores rely on LLM-as-a-judge rubrics over 170 traces per backbone; establishing how stable and reproducible these scores are across judges and trace samples is an open methodological question.
• Disentangling environmental variation from agent capability. Because outcomes reflect accumulated state, market variation, and agent behavior jointly, methods that attribute return differences to specific mechanisms under non-stationary conditions remain an open problem.

Target Audience

Researchers and engineers working on LLM agent evaluation, agentic architectures (tool use, memory, orchestration), and benchmark design will get the most from this paper. It is also relevant to quantitative finance and compliance practitioners interested in autonomous trading agents, and to teams choosing among frontier models for stateful, long-horizon deployments. Readers seeking trading strategies will find little here, since the paper's purpose is diagnostic measurement rather than strategy development.

Authors’ abstract

Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability