COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Context

AI summary

Overview

Research area: LLM agent architectures for long-horizon reasoning, tool use, and context management (arXiv:2510.08790v1 [cs.AI], published 2025-10-09). Technical level: Advanced. Scope: The paper introduces COMPASS (Context-Organized Multi-Agent Planning and Strategy System), a hierarchical framework that separates tactical execution, strategic oversight, and context organization to improve agent reliability on long-horizon tasks.

What This Paper Is About

Long-horizon tasks that require sustained reasoning and multiple tool interactions remain hard for LLM agents: small errors compound across steps, and even state-of-the-art models hallucinate or lose coherence. The authors identify context management as the central bottleneck, arguing that extended histories cause agents to overlook critical evidence, become distracted by irrelevant information, and fail to replan or reflect on earlier mistakes. COMPASS addresses this by dividing the work among three specialized components.

Key Contributions

1. Formalization of strategic reasoning and context for long-horizon tasks (LHT), highlighting the dependence of task success on explicit context management.
2. The COMPASS framework, a hierarchical architecture that separates tactical execution from strategic oversight through explicit role and context separation.
3. Comprehensive evaluation on GAIA, BrowseComp, and Humanity's Last Exam (HLE), including a test-time scaling extension (COMPASS-TTS) and a post-training pipeline that produces the Context-12B model for efficient context management.
4. New trajectory-level strategic reasoning metrics (PAR, PVR, CA, ERC) evaluated via LLM-as-a-Judge, which expose precision-recall trade-offs at critical decision points.

Main Findings

• Three specialized components: COMPASS uses a Main Agent (ReAct-style tactical reasoning and tool use), a Meta-Thinker (asynchronous monitoring and strategic interventions), and a Context Manager (compresses histories into concise structured briefs).
• Accuracy gains across benchmarks: COMPASS improves accuracy by up to 20% relative to both single- and multi-agent baselines. With Gemini 2.5 Pro, COMPASS reaches 35.4 on BrowseComp, 67.8 on GAIA, and 31.7 on HLE, versus 16.8, 58.6, and 14.8 for a single agent with search and browse tools only.
• Ablation results on BrowseComp: Removing the Meta-Thinker drops Pass@1 from 35.4 to 15.2, with PVR falling from 0.48 to 0.12 and ERC from 0.55 to 0.21 (blind persistence). Removing the Context Manager drops Pass@1 to 26.4 while raising total tokens from 185K to 156K in that configuration (the paper reports token bloat from repeatedly revisiting failed attempts).
• Context-12B efficiency: A Gemma-3 12B model trained with supervised fine-tuning followed by direct preference optimization (DPO) achieves performance comparable to Gemini 2.5 Flash on BrowseComp while using only 70% of its tokens.
• Test-time scaling: COMPASS-TTS improves monotonically with the number of parallel samples n ∈ {1, 2, 4, 8}. Full-pipeline sampling gives the strongest gains but steepest token growth, meta-thinking sampling is a balanced middle ground, and context-management sampling is the most efficient. Performance plateaus around n = 4, with n = 2–4 described as a practical sweet spot.
• Comparison to established agents: COMPASS-TTS (Gemini 2.5 Pro) reaches 43.7 on BrowseComp, 72.1 on GAIA, and 35.2 on HLE. For reference the paper lists DeepResearch (o3) at 51.1, 67.4, 26.6; DeepSeek V3.1 Terminus Agent at 38.5, 63.1, 21.7; and Test Time Diffusion (Gemini 2.5 Pro) at 69.1 on GAIA and 33.9 on HLE (BrowseComp not reported, marked "—").
• Strategic behavior: Single-agent baselines show high PAR but low PVR (blind persistence), while multi-agent baselines improve pivoting but sacrifice persistence. COMPASS balances both: for Gemini 2.5 Pro the full system scores PAR 0.85, PVR 0.48, CA 0.88, ERC 0.55, for a Strategy Adequacy of 0.69.

Methodology in Plain English

The authors first define what makes a task "long-horizon" — success requires a substantial sequence of interdependent reasoning and action steps (e.g., more than 10), often with iterative tool use and dynamic plan revision. They note that the accumulated dynamic context grows roughly linearly with time and can exceed the model's finite context window, obscuring earlier but essential information.

They then split reasoning into two roles: tactical reasoning (executing the current plan step faithfully) and strategic reasoning (monitoring for anomalies and deciding whether to correct, adapt, or conclude). COMPASS assigns these to separate agents with separate contexts.

Operationally, COMPASS runs an outer loop: the Meta-Thinker performs initial planning; the Context Manager synthesizes a context brief for the Main Agent; the Main Agent executes a ReAct-style loop with monitoring; the Meta-Thinker issues a decision (persist, pivot, verify, or terminate); if the decision is stop, an Answer Synthesizer produces the final answer; otherwise the notes are updated and the next iteration begins, until convergence or the maximum round is reached.

Evaluation uses Pass@1 accuracy on GAIA (all Level 1–3 non-image tasks), BrowseComp (1,266 web navigation tasks requiring verification of entangled facts), and HLE (2,158 questions across mathematics, humanities, and natural sciences after excluding image-based items). Baselines include single-agent systems with varying tools, multi-agent systems (agent-as-a-tool and decentralized handoffs), and Iterative Refinement workflows. All experiments use Gemini 2.5 Pro/Flash as backbone reasoning models with native Google search/browsing and code execution tools.

For Context-12B, the authors used Gemini 2.5 Pro with a searching tool as a data engine, collecting training data from COMPASS rollouts on GAIA, SimpleQA, MMLU-Pro and similar benchmarks. They filtered out trajectories with fewer than three tool interactions, excluded degenerate completions where the correct answer was reached without reflection, and upsampled cases where context management drove recovery or proper task termination. SFT on Gemma-3 12B produced Context-12B-SFT, then DPO selected preferred summaries based on successful completions with fewer tokens.

Why This Matters

The paper argues that context management, not raw model capability, is the bottleneck for reliable long-horizon autonomy. By making strategic reasoning and context curation explicit architectural components — while retaining the fluidity and autonomy of single-agent systems — it offers a middle path between undifferentiated ReAct loops (which exhaust context) and human-in-the-loop multi-agent workflows (which are unscalable and require manual design of when to inject signals).

Real-world applications implied by the tasks and tools used:

• Deep research and fact verification across many web sources (BrowseComp-style entangled facts).
• Multi-step tool-augmented assistants that combine search, text browsing, and code execution.
• Reduced-cost deployment of agent systems via smaller specialized context models (Context-12B).
• Reliability engineering for agents that must decide when to stop versus keep investigating.

Industry relevance: the finding that oversight quality matters more than raw capacity, and that context curation can be delegated to smaller models, has direct cost and latency implications for production agent deployments built on proprietary frontier models.

Future Directions

• Extending COMPASS beyond controlled reasoning environments to more open-ended domains with richer interoperability mechanisms such as MCP servers and agent-to-agent (A2A) communication protocols.
• Evaluating and post-training open-source models within the framework, since the paper notes common open-source models currently underperform significantly on such tasks and the study primarily focuses on proprietary frontier models.
• Better understanding of the trade-off observed in context management, where certain configurations (for example Gemini 2.5 Flash as Context Manager) exhibit excessive plan revision, triggering strategic interventions that extend execution while improving adaptability.
• Refining test-time scaling policies beyond the observed plateau around n = 4, balancing accuracy against token cost across sampling strategies.

Target Audience

Researchers and engineers working on LLM agent architectures, tool-use systems, and long-horizon reasoning, as well as practitioners building deep-research or multi-step autonomous agents who need to manage context windows, cost, and reliability. The formal definitions of tactical versus strategic reasoning also make the paper useful for readers studying planning, self-reflection, and agent evaluation metrics.

Authors’ abstract

Long-horizon tasks that require sustained reasoning and multiple tool interactions remain challenging for LLM agents: small errors compound across steps, and even state-of-the-art models often hallucinate or lose coherence. We identify context management as the central bottleneck -- extended histories cause agents to overlook critical evidence or become distracted by irrelevant information, thus failing to replan or reflect from previous mistakes. To address this, we propose COMPASS (Context-Organized Multi-Agent Planning and Strategy System), a lightweight hierarchical framework that separates tactical execution, strategic oversight, and context organization into three specialized components: (1) a Main Agent that performs reasoning and tool use, (2) a Meta-Thinker that monitors progress and issues strategic interventions, and (3) a Context Manager that maintains concise, relevant progress briefs for different reasoning stages. Across three challenging benchmarks -- GAIA, BrowseComp, and Humanity's Last Exam -- COMPASS improves accuracy by up to 20% relative to both single- and multi-agent baselines. We further introduce a test-time scaling extension that elevates performance to match established DeepResearch agents, and a post-training pipeline that delegates context management to smaller models for enhanced efficiency.