FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents

AI summary

Overview

Research area: LLM agents for deep research — long-horizon, tool-using systems that gather web evidence and synthesize long-form reports.

Technical level: Intermediate. The ideas (a persistent file workspace, two cooperating agents, iterative review loops) are conceptually simple, but the paper assumes familiarity with ReAct-style agent loops, context windows, and LLM-as-a-judge benchmarking.

Scope: The paper proposes a dual-agent, file-system-based framework that moves all intermediate research state out of the model context and into a persistent on-disk workspace, and shows that spending more computation on evidence building yields better reports.

What This Paper Is About

Deep research tasks push LLM agents through hundreds of webpages and into reports longer than 10K tokens, which routinely exceeds the model's context window. When that happens, the tokens available for collecting evidence and for writing compete with each other, and agent execution can halt entirely — so systems cannot simply "think longer" to get better. FS-Researcher's goal is to make test-time scaling possible for deep research by storing evidence, notes, plans, and logs in a file system that can grow far beyond the context window, rather than keeping them in the prompt.

Key Contributions

1. A dual-agent, file-system-based framework. A Context Builder acts as a librarian that browses the internet, deconstructs the research topic, writes structured notes, and archives raw sources into a hierarchical knowledge base; a Report Writer then composes the report section by section, treating the knowledge base as its only source of facts.
2. A persistent workspace as shared memory and coordination medium. All workspace files are stored in Markdown and split into deliverables (e.g., index.md, the knowledge_base/ and sources/ directories, report.md) and control files (Todos with [PENDING]/[IN-PROGRESS]/[COMPLETE] statuses, a curated Checklist, and Logs). The framework natively supports multi-session workflows and iterative refinement beyond the context window.
3. Empirical validation across three benchmarks and multiple backbones, with state-of-the-art report quality on DeepResearch Bench and DeepConsult, and improvements over official agent harnesses on the answer-verifiable BrowseComp.
4. Evidence of effective test-time scaling, showing a positive correlation between final report quality and the amount of computation allocated to the Context Builder.

Main Findings

• State-of-the-art on DeepResearch Bench. FS-Researcher with Claude-Sonnet-4.5 reaches 53.94 RACE, outperforming the strongest baseline (RhinoInsight) by +3.02. Its comprehensiveness and insight improve by +3.74 and +4.4 over the previous best. FS-Researcher (GPT-5) scores 52.76 RACE and FS-Researcher (Gemini-2.5-Pro) scores 52.51.
• Gains are not just from a stronger backbone. With the same GPT-5 backbone, FS-Researcher improves RACE by +2.16 over LangChain Open Deep Research; with the same Gemini-2.5-Pro backbone, it beats RhinoInsight by +1.59 RACE.
• The harness itself matters. Gemini-2.5-Pro with only a search tool scores 31.9 RACE; inside its official harness it reaches 49.71 RACE; FS-Researcher pushes it to 52.51 RACE.
• Citation behavior varies by model. FS-Researcher (Gemini-2.5-Pro) reaches 78.20 citation accuracy and (Claude-Sonnet-4.5) 76.17, while GPT-5 reaches 60.04 — the paper attributes this to GPT-5 stacking several citations at the end of a paragraph, creating misalignment between cited sources and extracted statements.
• Best win rate on DeepConsult. FS-Researcher (Claude-Sonnet-4.5) attains the highest win rate (80.00%) and best average score (8.33), with the lowest loss rate (9.58%). Its results are based on a sampled 20 queries due to budget limits. FS-Researcher (Gemini-2.5-Pro) achieves 74.88% win rate and 7.62 average score; FS-Researcher (GPT-5) achieves 73.28% and 7.26.
• Improvements hold on a verifiable benchmark. On a random 100-query subset of BrowseComp, FS-Researcher (Claude-Sonnet-4.5) scores 55.0% accuracy versus 43.9% for the official harness, and FS-Researcher (GPT-5) scores 68.0% versus 54.9%.
• Knowledge bases grow with more context-building rounds. Going from 3 to 10 rounds increases archived sources, notes, unique URLs, report length, and citation count. Most gains occur from 3 to 5 rounds (+11.7 sources / +10.8 URLs), with smaller increments from 5 to 10 (+5.5 / +5.9), indicating diminishing returns as the KB becomes more complete.
• More context-build computation yields better reports. On 10 sampled DeepResearch Bench queries with GPT-5, RACE rises from 51.18 (3 rounds) to 52.37 (5 rounds) to 53.05 (10 rounds); every quality dimension improves except readability.
• Readability is the one non-monotonic dimension. It peaks at 5 rounds (51.93) and slightly drops at 10 rounds (51.66). The paper argues comprehensiveness and readability are conceptually orthogonal and that the tradeoff is a presentation-level artifact from denser, more technical writing, which a targeted post-hoc rephrasing pass can recover.
• All three core design choices matter (ablations, GPT-5). Removing the persistent workspace drops RACE from 52.76 to 48.69 (insight −7.95); merging the two agents into one drops it to 42.41 (insight −16.89, comprehensiveness −11.06); writing the report in one shot drops it to 47.63 (insight −8.80). Insight is the most affected dimension in every case; instruction following is the least affected.
• Case study confirms the pattern. For an insurance-companies query, archived sources rose only from 54 to 59 between 5 and 10 rounds, but evidence notes rose from 75 to 98. Readability decreased monotonically across 3/5/10 rounds (59.32 → 55.49 → 54.79) as jargon and citation density increased.
• File I/O is cheap. File operations introduce negligible latency, less than 0.03% of total wall-clock time.
• Costs scale with rounds. Average cost rises from 6.10 to 8.16 to 12.54 $/query for 3/5/10 context-building rounds. Context compression with a smaller summarizer model can reduce Context Builder cost by 47% with negligible quality loss.

Methodology in Plain English

FS-Researcher splits deep research into two stages that share one workspace.

Stage 1 — Context Builder. Given a topic, this agent uses a standard ReAct loop (thought → action → observation) with two tool families: file-system tools (ls, grep, read_file, paginated; insert/delete/replace) and web tools (search_web via Google SERP API, read_webpage via Jina AI API). It writes three deliverables: index.md (a "table of contents" describing how the topic is deconstructed and how the knowledge base is organized), a knowledge_base/ directory of distilled notes in a descriptively named tree, and a sources/ directory of archived raw webpages. Every statement in the notes carries a citation pointing to a file in sources/. The structure is not planned once and then filled — index.md and the directory tree evolve as the agent's understanding develops. At the end of each session, the agent reviews the KB against a curated checklist, marks any problematic items [IN-PROGRESS], logs the findings, and continues until it hits the session budget or finds no issues.

Stage 2 — Report Writer. Web browsing tools are removed, so the KB is the only source of facts. A first writing session produces an outline that also serves as the todo file. Each subsequent session composes exactly one section, then self-checks against a section-level checklist before marking it [COMPLETE]. This matters because one-shot generation "tends to read like a mere list of facts." After all sections pass, a report-level checklist review runs; failures re-open the relevant sections. This stage has no budget limit.

Evaluation. DeepResearch Bench uses RACE (LLM-judge report quality across comprehensiveness, insight, instruction following, and readability, scored relative to a reference report) and FACT (effective citations and citation accuracy), with Gemini-2.5-Pro as RACE judge and Gemini-2.5-Flash for FACT. DeepConsult uses a pairwise LLM-as-a-judge protocol over 4 dimensions. BrowseComp provides answer-verifiable accuracy. All scores are averaged over 3 test runs. Scaling and ablation experiments use 10 randomly sampled DeepResearch Bench queries with GPT-5.

Why This Matters

Impact on research. The paper reframes the context-window limit as a state-management problem rather than a compression problem. Prior work summarizes tool observations to keep trajectories alive, which the authors describe as an inherently lossy bottleneck where fine-grained evidence and provenance are dropped and errors compound across summarization stages — and which remains bounded by a hard context limit. By making intermediate artifacts persistent and revisitable, FS-Researcher converts previously ephemeral thoughts and observations into a durable resource, and shows that this is what unlocks test-time scaling for deep research.

Real-world applications:

• Automated market, strategy, investment, and risk-analysis briefings of the kind covered by DeepConsult's business and consulting queries.
• Technical or scientific literature reviews requiring provenance, since every distilled note carries a citation back to an archived source file.
• Compliance and due-diligence research where the audit trail of what was read and when matters as much as the conclusion.
• Any long-horizon agent workflow needing reference lists, time-horizon roadmaps, and structured deliverables that human collaborators can inspect mid-task.

Industry relevance. The framework is model-agnostic across GPT-5, Gemini-2.5-Pro, and Claude-Sonnet-4.5, and the paper reports that GPT-5-mini with additional context-building rounds achieves performance comparable to OpenAI-DeepResearch at a substantially lower cost ($2.51 vs. $6.10/query). Together with the 47% cost reduction from summarizer-based context compression, this suggests a practical route to cost-controlled research agents. The code and data are anonymously open-sourced at https://github.com/Ignoramus0817/FS-Researcher.

Future Directions

• Reducing dependence on strong backbones. The paper's stated main limitation is that the framework needs robust multi-turn planning, broad web search, long-form writing, and strong function calling. Smaller or less capable backbones such as gpt-5-mini may take shorter trajectories, stop prematurely, and make more file-operation errors (incorrect edits, inconsistent state updates, erroneous tool use), lowering task success rates. Designing a less demanding framework for smaller models is named as future work.
• Closing the readability gap. Readability peaks at 5 context-building rounds and falls at 10, and writing quality is the consistent relative weakness on DeepConsult — especially for Claude, where the term-heavy, citation-dense style is not preferred by the judge. Improving clarity, structure, and conciseness without sacrificing coverage is an open problem.
• Improving citation grounding. GPT-5's habit of stacking citations at the end of a paragraph produces lower citation accuracy (60.04) than other backbones, raising the question of how to enforce fine-grained statement-level attribution regardless of model style.
• Managing persistence risks. Persisting retrieved materials and intermediate notes may inadvertently store sensitive or copyrighted content and, in untrusted environments, increases the attack surface for prompt injection or malicious pages that attempt to influence tool actions. The paper also notes that citation grounding does not prevent propagation of inaccurate, biased, or outdated web content.

Target Audience

Researchers and engineers building LLM agents for long-horizon tasks, especially those working on deep research, web-search agents, or context-management strategies; practitioners evaluating agent harnesses for report generation, due diligence, or consulting-style analysis; and anyone interested in test-time scaling techniques that extend agent capability without changes to the underlying model. The paper will also be valuable to readers tracking how file-system workspaces — familiar from coding agents and AI-powered IDEs — transfer to knowledge-work domains.

Authors’ abstract

Deep research is emerging as a representative long-horizon task for large language model (LLM) agents. However, long trajectories in deep research often exceed model context limits, compressing token budgets for both evidence collection and report writing, and preventing effective test-time scaling. We introduce FS-Researcher, a file-system-based, dual-agent framework that scales deep research beyond the context window via a persistent workspace. Specifically, a Context Builder agent acts as a librarian which browses the internet, writes structured notes, and archives raw sources into a hierarchical knowledge base that can grow far beyond context length. A Report Writer agent then composes the final report section by section, treating the knowledge base as the source of facts. In this framework, the file system serves as a durable external memory and a shared coordination medium across agents and sessions, enabling iterative refinement beyond the context window. Experiments on two open-ended benchmarks (DeepResearch Bench and DeepConsult) show that FS-Researcher achieves state-of-the-art report quality across different backbone models. Further analyses demonstrate a positive correlation between final report quality and the computation allocated to the Context Builder, validating effective test-time scaling under the file-system paradigm. The code and data are open-sourced at https://github.com/Ignoramus0817/FS-Researcher.