Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

AI summary

Overview

Research area: Machine Learning — long-context reasoning, retrieval-augmented generation (RAG), LLM agents, and reinforcement learning for tool-using language models. Technical level: Advanced (assumes familiarity with MDPs, policy-gradient RL such as GRPO, and agentic retrieval pipelines). Scope: The paper proposes and evaluates Librarian Agents, a framework that trains a "builder" LLM via reinforcement learning to reorganize a document corpus into a reusable filesystem workspace that a frozen "answering" agent can navigate for future queries.

What This Paper Is About

LLMs struggle to reason over corpora that exceed their context windows, and today's retrieval systems are reactive: every new query triggers a fresh search, forcing the model to rediscover the same cross-document relationships over and over. Precomputed structures such as knowledge graphs help, but they rely on rigid heuristics (like entity extraction) that may not match what future queries need. The paper asks whether an agent can instead anticipate future queries by proactively curating the corpus into a hierarchical filesystem of synthetic files and reference links before any query arrives, and whether that curation behavior can be learned rather than hand-designed.

Key Contributions

1. Anticipatory workspace construction as a learnable behavior. The paper frames the problem of reorganizing a corpus into an "answer-facing workspace" prior to inference, shifting cost from query time to a one-time precomputation phase that is amortized across many queries.
2. The Librarian Agents framework. A builder agent transforms raw documents into a multi-level filesystem of synthetic files, directories, and reference links back to the original corpus; a frozen answering agent then navigates that workspace to answer individual queries.
3. An MDP formulation with RL optimization. Workspace construction is cast as a finite-horizon Markov Decision Process with four constrained actions — CreateFile, MergeFiles, Delete, and Stop — and the builder policy is optimized with GRPO to directly maximize downstream answering accuracy, using periodic answerability probes plus a terminal correctness signal.
4. Empirical validation on two benchmarks. Evaluation on BrowseComp-Plus (in-domain), an amortized multi-query variant, and zero-shot transfer to BRIGHT, with an ablation isolating the contribution of RL fine-tuning.

Main Findings

• Large in-domain gains on BrowseComp-Plus. With the frozen Qwen3.5-4B answering agent, Librarian Agents reaches 0.75 accuracy versus 0.42 (Embedding) and 0.44 (Reranker). With Gemini 3.1 Flash Lite it reaches 0.77 versus 0.30 and 0.42. The abstract reports an average 33 percentage point improvement over the strongest retrieval and precomputation baselines.
• RL fine-tuning is essential. The builder without RL fine-tuning ("No FT") scores 0.28 accuracy at 127,683 construction tokens, below the RL-trained builder's 0.77, indicating the base model does not naturally produce well-structured memories for downstream use.
• Accuracy-versus-construction-cost comparison (Table 1). GraphRAG: 0.36 accuracy, 11,923 construction tokens. AutoCompaction: 0.04 accuracy, 501,637 tokens. Builder (No FT): 0.28 accuracy, 127,683 tokens. Librarian Agents: 0.77 accuracy, 178,926 tokens.
• Amortization across repeated queries. On 443 frozen synthetic queries over 50 document sets (7–10 queries per set, generated by Gemini 3.1 Flash Lite), the fine-tuned builder improves accuracy from 0.242 ± 0.036 to 0.637 ± 0.029, though the answering agent reads more per query: 9,346 ± 596 tokens and 26.90 ± 1.70 files versus 1,238 ± 155 tokens and 2.72 ± 0.35 files for the base builder. At a hypothetical reuse load of 20 queries per document set, the reported construction cost spreads to roughly 18K tokens per query.
• Better token efficiency after normalization. Librarian Agents uses 33.8% fewer recorded tokens per expected correct answer at ten questions per corpus (RL/Base ratio 0.662, 95% CI [0.504, 0.836]).
• Zero-shot transfer to BRIGHT. Librarian Agents averages 52.0 pass@1 and 66.8 pass@3 across AOPS, Biology, and Stack Overflow, versus the best baseline averages of 20.8 pass@1 (Dense) and 28.0 pass@3 (Dense). Per-domain pass@1: AOPS 36.0, Biology 34.5, Stack Overflow 85.5. The abstract reports a 31-point average accuracy improvement.
• RL changes construction structure. Across the three BRIGHT sub-domains, the trained builder averages 53.4 active synthetic files organized into 45.4 clusters and 8.0 merged artifacts, compared with 4.8 files, 4.2 clusters, and 0.6 merges for the base builder.
• Fallback behavior works qualitatively. In a case study, a BrowseComp-Plus query requiring a detail omitted from the synthetic layer is answered by following a provenance link to the raw file, without enumerating the full corpus.

Methodology in Plain English

The researchers set up two roles. A builder receives a flat pile of raw documents and edits it into a folder structure: it can create new summary files that cite their sources, merge related files into higher-level artifacts, delete unhelpful synthetic files (never raw documents), or stop. A separate, frozen answering agent — Qwen3.5-4B or Gemini 3.1 Flash Lite — then uses only file-listing and file-reading tools to navigate the resulting workspace and answer each query.

To train the builder, they treat the sequence of edits as a decision process. The reward comes from how well the answering agent actually performs: every 8 turns the builder pauses, the answering agent attempts a batch of training queries the builder never sees, and the change in accuracy becomes a reward signal; a final accuracy score is added at the end. They formalize this as potential-based reward shaping, showing the per-step "dense" rewards telescope into the sparse end-of-trajectory objective. Policies are updated with GRPO. In the primary configuration, navigation costs and construction costs are switched off (λ_tok = λ_read = α = 0), so only answerability and terminal correctness drive learning.

Evaluation covers 680 training and 50 held-out BrowseComp-Plus tasks, an amortized variant where one workspace serves many queries, and a zero-shot BRIGHT setup of 150 questions (50 each from AOPS, Biology, and Stack Overflow) where one filesystem per domain supports all its questions. Baselines include embedding retrieval with and without a reranker, late-interaction embedding, Graph RAG, Agentic RAG, RC-style autocompaction, BM25, Dense, Hybrid, and Rerank retrieval variants, plus the no-fine-tuning ablation.

Why This Matters

• Research impact. The paper reframes corpus organization from a hand-engineered preprocessing step into a learned policy judged by downstream answer quality, connecting long-context reasoning, agentic memory, and RL-based agent training. It also offers a clean demonstration that RL can teach a model a capability the base model lacks.
• Real-world applications:
  • Enterprise knowledge bases where design docs, issue threads, and customer reports must serve many different future questions.
  • Large software repositories where code navigation and architecture questions recur across users.
  • User chat histories and personal corpora that an assistant repeatedly queries.
  • Deep-research web corpora, where evidence must be located and combined across scattered pages.
• Industry relevance. Systems like coding assistants already navigate filesystems well but do not pre-build reusable workspaces. A one-time, amortized construction step with measurable token savings per correct answer (33.8% fewer recorded tokens per expected correct answer at ten questions per corpus) is directly relevant to serving cost and latency economics. The approach also composes with existing retrieval infrastructure rather than replacing it.

Future Directions

• Maintaining memories as documents change. Evaluation is limited to static corpora; keeping the anticipatory workspace current under document churn is stated as an open problem.
• Scaling beyond a single large file context. The builder operates over a large file context, which may limit scalability; pairing it with clustering or retrieval is suggested as an extension.
• Deciding when to bypass the synthetic layer. Reference links preserve fallback access, but determining when the answering agent should skip synthesized artifacts — especially for needle-in-a-haystack facts unlikely to survive compression — remains unresolved.
• Richer action spaces and traffic-driven updates. Future work could let the builder compose atomic filesystem commands (touch, mkdir, mv, rm) of which the current four actions are special cases, and let the filesystem update from its own query traffic, treating each answered query as evidence of what the next reader will need.

Target Audience

Researchers and advanced practitioners working on LLM agents, long-context reasoning, retrieval-augmented generation, and reinforcement learning for tool use. It will be most useful to those designing agentic memory systems or serving infrastructure for repeated queries over shared corpora, and to readers interested in how RL can shape structured, reusable artifacts rather than only final answers. Beginners can follow the problem framing and results but will need background in MDPs and policy-gradient methods to engage with the training details.

Authors’ abstract

Reasoning over long document contexts remains a challenge for large language models (LLMs). Standard retrieval methods are inherently reactive, bottlenecking agents with complex, cross-document reasoning at inference time, while precomputed structures like knowledge graphs often fall short because they rely on rigid heuristics. Instead, can we allow agents to perform anticipatory directed curation: learning to proactively reorganize unstructured corpora prior to inference to efficiently serve diverse future queries. We build on a key asymmetry in modern agentic harnesses (e.g., Claude Code): while LLM agents are adept at navigating existing filesystems to solve an immediate task, they are rarely trained to construct reusable workspaces for future use. To instantiate this capability, we introduce Librarian Agents, a framework in which a builder agent transforms raw documents into a hierarchical filesystem of synthetic files and reference links, which forms a workspace tailored for a downstream answering agent. We formulate workspace construction as a Markov decision process and optimize the builder policy via reinforcement learning to directly maximize downstream answering accuracy. Empirically, Librarian Agents outperforms the strongest retrieval and precomputation baselines by 33 percentage points on BrowseComp-Plus, amortizes upfront construction costs across repeated queries, and generalizes zero-shot to unseen domains on BRIGHT with a 31-point average accuracy improvement.