PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents

AI summary

Overview

Research area: Long-context natural language processing and LLM agent architectures, specifically multi-hop question answering over documents that are far longer than a model's usable effective context.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (GRPO, RLVR), ReAct-style tool-calling agents, KV-cache and prefill/decode cost, and positional-bias phenomena such as "lost in the middle" and "context rot."

Scope: The paper introduces ParSer, an agent architecture that reads every document chunk in parallel with frozen lightweight subagents while a single reinforcement-learning-trained lead agent reasons iteratively over the gathered findings, and it evaluates this design on multi-hop QA with contexts from 7K to 896K tokens.

What This Paper Is About

Long documents must currently be handled by "sequential memory" agents that read one chunk at a time and repeatedly rewrite a compact memory, which makes inference latency grow linearly with document length and makes accuracy depend on where evidence happens to sit. ParSer instead splits the job in two: a bank of frozen per-chunk subagents reads the whole document in parallel, while a trained lead agent reasons in rounds, broadcasting a new query each time based on what it has already found. The goal is to make long-context reasoning accurate, position-robust, and much faster at inference.

Key Contributions

1. A decoupled read/reason architecture. ParSer assigns reading to a bank of lightweight subagents (one bound to each chunk) and reasoning to a lead agent that never sees a raw document token. Reading becomes parallel width; only reasoning stays sequential, so dependency depth scales with the number of reasoning hops K rather than the number of chunks T.

2. An iterative scatter–gather loop for multi-hop reasoning. At each round the lead agent scatters a question-conditioned query to all subagents concurrently, gathers their findings (dropping abstentions), and formulates a deeper follow-up query conditioned on the accumulated evidence, terminating when it commits to an answer or hits the round cap.

3. A training scheme that trains only the orchestrator. The lead agent is optimized with reinforcement learning using a verifiable binary exact-match reward and GRPO, while all subagents remain frozen off-the-shelf models. This keeps training cost independent of document length and concentrates all learnable behavior in query formulation and aggregation.

4. Controlled evidence-perturbation studies plus a latency study. The paper isolates absolute evidence position, logical evidence order, and evidence distance as the three failure modes of sequential memory, and measures wall-clock latency under multiple concurrency settings.

Main Findings

• Accuracy across context lengths. On HotpotQA, ParSer with a 4B backbone reaches a reported average accuracy of 84.6% and the 9B backbone 86.8% (Table 1 lists 84.57% and 86.79% respectively). On 2WikiMultiHopQA, ParSer averages 87.04% (4B) and 88.48% (9B).

• Margins over sequential memory baselines. On HotpotQA, ParSer with 4B and 9B backbones outperforms the strongest sequential memory baseline by 5.7 and 6.7 percentage points on average; at the longest 896K-token setting the gaps widen to 12.0 and 9.9 points, because sequential methods degrade sharply with length while ParSer stays stable.

• Comparison to a million-token frontier model. Scaling to a 9B backbone, ParSer surpasses DeepSeek-V4-Pro — which natively supports a one-million-token context and was run at its Max reasoning-effort mode and scored 80.47% average on HotpotQA — by 6.3 percentage points.

• Full-context baselines collapse with length. Qwen3.5-4B in thinking mode falls from 80.47% at 7K tokens to 31.25% at 896K on HotpotQA; Qwen3.5-9B in thinking mode falls from 77.34% to 46.88%. DeepSeek-V4-Pro (think-max) falls from 82.03% to 78.91% over the same range.

• Robustness to evidence position. Across 512 HotpotQA questions with all evidence placed in a random [st, st+10] percentile band of an 894K-token document (st stepping from 0 to 90 in increments of 10), ParSer stays flat while MemAgent drops sharply when evidence sits between the 50th and 70th percentiles; ReMemR1's callback module only partially mitigates this.

• Robustness to evidence order. Using 512 two-hop bridge-comparison questions from 2WikiMultiHopQA in 894K-token documents with the same paragraphs in logical versus reversed dependency order, MemAgent and ReMemR1 degrade substantially under reversal, while ParSer remains stable.

• Robustness to evidence distance. With the two evidence paragraphs separated by a varying number of distractor paragraphs and padded by 1,600 paragraphs before and after, sequential methods deteriorate as the number of intervening paragraphs grows, while ParSer stays stable.

• Latency. Parallel reading yields an 11× reduction at 896K tokens under single concurrency (78.22 s versus MemAgent's 876.20 s per sample) and a 1.7× advantage under a concurrency of 16 (58.86 s versus 101.94 s). At concurrency 32, the figures are 58.76 s for ParSer versus 74.96 s for MemAgent.

• Full-context latency trade-off. Full-context non-thinking has the lowest latency on short subsets (1.16 s at 7K tokens, concurrency 1) but reaches 185.42 s at 896K tokens and cannot finish high-concurrency 896K subsets due to GPU memory limits; full-context thinking reaches 375.32 s at 896K tokens.

• Subagent size saturates. Replacing the default Qwen3.5-4B subagent with Qwen3.5-2B drops the HotpotQA average from 84.57% to 78.26%; scaling up to a Qwen3.5-9B subagent gives 84.77%, essentially saturating, which the authors attribute to the simplicity of the per-chunk query task.

• Chunking matters a lot. Removing chunking entirely (a single subagent fed the full document) drops the average to 73.76%, with 53.13% at 896K tokens. Larger intermediate chunks also hurt: 16,384 tokens gives 83.59%, 65,536 gives 81.84%, and 131,072 gives 80.34%, against 84.57% for the default 4,096-token chunks.

• Out-of-distribution behavior. MemAgent and ReMemR1 perform considerably worse on the out-of-distribution 2WikiMultiHopQA than in-distribution, whereas ParSer holds up; the authors attribute this to training only the lead agent, which never sees the document and therefore learns question reasoning rather than document-specific summarization.

Methodology in Plain English

The document is cut into fixed-size chunks (at most 512 tokens per chunk during training, 4,096 tokens at inference). One small, frozen model instance is assigned to each chunk. A separate "lead" agent is given only the question and an iterative interface: it writes a query, all chunk agents answer simultaneously against their own chunk (returning JSON findings or a short abstention), and the non-abstaining findings come back as an observation. The lead agent then thinks and either asks another, more specific question informed by what it just learned, or answers.

This loop is a ReAct-style sequence of interleaved thinking and actions, capped at 9 turns during training and 12 at inference, with each turn capped at 2,048 tokens. Because cross-chunk dependencies are carried through successive rounds of the lead agent's query chain rather than through a document-ordered memory, the architecture handles multi-hop questions where a later hop is only identifiable after an earlier one is found.

Training uses RLVR with a binary exact-match reward and GRPO, with no format reward because the lead agent uses the backbone's native multi-turn tool-calling format. Observation tokens (the findings from subagents) are masked so the policy gradient applies only to the lead agent's own tokens. Hyperparameters: learning rate 1e-6, mini-batch size 128, 70 warm-up steps, PPO clipping at 0.2, KL regularization coefficient 1e-3, rollout group size 5. Training data is 32,768 synthetic samples derived from HotpotQA, each with 200 paragraphs and roughly 28K tokens. The implementation uses VERL with a Megatron training backend and SGLang rollout service, running asynchronously on 6 H100 GPUs (4 for rollouts, 2 for the actor) plus 10 additional H100 GPUs to serve subagents with SGLang. Qwen3.5-4B serves as the subagent for both the 4B and 9B lead agents. Evaluation uses the Sub_EM metric, selecting the checkpoint with the best in-distribution overall performance and reporting the average over 3 runs.

Why This Matters

Impact on research. The paper reframes long-context handling as a problem of ordering: the document imposes a reading order, the question imposes a reasoning order, and sequential memory agents let the first drive the second. By showing that position, order, and distance sensitivities all stem from a single structural commitment — capacity-limited, document-ordered recurrent compression — it gives a clean diagnostic frame for evaluating future long-context agents. It also demonstrates that a frozen-worker, trained-orchestrator recipe transfers effectively to the long-context setting, and that trainable parameters can be concentrated in the part of the system that actually has to learn something.

Real-world applications (illustrative, extrapolating from the paper's motivating examples):

• Multi-document QA and enterprise retrieval, where the answer requires stitching facts across many unrelated files.
• Legal analysis over case files and contracts, one of the two domains the paper names explicitly.
• Financial or regulatory due diligence, where evidence trails span hundreds of thousands of tokens and the order in which facts appear is arbitrary.
• Large-scale log, incident, or audit-trace investigation, where latency budgets make a linear-in-length traversal impractical.

Industry relevance. The 11× single-concurrency latency reduction at 896K tokens (876 s to 78 s per sample) and the persistent advantage under concurrency are directly relevant to serving costs, since sequential memory agents require GPU work proportional to document length while ParSer's work is proportional to reasoning depth. The design also keeps training cost independent of document length and lets deployments swap in smaller or larger frozen subagents — with a 2B subagent losing roughly 6 points and a 9B subagent adding essentially nothing over 4B — which makes cost/quality tuning straightforward. The paper notes that ParSer uses two GPUs (one H100 for all subagents, one RTX3090 for the lead agent) versus one H100 for the baselines, and argues the comparison is valid because the lead agent and subagents within an instance alternate rather than run simultaneously.

Future Directions

• Closing the deferred results. The provided content truncates Section 5.4 (compatibility with alternative subagent implementations) mid-way through Table 5, so which alternative subagent designs were tested and how they performed is not reported here. Similarly, the RL ablation referenced in Appendix B is cited but its numbers are not in the provided text.
• Pushing context far beyond 896K tokens. Evaluation stops at 6400 paragraphs (896K tokens); whether the parallel-width design keeps its flat accuracy and latency profile at multi-million-token scale is not tested.
• Choosing and adapting subagents more systematically. Since performance saturates between 4B and 9B subagents, the open question is how to pick a subagent and query format that maximizes sparse, high-precision extraction, and whether any subagent-side learning is ever worth the cost.
• Handling early-round query formulation failures. Because the whole system depends on the lead agent's ability to decompose a question into independently answerable sub-queries, the failure modes of a poorly trained or out-of-domain lead policy — and mitigations such as process rewards or better query priors — remain unexplored in the reported content.

Target Audience

Researchers and engineers working on long-context LLMs, agentic retrieval and orchestration, and multi-hop QA. It is most useful for readers who already understand LLM agent loops and RL fine-tuning and want a concrete architectural alternative to sequential memory agents, along with the controlled experiments that justify it. Practitioners building document-analysis pipelines at scale will find the latency and subagent-sizing results directly applicable; readers new to long-context methods should treat the Related Work and Method sections as prerequisites.

Authors’ abstract

Sequential memory agents process long documents by reading chunks one after another while maintaining a compact memory state, coupling document traversal to reasoning depth. This coupling introduces sensitivity to evidence placement and ties inference latency linearly to document length. We introduce PARSER, which decouples reading from reasoning. A bank of lightweight subagents each bound to a single chunk read the entire document in parallel, while a lead agent reasons in depth through iterative scatter--gather rounds: at each round it broadcasts a query to all subagents, aggregates the returned evidence, and formulates a deeper follow-up query conditioned on what has been found so far. This decoupled design concentrates all learnable behavior in the lead agent, which is optimized with reinforcement learning, while the subagents remain frozen off-the-shelf models. On multi-hop QA with contexts ranging from 7K to 896K tokens, PARSER with a 4B backbone outperforms the strongest sequential memory baseline by 5.7 points on average and by 12.0 points at 896K tokens. Scaling to a 9B backbone, PARSER surpasses DeepSeek-V4-Pro by 6.3 points. Controlled experiments confirm that PARSER is robust to perturbations in evidence position, order, and distance, conditions that cause large accuracy swings in sequential methods, while reducing inference latency by up to 11x.