PANINI-DR: Controlled Deep Research through Generative Semantic Workspaces

AI summary

Overview

Research area: Natural Language Processing — specifically multi-agent "deep research" systems that answer questions by searching, reasoning over, and connecting evidence across many documents.

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, ReAct-style agents, and LLM-as-judge evaluation, but its core ideas are explained through a worked example.

Scope: The paper introduces PANINI-DR, a multi-agent harness that keeps an explicit, verifiable research workspace for subquestions and their dependencies, and shows it achieves the strongest average accuracy across three deep-research benchmarks (BrowseComp-Plus, FRAMES, MoNaCo) among seven compared harnesses sharing one model backbone and frozen corpora.

What This Paper Is About

Deep research questions — such as "How old was the U.S. President's wife when the Camp David Accords were signed?" — cannot be answered in one retrieval step. An agent has to break the question into subquestions, search for each, and carry earlier findings forward. The problem is that as investigations grow, the running record of what has been found, what depends on what, and what still needs checking becomes unwieldy and error-prone. PANINI-DR's goal is to make that state explicit and inspectable in a shared workspace, and to verify each intermediate finding against its cited evidence before it is reused to guide further research.

Key Contributions

1. A shared workspace for research and verification. The authors introduce PANINI-DR, a multi-agent harness that organizes subquestions, their dependencies, findings, and supporting evidence in a shared workspace built on the Generative Semantic Workspace (GSW) representation. This separates coordinating the overall investigation from researching and verifying individual subquestions.

2. Strong performance backed by ablations. PANINI-DR achieves the strongest average performance across the evaluated benchmarks, with the largest gains on harder questions. Ablation studies isolate how structured planning, persistent state, and verification with evidence checks each contribute.

3. Evaluation beyond answer correctness ("observed support"). The authors propose observed support, a metric that grants answer credit only when the credited content is recoverable from the system's saved evidence — addressing the fact that a model can answer correctly from memory alone.

4. A typed execution protocol. Beyond a dependency graph, the contribution is the protocol connecting typed research assignments, subquestion-scoped review, and admitted values that subsequent work can consume, plus plan-checking and a bounded repair procedure for structural errors in the plan.

Main Findings

• Best average across benchmarks: PANINI-DR scores 73.3% on BrowseComp-Plus, 79.1% on FRAMES, and 0.549 mean F1 on MoNaCo, for an average of 69.1 — the highest among the compared harnesses.

• Large margin on BrowseComp-Plus: The 73.3% accuracy outperforms the next-best harness, SLEUTH at 57.3%, by 16.0 percentage points.

• Hard questions are where it separates: On the BCP hard group, PANINI-DR answers 52 of 50... specifically, 52 accuracy versus 28 for the next-best harness — described by the authors as nearly twice as many. Its easy and medium scores are 92 and 76.

• Not the top on MoNaCo by official F1: HyMem has the highest MoNaCo F1 (0.612) and PANINI-DR ranks fifth there, though the authors state their collection answers are the most precise and PANINI-DR leads MoNaCo once answers must be backed by saved evidence.

• Highest observed support on all three benchmarks: PANINI-DR scores 70.7 on BCP, 66.1 on FRAMES, and 0.458 on MoNaCo, for an average of 60.9 versus 49.6 for SLEUTH — a wider margin than on official accuracy.

• Planning and persistent state help: Adding a plan, findings linked to its subquestions, and plan revision raises BCP accuracy from 41.3% to 58.0%. The 16.7-point gain persists when both conditions use the same final answer synthesis (56.7% versus 73.3%).

• Findings actually feed later research: With planning and state, 77.4% of newly recorded entities absent from the question appear in a later search, versus 32.1% without them.

• Verification before reuse adds 16.0 points: Removing the Verifier plus citation and evidence-attachment checks costs 16.0 points on BCP accuracy and observed support, with assignment and answer-type checks held constant. Of 101 proposals endorsed after an earlier rejection, 77 changed the answer or cited sources.

• Better source coverage: On BCP, PANINI-DR reaches most annotated source pages while SLEUTH reaches fewer than half, and more of the pages PANINI-DR cites are annotated; the pattern holds on FRAMES and MoNaCo.

• Evidence is independently useful: In a common-reader test where the same gpt-oss-120b reader answers from each system's evidence, PANINI-DR's evidence yields the highest answer quality on all three benchmarks.

• Cost is higher but not the whole story: An estimated 66.5% of input tokens came from cached prefixes, so estimated computation is 7.3× SLEUTH's while prompt tokens sent are 19.2×. Running SLEUTH 8 times per question at 1.12× PANINI-DR's computation reaches 64.7% BCP accuracy versus 73.3%.

• More turns give diminishing returns: Raising the Researcher's turn allowance from 20 to 50 and 80 improves BCP accuracy to 74.7% and 76.0%, but 80 turns uses 2.8× the default's prompt tokens for 2.7 more points, so 20 turns is kept.

• Results survive a grader swap: Re-grading all 150 BCP questions from 5 systems with gpt-5.6-terra instead of gpt-5.6-luna leaves four systems identical and adds 1 correct answer for PANINI-DR (73.3 to 74.0).

Methodology in Plain English

The system works like a small research team with a shared whiteboard.

Plan first. An Orchestrator agent asks a Planner to turn the question into a structured plan of subquestions and the relationships and computations that connect them. Unknown answers appear as "blanks," each with a natural-language retrieval hint and metadata such as an expected answer type (e.g., "date") and a verification claim describing what a valid answer must satisfy. The plan forms a directed acyclic graph, so the system knows which subquestion feeds which.

Check the plan. Before research starts, the harness looks for structural errors — invalid references, missing or multiple answer targets, circular dependencies, and hints inconsistent with declared dependencies. A separate LLM call proposes corrections within a bounded repair procedure, and the harness checks those corrections before applying them.

Research in parallel. The Orchestrator launches Researcher agents on subquestions whose inputs are available. Each Researcher tries one retrieval step first, substituting known findings into the hint (e.g., "When was Rosalynn Carter born?"), and proposes an answer with citations. If that fails, it continues with ReAct-style iterative search and reasoning. A separate reading tool uses its own LLM to extract relevant information and quotations, keeping raw source text out of the main context.

Verify before admitting. A Verifier agent checks whether the cited evidence supports the candidate and whether it distinguishes that candidate from plausible alternatives; both must hold. Failed checks return feedback that the Researcher can use, with a limited number of retries. Admitted findings fill the blank in the workspace and record evidence and review status.

Compute and answer. Planned computations run in code once their inputs exist. In the worked Camp David example, the birth date of 18 August 1927 and signing date of 17 September 1978 yield 51 completed years, because the birthday had already passed. If a Researcher discovers a missing intermediate fact, it can return a suggested subquestion, and the Orchestrator can have the Planner revise the plan.

Compare fairly. All systems run on the same gpt-oss-120b backbone, the same frozen corpora and retrieval service (BM25 plus Qwen3-Embedding-8B, then jina-reranker-v3.5), and are graded by 3 gpt-5.6-luna judgments aggregated by majority vote on BCP and FRAMES and by maximum F1 over complete assessments on MoNaCo. Evaluation uses the BCP CF-150 split with 50 questions each in easy, medium, and hard groups over the official 100K-document corpus, all 824 FRAMES questions, and all 1,315 MoNaCo questions.

Why This Matters

The paper argues that reliability in agentic research depends as much on what an agent remembers and checks as on how well it reasons. By making research state explicit and gating findings behind evidence-based verification, it offers a template for agents whose conclusions can be audited rather than merely trusted. The observed-support metric pushes evaluation past "did it get the right answer" toward "could a reader reproduce that answer from what the system saved" — a distinction the authors show changes how harnesses rank.

Real-world applications:

• Due-diligence and competitive intelligence reports, where each claim needs a traceable source.
• Scientific and medical literature synthesis, where partial findings must be combined across papers without compounding errors.
• Legal and regulatory research, where the chain of reasoning behind a conclusion is as important as the conclusion.
• Enterprise knowledge work and investigative journalism, where multi-hop questions span many internal or public documents.

Industry relevance: The harness is model-agnostic in design and evaluated on a single open-weight backbone, and the paper reports concrete cost accounting (cached versus uncached tokens, computation multipliers, turn allowances). That makes the trade-off between accuracy and compute directly actionable for teams deploying research agents at scale.

Future Directions

• Generalize beyond one backbone and frozen corpora. The authors note that their primary comparisons use one research backbone and fixed benchmark corpora, and that generalization to other models and open-web research remains untested.

• Make verification cheaper and more reliable. The paper reports that increasing the verification allowance does not consistently improve answer quality, and identifies computationally efficient scaling of verification as an important direction.

• Close the gaps in verification's reach. Verification depends on the subquestions the Planner identifies, the candidates considered, and the evidence available to the Verifier; checking individual findings cannot guarantee the plan covers everything needed to answer the original query.

• Strengthen evidence-level guarantees. Observed support measures answer recoverability from saved evidence whose coverage and format vary across systems, and does not establish that every claim is supported; the complementary claim-support audits also rely on model judgments.

Target Audience

Researchers and engineers building LLM agent systems for multi-step question answering will get the most from this paper, particularly those working on agent memory, planning, and verification. It also suits evaluation researchers interested in metrics that go beyond answer accuracy, and practitioners who need to judge whether a deep-research system's outputs can be audited against their sources. Readers without background in retrieval-augmented generation or ReAct-style agents will find the worked Camp David example accessible but the benchmark comparisons dense.

Authors’ abstract

Deep research requires connecting multiple steps of search, reasoning, and computation, with intermediate findings guiding what to investigate next. As investigations grow, agents must maintain research state across model calls and check findings before errors propagate into subsequent work. We introduce PANINI-DR, a multi-agent harness that separates coordination of the overall investigation from the research and verification of individual subquestions. A shared workspace, built on the Generative Semantic Workspace representation, records subquestions, their dependencies, findings, and supporting evidence. Researchers investigate individual subquestions, while a separate verification agent checks proposed answers against those subquestions and their evidence. Verification feedback guides further investigation, and recorded findings inform subsequent assignments. We evaluate PANINI-DR on BrowseComp-Plus, FRAMES, and MoNaCo against seven state-of-the-art deep-research harnesses, all using a shared language-model backbone and frozen corpora. PANINI-DR achieves the strongest average performance across the three benchmarks. On BrowseComp-Plus, it achieves 73.3% accuracy, outperforming the next-best harness by 16.0 points. Its advantage is strongest on hard questions, where it answers nearly twice as many as the next-best harness. Ablations further show that structured planning and persistent state improve performance, as does checking each finding against its cited evidence before it is reused. Because a model can answer correctly from memory alone, we also analyze the evidence retained by each harness. PANINI-DR achieves the highest answer recoverability from saved evidence on all three benchmarks, while complementary claim-support audits show stronger evidence-backed answers. Accounting for this evidence changes how harnesses compare and further strengthens PANINI-DR's advantage.