JevGPT: Thinking Fast and Slow in Autonomous Research

AI summary

Overview

Research area: Autonomous research agents — specifically, the intersection of retrieval-augmented question answering, document ranking, and dual-process ("fast vs. slow") agent architectures for language models. The paper is categorized under Natural Language Processing.

Technical level: Intermediate. It assumes familiarity with retrieval-augmented generation, BM25 and hybrid retrieval, reranking metrics such as nDCG, and multi-agent search loops, but the architecture itself is described in straightforward component terms.

Scope: One paper proposing JevGPT, a dual-system agent that pairs a GPT-based research planner with a fast judgment model (Jev) for parallel source inspection, evaluated on two fixed-corpus benchmarks.

What This Paper Is About

Autonomous research agents must gather and reason over evidence inside a limited budget, but retrieved sources often go uninspected or unused — the paper notes that retrieval coverage does not guarantee evidence use, and that increasing retrieval depth can improve candidate recall even after preview recall saturates. The goal is to make fast, local source judgments align with the planner's changing evidence needs, so that findings from parallel source inspection feed directly back into research decisions and into the final answer or document ranking.

Key Contributions

1. A dual-system architecture for autonomous research. System Two (a GPT planner) sets evidence objectives, plans searches, and integrates findings; System One (Jev) locates supporting passages, scores relevance, and adapts reading, with independent source judgments running in parallel and selected original passages returned to the planner.
2. An explicit evidence-request interface. System Two issues evidence objectives that bound each local judgment, and readers return original passages with source coordinates rather than summaries, connecting local inspection to the next research action and to the final output.
3. Three coordinated evidence modules. Evidence Localization for QA, Relevance Assessment for ranking, and Evidence Discovery, a proactive asynchronous branch that assesses additional retrieved candidates against the original task while the planner continues working.
4. Empirical evaluation plus evidence analysis and ablations. Experiments on 830 BrowseComp-Plus questions and 105 NeuCLIRBench topics, a gold-document coverage analysis, a cumulative ablation study, and comparisons across different System Two models.

Main Findings

• Best reported accuracy on BrowseComp-Plus. JevGPT with Hybrid BM25 reaches 90.48% accuracy, above Reason-ColBERT (90.00%), RISE (89.88%), AtlasNav (89.16%), Agentica (84.46%), Pi-Serini (82.05%), and BCP/ReAct (77.11%).
• Lower cost and faster completion at comparable accuracy. JevGPT uses 433K System Two tokens, 9.13 searches, 91.7 seconds, and $0.6081 System Two plus $0.0600 System One cost per question, versus $1.6090 for Reason-ColBERT, $1.1050 for RISE, and $0.4053 for Pi-Serini (which is cheaper but less accurate).
• Best reported ranking quality on NeuCLIRBench. JevGPT achieves nDCG@20 of 0.5390 and F1@20 of 36.10, ahead of Pi-Serini (0.5240 / 35.02), Qwen3-Reranker (0.5138 / 35.06), Agentica (0.5107 / 34.19), BCP/ReAct (0.4847 / 30.60), and RankGPT (0.4843 / 32.48), with 158K System Two tokens, 2.68 searches, 79.3 seconds, and $0.5756 + $0.0082 cost per topic.
• Local judgments are cheap. System One API cost is $0.0600 per question on BrowseComp-Plus and $0.0082 per topic on NeuCLIRBench; total cost remains lowest among the GPT-based ranking methods.
• Hybrid BM25 and Evidence Discovery drive the largest QA gains. In the cumulative ablation, adding Hybrid BM25 to Pi-Serini raises accuracy from 72.05% to 79.52%; Evidence Localization adds a smaller gain to 81.20% while reducing searches from 8.70 to 7.17; Evidence Discovery raises accuracy to 86.51% and cuts searches to 5.72 and time to 70.2 seconds, with cost rising from $0.1353 to $0.1403 in System Two spending and from $0.0148 to $0.0448 in System One spending.
• Relevance Assessment drives ranking gains. On NeuCLIRBench, Relevance Assessment raises nDCG@20 from 0.4855 to 0.5256 and F1@20 from 30.56 to 35.28 and cuts searches from 4.90 to 2.92; adding Evidence Discovery slightly lowers nDCG@20 to 0.5206 and raises F1@20 to 35.63 while reducing time (72.2 to 61.7 seconds) and estimated total cost ($0.4147 to $0.3706). Hybrid BM25 alone leaves ranking quality nearly unchanged (0.4855 vs. 0.4904).
• Broader source inspection and delivery. The gold-document coverage analysis over 830 BrowseComp-Plus questions finds that JevGPT and Pi-Serini have similar candidate-pool coverage, but JevGPT achieves higher recall for both inspection and delivery of original text to System Two.
• Planner choice trades off accuracy against cost and time. Across System Two models on 415 matched BrowseComp-Plus questions, GPT-5.5 gives the highest accuracy (92.29%) at $0.5955, GPT-6.1 Sol gives 88.43% at $0.1320, GPT-5.4 mini minimizes spending ($0.0707, 79.76% accuracy), and GPT-5.6 Terra finishes fastest (82.9 seconds, 86.27% accuracy). On NeuCLIRBench, GPT-5.5 again leads (0.5390 nDCG@20, 36.10 F1@20), and GPT-5.4 mini is cheapest ($0.0690, 0.4580 nDCG@20).
• A case study shows localization separating entity identification from fact verification. For a question about Ancrum's by-name, Pi-Serini answers with the heritage society's phrase "a small village with a big history," while JevGPT's System One locates a statement naming "Little Lovely" in another retrieved source titled "Jedburgh," which the planner uses in its next query.

Methodology in Plain English

The agent treats research as a loop. A slow planner (System Two) looks at the accumulated investigation and decides what to do next: issue a search query, issue an explicit evidence request, or produce a final answer or ranking. Candidate retrieval uses Hybrid BM25, which combines a document-level BM25 score with the maximum passage-level BM25 score, min-max normalizes the two lists separately, and fuses them with equal weight (1/2 each).

A fast model (System One, Jev) then does the local work against a stated objective. Jev produces three output types: Noul judgments (probability that a region satisfies a property), Choice judgments (a distribution over options such as text blocks or reading actions), and Score judgments (a distribution over ordinal levels). For QA, the planner supplies up to six local evidence questions; Noul judgments estimate whether a region addresses each question, and Choice judgments pick the most probable original block, plus a second block when the top probability is below 0.8 and the second is at least 0.15. For ranking, Score judgments assign documents to four levels — unrelated, tangential, relevant, highly relevant — and a document is retained when its probability of being relevant or highly relevant reaches 0.25.

Reading adapts: a Choice judgment decides between inspecting further candidates, reading surrounding context, and returning current evidence. A separate Evidence Discovery branch runs asynchronously, assessing the first 60 retrieval units of each new ranking against the original task and returning at most three QA sources or ten ranking documents above a 0.5 lead cutoff. Led by these local findings, the planner issues its next query or assembles the final output.

Why This Matters

The paper argues that the bottleneck in autonomous research is not only retrieving enough candidates but actually converting them into usable evidence, and that shifting effort from repeated searching toward cheap parallel inspection can improve both quality and efficiency. Impact on research: it offers a concrete architectural recipe for connecting fast local computation to a deliberative planner, and provides evidence that local judgment models can be a small fraction of total cost.

Real-world applications:

• Enterprise and legal document search, where a ranked or answer-producing agent must cover a fixed internal corpus within a time and cost budget.
• Scientific literature review and prior-art search, where agents must locate specific supporting statements rather than just relevant papers.
• Multilingual or translated document collections, as evaluated by NeuCLIRBench's English queries over Persian, Russian, and Chinese document translations.
• Customer-support and knowledge-base question answering, where verification of a specific fact (as in the Ancrum by-name example) matters more than topical similarity.

Industry relevance: the results speak directly to the economics of deployed agent systems — API cost per question or topic, token usage, search counts, and elapsed time are all reported per query, and the paper shows total cost stays lowest among compared GPT-based ranking methods despite the added System One model.

Future Directions

• Unify fast and slow thinking in a single model. The limitations section proposes integrating local evidence judgments and research planning into one model that learns when to make rapid local judgments and when to engage in extended reasoning, with joint training to adapt the balance to an evolving investigation.
• Move beyond fixed corpora. The paper states that extending the approach to open-web research and long-form synthesis would explore how fast and slow thinking supports a broader range of research tasks.
• Clarify cost-versus-time tradeoffs in planner selection. The results show that a cheaper System Two does not necessarily produce faster research, and that GPT-5.5, GPT-6.1 Sol, GPT-5.4 mini, and GPT-5.6 Terra each occupy different points relative to the observed Pareto frontier.
• Characterize System One and inspection-depth choices. The paper describes a separate study of System One models and candidate inspection depth under a 3,600-second research budget with a 600-second limit per local judgment, with corresponding results reported in appendices; the retrieved content used here does not include those appendix figures, and the NeuCLIRBench GPT-5.6 Terra row of Table 6 is truncated in the provided text.

Target Audience

Researchers and engineers working on retrieval-augmented agents, autonomous research systems, and information retrieval evaluation will benefit most. It is also relevant to practitioners building cost-sensitive search or question-answering pipelines who need to decide how much computation to spend on reading sources versus issuing more searches, and to readers interested in dual-process or fast-plus-slow model architectures inspired by Kahneman's distinction. Beginners can follow the high-level architecture and headline results, but the metrics, ablations, and retrieval configuration details assume some background in IR and LLM agent design.

Authors’ abstract

Autonomous research agents must acquire and reason over evidence within a limited budget. Yet relevant sources can remain uninspected or unused, leaving a gap between retrieval coverage and the evidence that informs research. The challenge is to align fast local judgments with changing evidence needs and feed their findings back into research decisions. We introduce JevGPT, a dual-system architecture for autonomous research that couples fast evidence processing with deliberative research planning. System Two sets evidence objectives and integrates findings, while System One locates support, assesses relevance, and guides further inspection. Independent source judgments run in parallel; selected original passages inform the next research step. Experiments on fixed-corpus question answering and document ranking show that JevGPT achieves leading task performance, with faster completion and lower cost than comparably effective research agents. Evidence analysis shows broader source inspection and greater coverage of supporting documents in the research loop, highlighting efficient evidence processing as a route to more capable autonomous research.