SearchJev: A Fast and Calibrated System-1 Model for Search Agents

AI summary

Overview

• Research area: Information retrieval (cs.IR), specifically LLM-based search agents, decision modeling, and confidence calibration.
• Technical level: Intermediate. The paper assumes familiarity with language models, token logits, calibration metrics such as expected calibration error (ECE), and agentic retrieval loops.
• Scope: The paper proposes SearchJev, a non-generative "System-1" model that directly scores short search decisions from an LM's output logits, trains it with calibrated soft-label learning, and embeds it in a dual-system search agent alongside a generative "System-2" reasoner.

What This Paper Is About

Search agents constantly make short judgments — is this passage relevant, is the evidence sufficient, what action comes next — and using a generative language model for each of these judgments is slow and produces unreliable confidence. SearchJev addresses this by separating those short decisions from the generative reasoning process: given a search state and a decision schema, it directly scores the legal options instead of generating text token by token. The goal is a single model that handles many decision types quickly while producing confidence estimates trustworthy enough to decide when the agent should escalate to System 2.

Key Contributions

1. SearchJev, a schema-conditioned decision model. It normalizes a causal LM's first-output-token logits over the legal option labels for each decision field, producing a decision distribution for Choice, Score, and Noul outputs without autoregressive generation or JSON parsing, using the existing LM head with no decision-specific parameters.

2. Soft-Label Learning for Calibrated Decisions (SLCD). A training method that learns decision distributions from soft targets reflecting supervision uncertainty, using a weighted combination of cross-entropy and Brier losses (both strictly proper scoring rules), followed by output-type temperature calibration fit on held-out validation decisions.

3. A dual-system search agent. System 1 (SearchJev) handles schema-defined short decisions; a confidence gate accepts a decision when its maximum calibrated probability is at least a threshold δ, otherwise delegating the judgment to System 2 (Qwen3.5-27B), which retains planning, query generation, and answer composition.

4. SearchDecision-Bench. A benchmark unifying six decision types (routing, rewriting, relevance, sufficiency, navigation, verification) and three output types (Choice, Score, Noul) as state-schema pairs with field-level targets, with in-distribution (ID) and out-of-distribution (OOD) splits for training and evaluation.

Main Findings

• Decision quality improves over same-size autoregressive models. On the six ID decision families of SearchDecision-Bench, SearchJev-0.8B and SearchJev-4B outperform same-size Qwen3.5 AR (JSON) across all six families, with the 4B model leading five. Relative to Jev 1.13, both improve sufficiency, routing, rewriting, and verification and are comparable on relevance, while Jev remains stronger on navigation. Example ID numbers: sufficiency accuracy rises from 49.9 (Qwen3.5-0.8B AR) and 57.7 (Qwen3.5-4B AR) to 89.4 and 96.0 for SearchJev-0.8B/4B.

• Calibration improves on average. Mean ECE falls by 74.4% and 40.7% relative to same-size Qwen3.5 AR (JSON) at 0.8B and 4B respectively. SearchJev-0.8B achieves lower ECE than Jev 1.13 on relevance, sufficiency, routing, and rewriting; SearchJev-4B improves relevance and rewriting and matches Jev on sufficiency. Jev retains the lowest mean ECE and is best calibrated on navigation and verification.

• Decisions are 5.2–5.3× faster. SearchJev at both sizes is 5.2–5.3× faster than same-size Qwen3.5 AR (JSON), with a 7.2–9.6× speedup in median decision latency versus Jev 1.13. Median per-decision latency at batch size 1 is 27.6 ms (SearchJev-0.8B) and 37.1 ms (SearchJev-4B), versus 144.2 ms and 195.2 ms for Qwen3.5-0.8B/4B AR (JSON) and 265.4 ms for Jev 1.13.

• End-to-end search accuracy improves with less work. On BrowseComp-Plus, dual-system agents with SearchJev-0.8B/4B reach 46.0%/54.0% answer accuracy, compared with 45.0% for the System-2-only agent and 46.0% for the Jev 1.13 agent. Relative to System-2-only, they achieve 3.7–4.7× faster search with 3.3–4.8× fewer output tokens.

• Per-question cost drops. On BrowseComp-Plus, the System-2-only agent uses 90.3 System-2 calls, 73.7k output tokens, and 4080 s median active time per question. SearchJev-0.8B uses 62.1 calls, 15.5k output tokens, and 868 s; SearchJev-4B uses 84.6 calls, 22.1k output tokens, and 1102 s. The Jev 1.13 agent uses 76.0 calls, 18.3k tokens, and 1018 s.

• Generalization to held-out datasets is partial but consistent for decision quality. On OOD splits across four decision families, both SearchJev variants beat same-size Qwen3.5 AR (JSON) on routing, rewriting, and relevance, with routing gains of 11.9–12.1 percentage points. Both exceed Jev 1.13 on rewriting, and SearchJev-4B also leads on relevance, while Jev remains stronger on routing and verification. Verification improves for SearchJev-0.8B but declines for SearchJev-4B relative to same-size Qwen3.5 AR (JSON).

• Calibration on held-out data is task-dependent. SearchJev-0.8B lowers ECE on rewriting and relevance relative to same-size Qwen3.5 AR (JSON), while SearchJev-4B improves routing calibration. Both outperform Jev 1.13 on routing calibration, but Jev remains better calibrated on the other three families.

• Efficiency advantage persists out of distribution. On OOD data, both sizes remain 5.3–5.5× faster than same-size Qwen3.5 AR (JSON) and 7.4–9.9× faster than Jev 1.13.

Methodology in Plain English

The approach starts from a simple observation: many search decisions are just picking one option from a small, known set. Instead of asking a language model to write out a JSON answer, SearchJev gives the model a prompt containing the search state, a natural-language question for the decision, and the legal options, each mapped to a single-token label such as A or B. The model's logits at the first output position are then read off for those labels and normalized into a probability distribution over the options. That is the entire decision step — one forward pass, no generated tokens, no parsing.

Training uses soft targets rather than hard labels, because the supervision for search decisions comes from mixed sources of differing reliability (original gold annotations, derived labels, LLM teachers or votes). A weighted sum of cross-entropy and Brier loss fits these target distributions, with lower weights (w_f = 0.5 versus w_f = 1) for LLM-assigned labels, demonstrations, and constructed rewrites. λ_B is set to 0.5. Because both losses are strictly proper scoring rules, the combination is uniquely minimized at the soft target distribution. After training, temperature scaling is fitted per output type (Choice, Score, or Noul) on 5,000 held-out validation decisions, using a fallback across all types when a type has too few validation examples.

For the agent, a confidence gate compares the maximum calibrated probability against a threshold δ. Confident decisions are accepted; uncertain ones are handed to System 2 (Qwen3.5-27B) to resolve. A search controller maps the resolved decisions to actions such as search, open, rerank, or stop, and retrieved evidence updates the state. Models are initialized from Qwen3.5-0.8B/4B and trained with LoRA. For BrowseComp-Plus evaluation, 100 questions are randomly sampled and each agent is run once per question, with BM25 as the retriever, using three decision types: relevance, sufficiency, and verification.

The benchmark itself was built by converting heterogeneous Chinese and English sources — including BANKING77, LCQMC, QReCC, T2Ranking, DuReader-Retrieval, MuSiQue, and VitaminC — into a common serialized state-schema format, with deduplication, removal of conflicting labels, masking of unjudgeable fields, and split isolation by identity keys and query text. Training counts per decision type are 77,627 (routing), 74,318 (rewriting), 153,990 (relevance), 14,652 (sufficiency), 56,981 (navigation), and 47,830 (verification); ID test counts are 26,427, 32,746, 17,581, 1,036, 10,213, and 10,334 respectively. Sufficiency and navigation have no OOD test sets; the paper reports OOD counts of 10,000 for routing, 10,000 for rewriting, 5,976 for relevance, and 20,631 for verification, drawn from held-out datasets including FRAMES, RAGTruth, CANDY, Search Arena, AgentRewardBench, and the QReCC test split.

Why This Matters

Research impact. The paper argues that short search decisions deserve their own model class rather than being folded into the same generative process as planning and answer synthesis. It contributes a reusable benchmark covering six decision families and a training recipe (proper scoring losses on soft labels plus per-output-type temperature scaling) for producing decision probabilities that can be acted upon. It also provides a concrete escalation mechanism — confidence-gated delegation — connecting the calibration literature to agent control.

Real-world applications:

• Search and retrieval systems: relevance, usefulness, and quality judgments can be made in a single forward pass instead of a generative call, cutting per-decision cost in production pipelines.
• Agentic assistants and research tools: multi-step systems that repeatedly decide whether evidence is sufficient or what to search next can offload those calls to a small calibrated model while reserving a larger model for planning and answer writing.
• Verification and fact-checking: the claim-verification and evidence-detection decision types, with calibrated support probabilities, map onto workflows that need to know when a claim is supported, refuted, or insufficiently evidenced.
• Cost- and latency-sensitive deployment: the 3.7–4.7× reduction in median active time and 3.3–4.8× reduction in output tokens on BrowseComp-Plus translate into fewer expensive reasoning-model calls per user query.

Industry relevance. All but one author is affiliated with Huawei Technologies Co., Ltd., with one author at Leiden University, and the models are initialized from Qwen3.5 backbones. The paper reports code and model artifacts alongside the benchmark, and the base models used are open ones, which lowers the barrier to reproducing the decision-level results.

Future Directions

• Improving navigation decisions. Jev 1.13 remains stronger than SearchJev on navigation in the ID results, and both SearchJev variants still lag Jev on routing and verification in the OOD results and on calibration outside routing — an open question is why certain decision families transfer less well.
• Making calibration transfer more reliably. The paper notes that learned decision preferences transfer more consistently than their calibration across evaluated families, and that OOD calibration is task-dependent. How to preserve calibration under distribution shift is left open.
• Extending threshold and escalation tuning. The confidence gate uses a threshold δ (and a retention threshold τ_keep) that controls how many judgments reach System 2; the paper does not report a systematic study of these thresholds, and joint tuning of gate and controller rules is a natural next step.
• Broadening the benchmark. The appendix lists source datasets and label provenance, and the paper's truncation aside, the counts show that sufficiency and navigation have no OOD test sets, so extending held-out coverage to all six families is an obvious extension.

Target Audience

This paper is most useful to researchers and engineers building retrieval-augmented and agentic search systems — particularly those working on decision modeling, reranking, adaptive retrieval, confidence calibration, and selective prediction. It also serves practitioners who need to reduce the latency and token cost of LLM agents in production, and benchmark builders interested in unifying heterogeneous supervision for search decisions. Readers should be comfortable with language model internals (logits, tokenization), standard IR metrics such as NDCG@10, and calibration metrics such as ECE.

Authors’ abstract

Search agents repeatedly make short decisions about relevance, evidence sufficiency, and search actions. Using generative language models for these decisions introduces latency and unreliable confidence. We present SearchJev, a fast and calibrated System-1 model that separates search decisions from System-2 reasoning and generation. Given a search state and a decision schema, SearchJev directly scores legal options without autoregressive output generation. We propose Soft-Label Learning for Calibrated Decisions (SLCD) to learn decision probabilities from uncertain supervision and calibrate their confidence. In a dual-system search agent, SearchJev handles short decisions and delegates uncertain judgments to System 2, which retains planning, query generation, and answer composition. We also introduce SearchDecision-Bench, a benchmark unifying six types of search decisions for training and evaluation. On SearchDecision-Bench, SEARCHJEV improves decision quality over same-size Qwen3.5 autoregressive models, achieves 5.2-5.3 times faster decisions, and reduces average expected calibration error by 41-74%. On BrowseComp-Plus, the dual-system agents achieve a 3.7-4.7 times speedup in active search time while improving answer accuracy from 45% to up to 54%.