Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction

AI summary

Overview

Research area: Large language model reasoning, retrieval-augmented generation (RAG), deep search, and multi-turn supervised fine-tuning.

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, chain-of-thought prompting, reinforcement-learning-based search training, and knowledge-graph retrieval, but its central ideas (decomposing questions, deciding when to search) are explained in accessible terms.

Scope (one sentence): The paper introduces Thinker, a hierarchical "deep search" LLM training method that decomposes complex questions into atomic sub-problems, represents each sub-problem in both natural language and a logical function, decides via a knowledge-boundary check whether external retrieval is needed, and is trained by supervised fine-tuning on multi-turn interaction data.

What This Paper Is About

Large language models struggle with complex questions that require investigating, combining, and reasoning over multiple pieces of external information, and existing training-based deep search models rely on end-to-end reinforcement learning that does not supervise the reasoning process itself. As a result, those models often produce interleaved, unstructured solution paths with unclear hierarchy and inconsistent granularity. Thinker aims to make the reasoning process supervisable and verifiable by decomposing complex problems into independently solvable sub-problems, solving each one at depth, and avoiding unnecessary searches when the model already knows the answer.

Key Contributions

1. A hierarchical deep search method for multi-turn interaction that decomposes complex problems breadth-wise into atomic sub-problems, where each sub-problem is dually represented as a natural-language "Step" and an equivalent logical function "Action" so that both plain-text retrievers and structured knowledge retrievers can be used.

2. Four logical function executors — Retrieval, Math, Deduce, and Output — that carry dependency variables (#n, o_n, s_n) between sub-problems so results from earlier sub-problems flow into later ones, improving logical coherence and rigor.

3. A knowledge boundary determination module using a "Generate First, Then Assess" strategy, combining prompt-based confidence assessment with a likelihood-based confidence score defined as the minimum token probability in the extracted answer, adopting the model's internal answer only when both assessments are True.

4. Empirical results showing Thinker outperforms established baselines across seven datasets and multiple model sizes, while achieving performance competitive with baselines using only a few hundred training samples (1 percent of the training data).

Main Findings

• Strong benchmark results at 3B and 7B scale. On Qwen2.5-3B-Instruct, Thinker reaches an average EM of 0.430 versus 0.349 for ReSearch, 0.325 for Search-R1, and 0.317 for ZeroSearch; the paper states that the 3B model achieves an average 7.9 percent gain over the ReSearch baseline. On Qwen2.5-7B-Instruct, Thinker reaches an average EM of 0.452 versus 0.411 for ReSearch, 0.391 for ZeroSearch, and 0.385 for Search-R1.

• Detailed 7B per-dataset numbers. Thinker (Qwen2.5-7B-Instruct) scores NQ 0.450, TriviaQA 0.642, PopQA 0.484, HotpotQA 0.421, 2Wiki 0.469, MuSiQue 0.221, and Bamboogle 0.480.

• Gains over non-retrieval and retrieval-augmented baselines. Using the 7B model as the example, Thinker exceeds Naive Generation and CoT by average margins of 27.1 percent and 34.6 percent, and outperforms Search-o1, IRCoT, and Naive RAG by average margins of 24.6 percent, 22.6 percent, and 14.8 percent respectively.

• Gains over RL-based approaches. Thinker outperforms the prior state-of-the-art ReSearch by an average of 4.1 percent EM across the seven datasets — an average improvement of 4.5 percent on single-hop datasets and 3.9 percent on multi-hop datasets.

• Better logical structure. In a GPT-4-judged evaluation on the HotpotQA test set using four metrics (Logical Hierarchy, Interleaved Solving, Granularity Consistency, Search Efficiency), Thinker scores 0.975 / 0.989 / 0.955 / 0.958 for an Overall of 0.904, versus ReSearch at 0.872 / 0.967 / 0.877 / 0.922 (Overall 0.705) and Search-R1 at 0.813 / 0.955 / 0.852 / 0.903 (Overall 0.638).

• Ablation results. Removing depth solving causes the largest degradation, dropping average EM from 0.452 to 0.415 (a 3.7 percent drop). Removing knowledge boundary determination gives 0.447, removing focusing and reasoning gives 0.445, and removing the logical function gives 0.449. The paper notes that removing knowledge boundary determination has minimal overall performance impact but significantly reduces unnecessary retrievals, and that the logical function is negligible for overall performance but indispensable for the graph-retrieval-enabled framework.

• Sample efficiency. With only 1 percent of training data (a few hundred samples), Thinker reaches an average EM of 0.406, close to the SOTA method ReSearch (average 0.411). Average EM rises to 0.431 at 10 percent, 0.438 at 20 percent, 0.447 at 50 percent, 0.444 at 80 percent, and 0.452 at 100 percent.

• Depth sensitivity. Limiting the model to a single search depresses performance (average EM 0.415 at D=1) and performance stabilizes once the maximum search count is two or greater (0.442 at D=2 and D=3, 0.439 at D=4, 0.452 at D=5). Single-hop datasets are reported as insensitive to retrieval depth.

• Knowledge boundary determination effectiveness. The module reduces search queries by 16.0 percent on TriviaQA and 17.8 percent on Bamboogle. Its accuracy is reported as 96.9 percent on TriviaQA and 97.8 percent on Bamboogle, validated with a dual verification process using Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-32B.

• KAG integration. Within a self-built corpus (1,000 examples each from HotpotQA, Musique, and 2Wiki; 26,990 documents), KAG-Thinker with a hybrid graph retriever improves over Thinker (BGE-M3) by 3.0 percent average EM and 3.8 percent average F1, reaching average EM 0.520 and F1 0.626 versus Thinker's 0.490 and 0.588.

• Reinforcement learning on top of SFT. Applying RL initialized from the best SFT model raises average EM from 0.452 to 0.479. The paper also reports that a more restrictive 8K output-length limit, despite initial underperformance, ultimately outperforms the 16K counterpart.

Methodology in Plain English

The approach has three main pieces.

First, breadth decomposition: instead of trying to solve a complex question in one pass, the model first breaks it into a set of atomic sub-problems that can each be solved independently. Each sub-problem is written twice — once as a natural-language statement (the Step) and once as an equivalent logical function (the Action). There are four logical functions: Retrieval (fetch subject-predicate-object triples), Deduce (extraction, judgment, entailment, or choice reasoning), Math (numerical or statistical calculation), and Output (emit the final answer). Because the logical functions bind named variables, an answer produced by one sub-problem can be passed as a parameter into a later sub-problem.

Second, knowledge boundary determination: before searching for a sub-problem, the model first tries to answer it from its own parametric knowledge. That answer is checked two ways — a prompt-based introspective assessment and a likelihood-based score computed as the minimum probability among the tokens of the extracted answer, compared against a threshold. Only when both checks say "True" does the model accept its internal answer and skip retrieval.

Third, depth solving: for sub-problems that do need retrieval, the model iteratively searches, focuses on the retrieved material, and reasons, repeating until it either hits a maximum number of turns or emits an answer between designated answer tokens.

Training uses supervised fine-tuning over multi-turn interaction sequences. Each whole sequence (system, then alternating user and assistant turns) is concatenated and cross-entropy loss is computed only over the assistant's response tokens, then averaged. The paper argues this is computationally efficient because it supports parallel processing and it supervises every assistant turn. Training data came from NQ and HotpotQA, producing 71K total samples, built from a 59K logical-form training set and an 18K deep-search training set generated with Qwen2.5-72B-Instruct and filtered through a data evaluation framework. The retriever is E5-base-v2 with December 2018 Wikipedia as the knowledge base, with indexing and embedding pre-processed using FlashRAG; ReSearch retrieves the top 5 documents per question while all other methods retrieve the top 3.

Why This Matters

Impact on research. The paper argues that end-to-end reinforcement learning for deep search leaves the intermediate reasoning process unsupervised, which makes logical rigor hard to guarantee. Thinker shows that supervised, hierarchical, structured reasoning — with explicit decomposition, dual natural-language/logical representation, and a search-skipping confidence check — can match or beat RL-trained search models. It also reports that adding RL on top of the SFT model yields further gains, suggesting a complementary rather than competing relationship between the two training paradigms.

Real-world applications (domains where the paper says logical rigor is paramount and where its systematic reasoning framework is well-suited):

• Medical diagnostics, discussed in the paper's appendix.
• Legal compliance, where the paper includes an example of calculating the amount involved in a credit card fraud charge by retrieving legal provisions, extracting relevant amounts, and summing them.
• Financial risk assessment.
• Other general deep-search question answering over web pages and structured knowledge bases, where the dual representation lets the same model query either plain-text retrievers or graph-based knowledge bases.

Industry relevance. The authors are affiliated with Ant Group, Tongji University, and Zhejiang University, and the code is released at https://github.com/OpenSPG/KAG-Thinker. The method's sample efficiency — competitive performance with a few hundred training samples — and its compatibility with the KAG framework make it practical for organizations that want controllable, auditable reasoning traces rather than unpredictable RL rollouts, especially in regulated professional domains.

Future Directions

• Scaling the RL stage. The paper applies reinforcement learning to the best SFT model and reports a gain from 0.452 to 0.479 average EM; this line of work is presented as exploring the model's potential rather than a finished result, leaving room for further RL investigation.

• Larger and different base models. The paper refers readers to appendix sections for experiments involving larger models and different base models; those details are not included in the provided content, so their outcomes are not reported here.

• Broader domain deployment. The paper positions the framework as suited to medicine, law, and finance, and its data synthesis covers general and medical domains. Extending the data construction and evaluation to other high-stakes domains is a natural next step.

• Better leveraging structured knowledge bases. The dual representation was designed to let the model use high-quality structured knowledge bases, and the KAG-Thinker results show additional gains from a hybrid graph retriever plus native Math and Deduce support. Extending this beyond the evaluation described remains an open direction.

Target Audience

This paper is most useful to researchers and engineers working on retrieval-augmented generation, agentic search, and LLM reasoning training — particularly those interested in supervised alternatives or complements to reinforcement learning for tool-using models. It also suits practitioners building question-answering systems over knowledge graphs or web corpora who need controllable, auditable intermediate reasoning steps, and readers who want a concrete example of how hierarchical decomposition, structured logical functions, and confidence-based search-skipping can be combined in a single trained model.

Authors’ abstract

Efficient retrieval of external knowledge bases and web pages is crucial for enhancing the reasoning abilities of LLMs. Previous works on training LLMs to leverage external retrievers for solving complex problems have predominantly employed end-to-end reinforcement learning. However, these approaches neglect supervision over the reasoning process, making it difficult to guarantee logical coherence and rigor. To address these limitations, we propose Thinker, a hierarchical thinking model for deep search through multi-turn interaction, making the reasoning process supervisable and verifiable. It decomposes complex problems into independently solvable sub-problems, each dually represented in both natural language and an equivalent logical function to support knowledge base and web searches. Concurrently, dependencies between sub-problems are passed as parameters via these logical functions, enhancing the logical coherence of the problem-solving process. To avoid unnecessary external searches, we perform knowledge boundary determination to check if a sub-problem is within the LLM's intrinsic knowledge, allowing it to answer directly. Experimental results indicate that with as few as several hundred training samples, the performance of Thinker is competitive with established baselines. Furthermore, when scaled to the full training set, Thinker significantly outperforms these methods across various datasets and model sizes. The source code is available at https://github.com/OpenSPG/KAG-Thinker.