TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

AI summary

Overview

Research area: Natural Language Processing / LLM reasoning and tool-using agents (chain-of-thought, latent reasoning, agentic tool use).

Technical level: Advanced — assumes familiarity with chain-of-thought prompting, latent/continuous reasoning tokens, self-distillation, and agentic tool-use loops (e.g., ReAct).

Scope: The paper proposes TAC-CoT (Tool-empowered Adaptive Continuous CoT), a framework that swaps verbalized reasoning traces for short continuous "thought" segments placed between explicit tool calls, observations, and final answers, and reports accuracy and inference-cost results on agentic math, question-answering, and tool-use benchmarks across three backbone models.

What This Paper Is About

Large language model agents usually "think out loud" with explicit chain-of-thought text, which is readable and supports tool calls but is expensive to generate and prone to compounding errors over long traces. Implicit CoT replaces that text with compact continuous tokens, but existing implicit methods spend one fixed block of latent tokens before any output, leaving no point at which the model can call a tool or read a result. TAC-CoT's goal is to make implicit reasoning compatible with agentic tool use by interleaving short latent reasoning segments with explicit textual tool calls, observations, and the final answer, while letting the number of segments — and therefore the total reasoning budget — adapt to each query.

Key Contributions

1. Extending implicit CoT to tool-using agents. The paper reframes implicit CoT so that each latent reasoning segment is followed by a decision point where the model either emits a tool call or produces the answer, making tool use feasible under latent reasoning.

2. An interleaved trajectory formulation. Each explicit reasoning segment of n discrete tokens in a standard tool-use trajectory is replaced by an implicit segment of b (< n) continuous thought tokens, while tool calls, observations, and answers stay in explicit text.

3. Query-adaptive reasoning budgets. Because the tool-call decision is made after every segment, simple queries can terminate after one segment while harder ones continue through multiple tool interactions; b is fixed only locally, and the total budget is k·b with k determined at inference time.

4. A training recipe and objective. The authors automatically construct interleaved tool-use trajectories from reasoning datasets (converting existing executable annotations, or synthesizing trajectories with DeepSeek-V3 and keeping only those matching gold answers), and train with a combined student reconstruction loss, teacher cross-entropy loss, and hidden-state distillation loss at the answer boundary.

Main Findings

• Accuracy gains over Explicit-CoT: TAC-CoT improves over the Explicit-CoT agent in both agentic math and QA; the abstract reports accuracy gains of up to 8.8 points. Averaged over all backbone models, the paper reports math accuracy up +5.8%p and QA +2.4 points versus Explicit-CoT.

• In-domain and out-of-domain transfer: Averaged over backbones, math improves by 6.5%p in-domain and 5.5%p out-of-domain; QA improves by 2.4 points in-domain and 2.5 points out-of-domain.

• Fewer forward passes: The abstract reports reductions of up to 6.8× in model forward passes. Section 3.3 reports up to 2.6× fewer forward passes on math and 6.8× on QA, with larger savings on QA attributed to longer multi-step trajectories.

• Tool use adds a large QA gain: Using tools yields up to a 29.9-point average gain over the no-tool variant on QA while reducing forward passes by 13.3× on Qwen2.5-0.5B.

• Improvement is not just shorter traces: In the no-tool math setting, TokenSkip (which prunes tokens from explicit rationales) underperforms Explicit-CoT, whereas TAC-CoT consistently outperforms the Explicit-CoT agent across all math datasets.

• Better retrieval behavior: On Answer Coverage@k with Qwen2.5-0.5B-Instruct, TAC-CoT reaches HotpotQA 51.8 (@5) and 55.4 (@10) versus Explicit-CoT's 48.8 and 51.7; 2WikiMultihopQA 73.7 and 74.4 versus 70.9 and 71.6; TriviaQA 74.5 and 77.4 versus 72.6 and 75.5. The paper notes TAC-CoT's AC@5 is comparable to Explicit-CoT's AC@10, meaning evidence is retrieved in fewer steps.

• Stronger with multiple tools: On ToolBench under the StableToolBench protocol with Qwen2.5-0.5B-Instruct, TAC-CoT achieves Solvable Win Rates of 51.8% in the single-tool and 58.6% in the multi-tool scenario, with the multi-tool gain statistically significant.

• Deeper replacement helps: Progressively replacing 1, 2, 3, and 4 explicit reasoning segments with implicit ones on Qwen2.5-0.5B raises average accuracy over GSM8K, GSM-Hard, and SVAMP to 41.7, 42.4, 43.4, and 44.9 (versus 41.1 for Explicit-CoT), while forward passes fall from 96.2 to 58.9 (versus 113.1). Replacing all segments gives 45.4 accuracy and 53.7 forward passes.

• More benefit as reasoning depth grows: Against the fixed-budget SIM-CoT baseline on GSM8K subsets grouped by gold reasoning steps, TAC-CoT's advantage grows from 4.6 points on one-step examples to 30.3 points on four-step examples.

• Error accumulation is reduced: When GSM8K examples are grouped by the offset between generated and gold reasoning length, both methods do best under matched lengths and degrade under over-thinking, but TAC-CoT degrades less.

• Wall-clock speedups: On a single NVIDIA L40S GPU, measured speedups are 2.3× on math and 2.4× on QA, with the caveat that latency depends on hardware, implementation, and external API overhead.

Methodology in Plain English

The researchers keep a standard tool-using agent loop but change what sits inside it. Instead of writing out a paragraph of reasoning before each tool call, the model produces a short run of continuous "thought" tokens — vectors fed back into the model, not words — and then stops. At that stopping point it compares the score for "call a tool" against the score for "answer." If the tool option wins, it emits a normal textual tool call, receives the observation, and starts another latent segment. If the answer option wins, that last segment is used to consolidate and the final answer is generated in text. Because the loop repeats, hard questions naturally consume more segments and easy ones fewer.

To train this, the authors need example trajectories that show how much explicit reasoning each latent segment should stand in for. Where a source dataset already contains executable annotations (GSM8K-Aug's arithmetic equations, GSM8K-Aug-NL's natural-language reasoning — 385k samples each), they align the two by question and splice the equations into the reasoning. Where no tool interactions exist (HotpotQA), they have DeepSeek-V3 act as a tool-augmented solver, keeping only trajectories whose final answer matches the gold annotation. That synthesis cost $102.73, with a 56.24% filtering pass rate, and a GPT-5.1 audit of 200 randomly sampled trajectories judged 99% valid.

Training uses self-distillation from the same backbone. An auxiliary decoder, discarded at inference, tries to reconstruct each chunk of the original reasoning text from its aligned continuous token, forcing the latent tokens to carry the omitted reasoning. Standard cross-entropy handles the explicit parts (tool calls, answers), a teacher trained on the full explicit trajectories supplies a reference, and an L1 distillation loss matches student and teacher hidden activations at the colon token before the answer. Experiments use Qwen2.5-0.5B-Instruct, Llama-3.2-1B-Instruct, and Qwen3-4B, with B = 3 continuous tokens per segment (B = 2 for Qwen3-4B) and 1,000 randomly sampled examples per QA dataset.

Why This Matters

Impact on research. The paper identifies a structural mismatch — implicit CoT's single uninterrupted latent block cannot host a tool call — and shows one way to fix it. It also gives evidence that latent reasoning is not merely a compression trick: gains persist without tools, grow with reasoning depth, and appear on out-of-domain benchmarks, which argues against the view that shorter traces simply trade accuracy for speed.

Real-world applications:

• Multi-hop question answering over retrieval APIs, where the model must search, read, and search again without paying for verbose narration at every hop.
• Mathematical problem solving with a calculator tool, where equations can be executed externally while intermediate deliberation stays latent.
• Multi-tool agent workflows (the multi-tool ToolBench setting showed the larger, statistically significant win), such as chaining search, lookup, and computation services.
• Cost-sensitive deployment of small models (0.5B–4B backbones) where forward-pass budget, not model size, is the binding constraint.

Industry relevance. The efficiency story is directly operational: fewer forward passes per query and measured 2.3×–2.4× wall-clock speedups on a single L40S translate into serving cost and latency, and the framework still exposes the explicit tool-call interface that production agent stacks expect.

Future Directions

• Token-level rather than segment-level routing. The authors list this as a limitation: the decision to call a tool or answer happens only at segment boundaries, not per token.
• Longer interaction horizons. The paper flags longer interaction horizons as an open direction for evaluating practical deployment.
• Budget granularity. Appendix C.1 begins an ablation on the number of continuous thought tokens per segment (0, 1, and 3 tokens give average math scores of 42.0, 44.9, and 45.4), but the content is truncated after the first table entry, so the full sweep is not reported here.
• Reducing dependence on a stronger teacher. QA trajectories were synthesized by DeepSeek-V3 and filtered by exact match; the authors acknowledge that exact-match filtering may reject semantically correct answers and underestimate usable yield, leaving open how much teacher quality bounds student quality.

Target Audience

Researchers and engineers working on LLM reasoning efficiency and agentic systems — particularly those already familiar with ReAct-style tool loops, CODI/SIM-CoT-style latent reasoning, and self-distillation training. Practitioners deploying small-to-mid-size models under tight inference budgets will find the forward-pass and wall-clock numbers most actionable; readers new to implicit CoT will need the related-work section to follow the baseline distinctions.

Authors’ abstract

LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.