Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generation

AI summary

Overview

Research area: Large language model (LLM) prompting strategies for automated code generation, with a focus on adaptive inference-time routing and token-cost efficiency.

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought (CoT) prompting, Pass@k evaluation, sampling parameters, and token-budget accounting, but its central idea — deciding when to reason and when not to — is explained in accessible terms.

One-sentence scope: The paper introduces RoutingGen, a difficulty-aware routing framework that sends simple programming problems to few-shot generation and complex ones to a new two-stage "Intention Chain-of-Thought" (ICoT) strategy, and reports state-of-the-art Pass@1 on most settings across three models and six benchmarks while cutting token usage by 46.37% on average.

What This Paper Is About

Existing CoT-style prompting methods for code generation are applied uniformly to every problem, which causes overthinking on easy tasks (disorganized logic and lower accuracy) and, separately, never explicitly models a task's intention — the core algorithmic design and efficiency requirement — so models focus on surface-level structure instead of the global objective. The authors' goal is to conserve reasoning effort by activating structured reasoning only when a problem is actually hard, and to make that structured reasoning capture algorithmic intent rather than procedural surface form. RoutingGen does this by classifying each problem as Simple or Complex and routing it to either few-shot prompting or the authors' new ICoT strategy.

Key Contributions

1. Problem identification. The authors focus on overthinking caused by uniformly applying structured prompting at the functional code level, and identify a core limitation of existing methods: the lack of intention abstraction in code generation, such as algorithmic design and efficiency modeling.
2. The RoutingGen framework. A difficulty-aware dynamic routing framework that uses a classifier to assign each problem to either a low-cost few-shot generation strategy (Simple) or a structured reasoning strategy (Complex).
3. Intention Chain-of-Thought (ICoT). A new structured reasoning strategy composed of a Specification element (input-output constraints) and an Idea element (core algorithmic logic and estimated time complexity), generated in one decoding pass and then used to condition code generation.
4. Empirical validation. Across three models and six benchmarks, RoutingGen achieves state-of-the-art performance in most settings while substantially reducing token usage, and ICoT consistently outperforms six prompting baselines on challenging benchmarks.

Main Findings

• Broad state-of-the-art Pass@1. RoutingGen achieves state-of-the-art Pass@1 performance across most models and benchmarks. With DeepSeek-V3-671B it reaches 91.83% on HumanEval and 90.21% on MBPP-sanitized.
• Large token savings. RoutingGen reduces total token usage by 46.37% on average across settings. Average per-benchmark reductions relative to SCoT are 46.57% (Qwen2.5-Coder-3B-Instruct), 47.67% (DeepSeek-Coder-6.7B-Instruct), and 44.54% (DeepSeek-V3-671B); relative to ICoT they are 47.16%, 46.23%, and 40.59% respectively. All reported token usage includes both routing tokens and inference tokens.
• Efficiency comes from routing, not from skimping. On MBPP-sanitized with DeepSeek-Coder-6.7B-Instruct, RoutingGen routes most tasks to few-shot generation and achieves a 59.22% token reduction versus SCoT while improving accuracy by 4.60 percentage points (72.30% vs. 67.70%).
• Hard problems still get full reasoning. On HumanEval, where the classifier labels 67.68% of tasks Complex, RoutingGen achieves 73.51%, comparable to ICoT's 73.14%, with a moderate 24.21% token reduction — reflecting the necessary computational investment.
• Structured prompting can hurt easy benchmarks. On MBPP-sanitized and MBPP-ET, several structured prompting methods perform worse than simple zero-shot or few-shot baselines. On MBPP-ET, Self-CoT with DeepSeek-Coder-6.7B-Instruct shows the largest drop, with Pass@1 falling 22.99% below the zero-shot baseline.
• Where routing lands. Most MBPP-sanitized and McEval tasks are classified Simple (64.40% and 60.00%), while HumanEval and OpenEval have substantially lower simple-task proportions, both below 36%.
• ICoT is a strong standalone strategy. With DeepSeek-Coder-6.7B-Instruct, ICoT reaches 38.23% Pass@1 on OpenEval, a 128.38% relative improvement over zero-shot. With Qwen2.5-Coder-3B-Instruct, ICoT shows a 45.15% relative improvement on McEval. With DeepSeek-V3-671B, ICoT reaches 82.68% on HumanEval-ET (a 6.10% relative gain) and 67.20% on McEval (a 102.41% relative gain).
• Robustness to the difficulty classifier. Swapping Qwen3-8B for GPT-4o produces conflicting difficulty labels on 22.56% of HumanEval, 23.42% of MBPP-sanitized, and 8.43% of OpenEval tasks, yet RoutingGen consistently outperforms all baselines under both classifiers (72.30% and 72.79% Pass@1 on MBPP-sanitized respectively). Self-routing — where the generation model classifies its own difficulty — yields broadly comparable results.
• Both ICoT components matter, but not on easy tasks. With DeepSeek-V3-671B, removing the Specification stage on HumanEval reduces Pass@1 from 92.07% to 88.17%, and removing the Idea stage on McEval drops it from 67.20% to 59.20%. The variant with no intermediate reasoning performs best on the simpler MBPP-sanitized and MBPP-ET datasets, supporting the overthinking finding.
• Beyond Python. On HumanEval-X (C++ subset), RoutingGen with self-routing and DeepSeek-V3-671B achieves 82.44% Pass@1 and ICoT 81.46%, versus 53.66% zero-shot and 81.95% few-shot. On LiveCodeBench, RoutingGen achieves 44.50% and ICoT 45.10%, versus 33.00% zero-shot and 43.35% for SCoT.

Methodology in Plain English

The design borrows from dual-process theories of human cognition: System 1 handles easy, familiar problems quickly, while System 2 engages deliberate structured reasoning when a problem is hard. The authors translate this into a routing decision for code generation.

1. Classify the problem. A separately chosen classifier model, Qwen3-8B, receives the problem under a specially designed prompt and outputs one of two labels — Simple or Complex — plus a short textual rationale for its decision.
2. Simple path. RoutingGen uses few-shot prompting: a low-cost template with a few example question-code pairs, from which the model samples n candidate code solutions. The default number of candidates is 20 (5 for DeepSeek-V3 due to API constraints).
3. Complex path (ICoT). In stage one, the model samples n candidate "ICoT" instances, each a structured pair consisting of a Specification (what the inputs and outputs are) and an Idea (the core algorithmic logic plus an estimate of time complexity), generated jointly in a single decoding pass using nucleus sampling. In stage two, each ICoT instance is used with the original problem to deterministically generate one code solution via greedy decoding.
4. The motivating example. For a task asking whether two strings contain the same set of characters, the Specification stage grounds inputs (two strings s0 and s1) and the output (a boolean). The Idea stage then states the intention abstraction — convert both strings to sets, then compare them — and notes the O(m+n) time complexity, steering the model to the concise solution return set(s0) == set(s1) instead of a surface-level procedural loop as in the SCoT baseline.
5. Measurement. Effectiveness is measured with the unbiased Pass@k estimator; efficiency is measured as total tokens per problem, summing input and output tokens, including the tokens consumed by the routing module. Sampling follows common practice: 20 candidates, temperature 0.8 and top-p 0.95 with a 300-token cap for single-stage methods; 20 reasoning chains at temperature 0.8 followed by deterministic generation at temperature 0 for multi-stage methods; Self-CoT is allowed 600 tokens for longer reasoning chains.
6. Evaluation setup. Six benchmarks: HumanEval (164 Python problems), MBPP-sanitized (427 verified tasks, three tests each), HumanEval-ET and MBPP-ET (extended versions with around 100 edge-case tests per problem), OpenEval (178 challenging problems drawn from AVATAR), and McEval's Python subset (50 problems). Baselines are zero-shot, few-shot, Self-CoT, Zero-shot-CoT, Self-planning, and SCoT. The paper notes that discrepancies between its baseline numbers and official reports are expected because it uses standardized instructions for fair cross-model comparison rather than model-specific optimized prompts.

Why This Matters

Impact on research. The paper reframes prompting strategy selection as a resource-allocation problem rather than a "more reasoning is always better" assumption. It provides evidence that elaborate prompt structures can be counterproductive on easy problems, and it introduces intention-level abstraction (algorithmic logic plus complexity) as a distinct target for structured prompting, separate from procedural step decomposition. It also opens a line of work on difficulty-aware routing between reasoning strategies rather than between demonstrations or models.

Real-world applications:

• IDE and editor coding assistants, where routing easy completions away from expensive reasoning directly lowers latency and serving cost.
• Competitive-programming and algorithmic tutoring tools, where ICoT's explicit complexity estimate and algorithmic idea are themselves useful artifacts for learners.
• Code translation and migration, since OpenEval's problems come from the AVATAR code translation dataset and the framework is evaluated on multilingual benchmarks.
• Multilingual and cross-language development, supported by results on McEval's Python and multilingual subsets and the C++ subset of HumanEval-X.

Industry relevance. Token usage is a direct proxy for inference cost. A 46.37% average reduction in total tokens across settings — with reported per-model averages ranging from 40.59% to 47.67% relative to ICoT — is a meaningful serving-efficiency claim for teams deploying code models. The robustness checks matter for deployment too: results hold under two different difficulty classifiers and under self-routing, meaning a pipeline does not depend on one specific classifier.

Future Directions

• Richer difficulty granularity. The current label space is binary (Simple / Complex). The paper does not explore finer-grained levels of difficulty or per-problem budget allocation.
• Better and cheaper difficulty estimation. Qwen3-8B and GPT-4o disagree on 8.43% to 23.42% of tasks depending on benchmark. The paper does not report a study of which classifier is more accurate, nor of learned routing policies beyond the self-routing variant.
• Generalizing intention abstraction beyond code. ICoT's specification-and-idea decomposition is motivated in part by progress in mathematical reasoning with LLMs, but the paper evaluates it only on code generation benchmarks. Whether it transfers to other structured reasoning tasks is not reported.
• Scaling the evaluation. Results span three models and six benchmarks plus HumanEval-X (C++) and LiveCodeBench; evaluations on additional programming languages, larger multilingual subsets, and other model families are not reported.

Target Audience

This paper is most useful to researchers and engineers working on LLM code generation, prompting strategy design, and inference-cost optimization — particularly those building production code assistants who need to balance accuracy against token spend. It is also relevant to software engineering researchers studying how models map natural language requirements to algorithmic intent, and to practitioners interested in adaptive inference pipelines that select reasoning depth per request. Readers without background in prompting methods and Pass@k evaluation will need to consult the cited CoT, Self-Consistency, and self-planning papers first.

Authors’ abstract

Large language models (LLMs) exhibit strong generative capabilities and have shown great potential in code generation. Existing chain-of-thought (CoT) prompting methods enhance model reasoning by eliciting intermediate steps, but suffer from two major limitations: First, their uniform application tends to induce overthinking on simple tasks. Second, they lack intention abstraction in code generation, such as explicitly modeling core algorithmic design and efficiency, leading models to focus on surface-level structures while neglecting the global problem objective. Inspired by the cognitive economy principle of engaging structured reasoning only when necessary to conserve cognitive resources, we propose RoutingGen, a novel difficulty-aware routing framework that dynamically adapts prompting strategies for code generation. For simple tasks, it adopts few-shot prompting; for more complex ones, it invokes a structured reasoning strategy, termed Intention Chain-of-Thought (ICoT), which we introduce to guide the model in capturing task intention, such as the core algorithmic logic and its time complexity. Experiments across three models and six standard code generation benchmarks show that RoutingGen achieves state-of-the-art performance in most settings, while reducing total token usage by 46.37% on average across settings. Furthermore, ICoT outperforms six existing prompting baselines on challenging benchmarks.