The Pulse
Researchers Show How Reasoning APIs Could Inflate Bills
A paper on arXiv describes how a provider could add reasoning paths to a reported sequence while making each appear necessary under its stopping rule. In benchmark experiments, the added-answer counts were heavy-tailed, with AIME averages w
AI.info Team · Written with AI · fact-checked ·
On AIME math problems, a simulated provider added an average of 101 reasoning answers per query with Llama-3.2-3B, according to a paper submitted to arXiv on September 24. The median was five, a sharp gap that points to a long tail of cases where the extra answers could be far more numerous. The researchers say those answers could be presented as necessary to reach a majority-vote result—and billed as additional compute.
How extra answers pass a stopping rule
The technique targets self-consistency, a method that samples multiple answers to a question and returns the most common one. Providers can use adaptive stopping rules to decide when they have enough samples to identify that majority with confidence. A customer who sees the reported sequence may check whether it follows the declared rule, but the paper shows how a provider could generate more answers and reorder them so the sequence still appears to stop correctly.
The authors’ algorithm starts with a sequence produced under the stopping rule, then generates extra answers and inserts them before the last answer when the rearrangement keeps the sequence compatible with that rule. In their analysis, a powerful likelihood-ratio audit can test whether a reported sequence looks statistically different from one produced faithfully. Yet the paper finds that, under the tested conditions, extra answers can remain possible even when the audit is designed to keep its false-positive rate at 10% or less.
“In our work, we have shown that under self-consistency, the incentive to report an artificially high number of reasoning paths can be substantial.”
The authors write
A long tail, not a typical bill
The experiments used Llama, Qwen and DeepSeek-R1 distilled models on GSM8K and AIME mathematics benchmarks and GPQA science questions. For AIME, the simulated provider added an average of 101 answers with Llama-3.2-3B, 82 with Qwen2.5-7B and 69.6 with DeepSeek-R1-Distill-Qwen-7B. Their respective medians were five, three and three, illustrating that a smaller number of cases pulled the averages upward.
The result is not a measurement of inflated charges on commercial APIs. The study tests an attack in benchmark experiments, and it reports added answer counts rather than a universal dollar figure. The authors note that real services commonly bill by generated tokens, while reasoning paths can vary in length; a modest increase in paths could therefore translate into a larger charge when those paths are long.
Auditing one request has limits
The paper also distinguishes a single-sequence audit from checks across many requests. It says repeated overcharging may be harder to conceal if auditors aggregate evidence over multiple queries, and suggests testing whether reported sequences behave consistently across those requests. For open-weight models, the authors discuss execution attestations as another possible safeguard, while warning that verifying proprietary systems can be difficult.
That leaves a specific accountability problem for buyers: a disclosed stopping rule and a plausible-looking record may not, by themselves, prove that each billed reasoning path was needed. The study establishes a technical possibility in its tested setup; it does not establish that any provider has used the method against customers.