On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models

AI summary

Overview

Research area: Natural Language Processing — specifically LLM reasoning behavior, multi-task prompting, and inference efficiency.

Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with reasoning models, thinking tokens, chain-of-thought traces, context windows, and batching.

Scope: An empirical study showing that composing several independently solvable tasks into one prompt makes language models silently cut per-task reasoning according to a power law, causing failures on tasks that needed long reasoning traces in isolation — and a batching strategy that turns this behavior into a 43% token saving.

What This Paper Is About

Benchmarks usually test a model one task at a time, but users routinely pack several tasks into a single prompt. The authors ask a narrow, controlled question: if a model solves a task correctly on its own, does it still solve that same task when other tasks are concatenated alongside it, with the task text left unchanged? They find it often does not, and they trace the failure not to error accumulation or context limits but to an implicit, per-task "thinking budget" that shrinks as more tasks share the prompt.

Key Contributions

1. Composition alone causes degradation. Concatenating tasks that a model already solved in three out of three isolated attempts produces substantial accuracy drops across open-weight models and the frontier model Claude Opus 5, on both mathematical and linguistic datasets, and the drop is larger than the constituent tasks' isolated error rates would predict.

2. An implicit per-task thinking budget is identified. Under composition, models spend fewer thinking tokens per task even with no explicit token limit and even when the sequence is far inside the context window. The per-task token ratio follows an approximate power law in composition size k, with a median fitted exponent of m = −0.54 (range −0.88 to −0.04, median R² 0.91).

3. "Fragile" tasks are defined and shown to be stable within a model. Degradation concentrates on tasks with long isolated reasoning traces. Which tasks are fragile is consistent within a model across composition sizes (disattenuated Pearson correlation 0.81 ± 0.04) but largely inconsistent across different models (0.24 ± 0.03), except for models sharing a training recipe (Qwen3 8B and 14B: 0.56 ± 0.09).

4. Fragility-aware batching. Grouping non-fragile tasks into larger batches and fragile tasks into smaller ones reduces token usage by 43% on average with accuracy held constant (+0.6 pp), and reduces generated tokens by 24% on GPQA-Diamond with Claude Opus 5 without degrading performance. A deployable variant that predicts fragility with a linear model over embeddings saves 31.0% of tokens while limiting accuracy loss to at most 0.5 pp on every dataset.

Main Findings

• Degradation is not explained by error accumulation. Because every composed task was first verified as solved in three out of three isolated attempts, and because the observed accuracy drop exceeds what independent per-task error rates would predict, the authors argue accumulation is only part of the story.

• Context length is not the cause. Most responses use under 25% of the context window, and in most datasets fewer than 1% of responses hit the limit; open-weight models used a context limit of at least 131,072 tokens. Composition sizes were k ∈ {3, 6, 10, 20}, with k = 20 omitted on GPQA-Diamond and AIME because of context overflow.

• Reasoning shrinks uniformly, following a power law. The token ratio r_i(k) = T_i(k)/T_i(1) decreases approximately linearly in log k, so each task's reasoning shrinks by roughly k^m. Long-reasoning tasks do not lose a larger share of their tokens; the reduction is roughly proportional across tasks, with no systematic dependence on isolated trace length. Almost all tasks decrease monotonically with k, the one exception being R1 Qwen 7B on MATH-500.

• Fragility tracks how much slack a task has. Short isolated traces appear to contain reasoning beyond what the task needs, so they absorb the same proportional cut; long traces sit closer to the minimum required and fall below it. Tasks in the top quintile of isolated trace length degrade far more than tasks in the low (≤20%) or medium (20–80%) bins.

• The degradation varies a lot by model. R1-distilled models degrade the most, Qwen3 models moderately less. Claude Opus 5 loses about 7 pp at k = 10 on GPQA-Diamond on tasks it had solved in all three isolated runs.

• Two datasets show no degradation. OneWordQA and StrategyQA, whose tasks require little reasoning, do not degrade — consistent with the slack explanation.

• Fragility is predictable enough to act on. With measured fragility, batching nearly eliminates degradation while saving tokens; on GPQA-Diamond, MATH-500 and MMLU-Redux the average accuracy change is +0.2, −0.2 and −0.2 pp respectively, versus −10.5, −5.7 and −7.5 pp for random fixed-size batches of six. On OneWordQA and StrategyQA, fragility-aware batching matches random batching at 68–74% token savings; on the remaining five datasets it preserves accuracy while saving roughly 32%.

• The simple predictor transfers only partly. The linear predictor (Qwen3 Embedding 8B features, five-fold cross-fitting, labels from 10 trials per composition size) works best on tasks with short, plain descriptions such as Integrals, GSM8K and OneWordQA, and saves only 8.9% on GPQA-Diamond, 4.3% on MATH-500 and 6.4% on MMLU-Redux on average.

• Post-training changes the damage, not the budgeting. Across OLMo 3 7B's SFT, DPO and RLVR stages, token shrinkage under composition is very similar, but at comparable token ratios the final RLVR stage loses substantially less accuracy. SFT and DPO degrade similarly and fail on similar tasks (disattenuated ρ = 1.03 ± 0.35); RLVR's fragile set overlaps less (ρ = 0.69 ± 0.23 vs. SFT, 0.89 ± 0.26 vs. DPO).

• Isolation wastes compute. Non-fragile tasks receive more reasoning in isolation than they need, so the same mechanism that breaks fragile tasks under composition also means models overspend on easy tasks when run one at a time.

Methodology in Plain English

The design deliberately removes confounds. For each model, the authors first build a pool of "confidently solved" tasks — items the model gets right in three out of three independent attempts — so capability is established before composition is introduced. Tasks are drawn from eight datasets: two the authors construct (1,000 indefinite integrals with randomized coefficients, verified symbolically, and OneWordQA, factual questions with single-word or numeric answers) plus GPQA-Diamond, GSM8K, MATH-500, MMLU-Redux, StrategyQA and AIME. The task text is then left untouched and simply concatenated into a single prompt using a fixed per-dataset template asking for all k answers in order; the single response is split into k answers and graded. Accuracy is estimated from five independent trials, with isolated accuracies taken from runs separate from the selection runs.

To see where the reasoning goes, each task in the prompt is prefixed with a unique tag (Q01, Q02, …) that does not appear in the task text, letting the authors attribute segments of the thinking trace to individual tasks. That yields a per-task token count under composition, T_i(k), which is compared with the isolated count T_i(1) via the ratio r_i(k). Fitting a line through the origin on log-log axes gives the power-law exponent per model and dataset. Tasks are then binned by the quantile of their isolated trace length (Low ≤20%, Medium 20–80%, High >80%) to connect token reduction to accuracy loss.

Fragility is measured as a per-task accuracy drop and compared across conditions using Pearson correlation with Spearman's correction for attenuation to counter noisy accuracy estimates. For the batching intervention, each task is assigned a composition size by maximizing expected accuracy drop plus λ times the token saving, with λ chosen by grid search on held-out splits under repeated nested five-fold cross-validation so that the mean expected drop stays within a 0.5% budget; tasks are then greedily packed into as few compositions as possible respecting their assigned size. The deployable version replaces measured fragility with a per-model linear predictor trained on embeddings, evaluated with five-fold cross-fitting. Everything is black-box: no fine-tuning and no access to internal states.

Why This Matters

Impact on research. The paper reframes a scattered set of observations — failures on multi-instruction prompts, sensitivity of deep-reasoning tasks to irrelevant context, self-compression of reasoning traces under batching, token savings from batching — under one mechanism: an implicit per-task thinking budget whose allocation does not generalize to composed prompts. It also gives a concrete, measurable task property (isolated reasoning-trace length) that predicts failure, and shows single-task benchmark scores can overstate robustness once capabilities saturate.

Real-world applications:

• Batch inference cost reduction. Routing easy, non-fragile items into large batches and hard items into small ones cuts generated tokens substantially with no measured accuracy cost — useful for large offline evaluation or data-processing jobs.
• Agentic workflows. The authors connect their result to "context rot": agents degrade long before the context window fills, and a high-effort step can be silently misbudgeted, propagating errors. Their practical suggestion is to keep agent context lean and hand reasoning-intensive steps to subagents that start from a fresh context.
• Proprietary model deployment. Because the intervention requires only prompt restructuring, it applies directly to closed frontier models such as Claude Opus 5 (24% fewer tokens on GPQA-Diamond at a 0.4 pp accuracy cost when fragility is measured).
• Benchmark design. Results argue for evaluating models on composed prompts, not just isolated items, since models that score well one task at a time can fail on the same tasks in a shared prompt.

Industry relevance. Token cost is a direct operating expense for anyone serving reasoning models. The finding that much isolated reasoning is redundant on easy tasks — and that providers effectively already compress it when tasks are batched — gives teams a black-box lever for cutting spend without retraining or accessing model internals.

Future Directions

• What causes the reduction is unknown. The authors state outright that they do not know what drives the per-task reasoning shrinkage, only that it follows a power law and is stable per model.
• Richer compositions. The study is limited to independent, same-dataset compositions. Whether the effect extends to mixed-domain, dependent tasks and to tasks embedded in conversation history or larger tasks (where prior work also observes shorter traces) is open.
• Reasoning-length analysis for proprietary models. Detailed token-level attribution was restricted to open-weight models; extending it to closed models remains future work.
• Better fragility predictors. The current linear predictor is deliberately simple and trained on few examples; scaling it up could make token savings reach unseen real-world tasks.
• Training budget management directly. The OLMo 3 7B results suggest post-training (particularly RLVR) mitigates the accuracy damage without fixing the misallocation, motivating explicit training on composed tasks or budget management inside RL pipelines.

Target Audience

Researchers and engineers working on LLM reasoning, inference cost optimization, and multi-task or agentic prompting. It is most useful to practitioners who serve reasoning models at scale and want a training-free way to cut token usage, and to evaluation researchers interested in why single-task benchmarks overstate robustness. Readers need basic familiarity with reasoning models and token accounting but not with the paper's optimization details.

Authors’ abstract

Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open-weight models and the frontier model Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more $\textit{fragile}$ under composition, i.e., models make more errors on prompts containing these tasks. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. However, by grouping tasks according to their fragility, the implicit budgeting can be manipulated to reduce token usage by $43$\% across a range of math and linguistic tasks while also preserving accuracy.