Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute

AI summary

Overview

Research area: Inference-time scaling for large language models; specifically a matched-compute comparison of input-side versus output-side diversity.

Technical level: Intermediate. The methods are conceptually simple and no model training is involved, but the paper assumes familiarity with chain-of-thought prompting, self-consistency, majority voting, and cost-accuracy Pareto analysis.

Scope: A systematic benchmark of Test-Time Augmentation (semantic rephrasing, lexical perturbation, visual transformation) against chain-of-thought prompting and self-consistency across six datasets, evaluated on Claude 4.5 Haiku, Sonnet, and Opus.

What This Paper Is About

Test-time scaling techniques such as self-consistency improve LLM accuracy by spending more inference compute, but that compute costs money and adds latency, so the real question is how much accuracy each extra call buys. Self-consistency spends the whole budget on the output side, sampling repeated reasoning paths from one fixed prompt. This paper asks whether spending part of the same budget on the input side, by generating paraphrases, character noise, or image transforms and then majority-voting across them, converts compute into accuracy more efficiently.

Key Contributions

1. A matched-compute comparison of input-side diversity (TTA) against output-side diversity (self-consistency) and single-call chain-of-thought prompting, measured on both accuracy and cost per additional LLM call across six benchmarks.
2. Evidence that semantic rephrasing outperforms the strongest baseline on five of six benchmarks and Pareto-dominates self-consistency on cost-effectiveness, delivering roughly 1.8x more accuracy per dollar than self-consistency, with statistically significant gains (p < 0.01 over CoT, p < 0.05 over self-consistency).
3. Ablation studies on the number of augmentations, cost-accuracy trade-offs, augmentation modality for multi-modal tasks, and base model scaling, yielding practical deployment guidance such as starting at k = 4 for semantic TTA.
4. Identification of the mid-tier model regime as where TTA is most cost-effective, with a public implementation released at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.

Main Findings

• Semantic TTA wins on accuracy: Across all six benchmarks, semantic TTA delivers the largest average gain of 1.8 percentage points (pp) over single-call CoT prompting and outperforms self-consistency on five of six benchmarks. The gap over self-consistency is largest on Math500 (+1.52 pp) and MMMLU (+1.50 pp).

• Lexical TTA is weaker but nearly free: Lexical TTA achieves smaller gains (1.2 pp on average), suggesting character-level perturbations introduce noise that partially offsets the benefits of aggregation. It requires no additional LLM call for augmentation.

• Statistical significance: Paired t-tests on mean accuracy gains across all six datasets show semantic TTA significantly beats single-call CoT (p < 0.01) and self-consistency (p < 0.05). Lexical TTA also beats single-call CoT (p < 0.01), as does self-consistency (p < 0.05), but lexical TTA and self-consistency are not significantly different from each other at the 5% level.

• Bootstrap confirmation: A non-parametric paired bootstrap over pooled per-question gains (n = 2,400, 5,000 resamples) gives semantic TTA a mean gain of +1.79 pp with a 95% CI of [+0.88, +2.71] pp (t = 3.86); lexical TTA +1.25 pp [+0.25, +2.21] (t = 2.47); self-consistency +0.92 pp [+0.04, +1.79] (t = 2.06).

• Cost-effectiveness: Semantic TTA achieves the highest accuracy gain per dollar spent and per additional LLM call, delivering roughly 1.8x more accuracy per dollar than self-consistency despite its higher per-call cost. The paper notes that gains of 1–2 pp may not justify a 2–6x cost increase in all settings, and that TTA is most valuable when baseline accuracy is moderate (40–80%).

• Fewer augmentations needed: Semantic TTA reaches its peak with fewer augmentations on average (k = 4.33) versus lexical TTA (5.33) and self-consistency (4.67). On Math500 with k up to 10, semantic TTA peaks at k = 5 with diminishing returns thereafter, while self-consistency continues to improve up to k = 10 and nearly matches semantic TTA at that point. Fluctuations from stochastic paraphrase generation and tie-breaking are within 1 pp.

• Text beats image augmentation on multi-modal tasks: On MMMU, text-based semantic TTA reaches 68.09% at k = 2, ahead of visual TTA at 67.59% (k = 6), self-consistency at 67.09% (k = 6), lexical TTA at 67.84% (k = 6), and single-call CoT at 66.08%. Combining text and image adjustments hurts: the alternating variant scores 66.33% (k = 2) and the combined variant 65.08% (k = 4), below single-call CoT.

• Bigger gains on smaller models: On MMMLU, semantic TTA gains over CoT are +2.75 pp for Claude 4.5 Haiku, +2.00 pp for Sonnet, and +0.25 pp for Opus. Lexical TTA improves Haiku and Sonnet but degrades Opus (87.00% versus the 88.25% baseline).

• TTA does not replace a bigger model: Haiku with semantic TTA (82.50%) does not reach Sonnet's single-call accuracy (85.50%), and Sonnet with TTA (87.50%) remains below Opus's single-call accuracy (88.25%). TTA is framed as a compute-efficiency tool for the mid-tier regime rather than a substitute for upgrading.

• Variance reduction, with one exception: All aggregation methods reduce prediction variance relative to single-call CoT on most benchmarks. The exception is HLE, where semantic TTA increases standard deviation from 17.06% to 19.00%, attributed to HLE's very low baseline accuracy of 3%.

Methodology in Plain English

The researchers take an input question and generate k altered copies of it. For semantic TTA, an LLM is prompted once to produce k paraphrases that preserve meaning, answer choices, formatting instructions, and image references. For lexical TTA, three kinds of character-level corruption are applied (random character swaps within words, character insertions and deletions, and injected typos) at a 5% probability per word with a maximum of 10 perturbations per question. For visual TTA on multi-modal inputs, images get small rotations of plus or minus 3 degrees and brightness or contrast shifts of plus or minus 5%.

Each variant is answered independently by the model, and the answers are combined by majority voting, with ties broken at random. Self-consistency instead feeds the same unmodified prompt k times at temperature T = 0.75 and majority-votes the sampled reasoning paths. Because semantic TTA also samples at T = 0.75, the comparison at matched k isolates the marginal contribution of varying the input rather than the reasoning path.

Evaluation spans six datasets: MMLU, MMMLU, MMMU, HLE, Math500, and IMDB Reviews. A randomly sampled subset of 400 examples from each dataset is used. The primary model is Claude 4.5 Haiku, with Sonnet and Opus added for a model-size ablation on MMMLU. Main experiments use k in {2, 4, 6}, with extended runs from k = 1 to 10 for ablations. The reported k for each method and dataset is selected on a held-out sample disjoint from the evaluation subset. Experiments run on Amazon Bedrock with a fixed random seed, with token costs recorded as of June 2026 using on-demand pricing.

Why This Matters

Impact on research: The paper reframes inference-time scaling as an efficiency question rather than purely an accuracy question, and shows that input-side diversity is a distinct and underexplored axis that can be layered on top of existing output-side methods. It also unifies prior task-specific rephrasing work (math, classification, factual probing) under one framework and benchmarks it at matched compute.

Real-world applications:

• Deploying mid-tier LLMs in production where a frontier model is too expensive, and squeezing extra accuracy from a fixed base model via a training-free wrapper.
• Multi-modal question answering pipelines where text paraphrasing is worth applying but image-level augmentation is not.
• High-volume classification or multiple-choice workloads where the near-free lexical variant is an acceptable fallback when even small augmentation overhead is undesirable.
• Cost-sensitive inference budgeting, where the accuracy-gain-per-dollar metric directly informs how many inference calls to allocate per query.

Industry relevance: The work comes from Amazon Web Services, uses Amazon Bedrock and on-demand token pricing, and ships a public implementation for Bedrock, making it directly actionable for teams running LLM inference at scale.

Future Directions

1. Extending TTA to open-ended generation such as summarization or translation, which would require aggregation mechanisms other than majority voting since there is no well-defined answer equivalence.
2. Verifying the findings on open-weight models or substantially different model families, since the evidence is limited to the Claude Haiku, Sonnet, and Opus family.
3. Decoupling the rephrasing model from the answering model, which the paper notes may yield further gains, and tuning the balance between input-side and output-side diversity rather than treating TTA as an extension of self-consistency.
4. Testing stronger or structurally different visual augmentations, since the conclusion that text beats images is scoped to the mild photometric and geometric transforms used here.

Target Audience

Practitioners deploying LLMs under cost or latency constraints, applied machine learning engineers choosing among inference-time scaling techniques, and researchers studying test-time compute allocation. The paper is also useful for teams working with multi-modal or multilingual benchmarks, and for anyone deciding whether to invest additional inference budget in more samples of the same prompt or in variations of the input.

Authors’ abstract

Test-time scaling improves LLM accuracy but multiplies inference cost, making the accuracy gained per unit of compute the metric that matters in deployment. Self-consistency is one of the established approaches, which spends this budget entirely on the output side by sampling repeated reasoning paths. We study Test-Time Augmentation (TTA), which extends self-consistency by also perturbing the input, aggregating predictions across transformed versions of the input, and ask whether input-side diversity converts compute into accuracy more efficiently than output-side diversity. We perform a systematic, matched-compute comparison: we evaluate three simple input-side strategies (semantic rephrasing, lexical perturbations, and visual transformations) across six datasets covering general and multilingual knowledge, mathematical reasoning, multi-modal question answering, and sentiment classification, against chain-of-thought prompting and self-consistency. Semantic rephrasing delivers consistent and statistically significant accuracy gains while Pareto-dominating self-consistency on cost-effectiveness, delivering roughly 1.8X more accuracy per dollar and outperforming it on five of six tasks. We further analyze the number of augmentations, multi-modal strategies, and base model scaling, finding that TTA is most cost-effective for mid-tier models where a stronger model is unavailable or too expensive. Our findings indicate that for current mid-tier LLMs, varying the input converts inference compute into accuracy more efficiently than varying the reasoning path alone. The TTA implementation is available at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.