On the Limits of Scale: Inverse Scaling in Vision-Language Models

AI summary

Overview

Research area: Multimodal machine learning — specifically, how vision-language model (VLM) accuracy changes with model size within a single model family, measured at the level of benchmark subtasks rather than aggregate scores.

Technical level: Intermediate. The core idea is intuitive (sometimes bigger models are worse), but the paper relies on familiarity with benchmarks like MMMU and BLINK, statistical paired testing (McNemar), and the dense-vs-Mixture-of-Experts distinction.

Scope: A large-scale empirical audit of 19 open-weight VLMs from four families across nine benchmark splits, using per-question paired comparisons to detect and characterize subtask-level inverse scaling.

What This Paper Is About

Scaling laws encourage the assumption that a larger model is a better model. This paper tests that assumption inside vision-language model families, where checkpoints from roughly 1B to 235B parameters share a name and general recipe. By comparing every pair of sizes within a family question-by-question on standard multimodal benchmarks, the authors show that on specific subtasks — multi-image spatial reasoning, certain perception tasks, and specific university-level STEM subjects — smaller models frequently beat their larger siblings, and these regressions are invisible in headline benchmark numbers.

Key Contributions

1. A per-question pairwise "flip" analysis for detecting inverse scaling. Instead of comparing subtask accuracies, the authors count questions where the smaller model is correct and the larger is wrong (FlipInv), the reverse (FlipNorm), and their difference (Net). This isolates concrete disagreements rather than net accuracy, and is applied to all within-family model pairs.

2. A large empirical audit. 19 open-weight models from four families (Qwen3-VL, InternVL3.5, Gemma 3, Molmo 2), spanning roughly 1B to 235B total parameters and including both dense and Mixture-of-Experts variants, evaluated across eight benchmarks (nine splits), 29,386 questions per model, 171 inference runs, and 558,334 model–question evaluations in total.

3. Quantification of inverse scaling breadth. 375 subtask–model-pair cases meeting the paper's criteria (at least 20 shared questions and Net ≥ 5), spanning all four families and every benchmark, and covering 4,342 unique queries where a smaller model succeeds and a larger one fails.

4. Evidence that the phenomenon is task-driven, not family-specific. 13 subtask–dataset pairs show inverse scaling in at least three of the four families, and the pattern survives restriction to dense-only pairs (193 cases, 8 cross-family subtasks) and to pairs with a twofold size gap (326 of the 375 cases).

Main Findings

• Aggregate scores hide the effect. The largest model in each family is highest or near-highest on most benchmarks, but exceptions exist: on BLINK, Qwen 30B-A3B reaches 62.4% versus 66.4% for the 4B model, and Molmo 4B (63.5%) beats Molmo 8B (59.3%). Molmo 4B also edges out 8B on MMBench (88.3% vs. 87.9%).

• 375 inverse-scaling cases across all families and all benchmarks. Case counts by family: Qwen3-VL 122, InternVL3.5 207, Gemma 3 26, Molmo 2 20, out of 10,434 eligible comparisons. Normalized rates are similar for Qwen3-VL (3.7%), InternVL (3.3%) and Gemma 3 (3.9%), with Molmo 2 higher (9.0%, from a single model pair).

• BLINK dominates in both raw counts and rate. BLINK contributes 95 cases (14.4% normalized), followed by MMT-Bench at 5.8%, MME at 5.3%, MMBench at 5.0%, MMMU at 2.3%, and ScienceQA at 1.2%. Raw counts are highest for MMMU (115), BLINK (95) and MMT-Bench (87), and lowest for AI2D (9) and MathVista (3); the authors note diagram understanding (AI2D) appears comparatively resistant.

• Some gaps are very large. The top 10 cases range from 7.2 to 39.4 percentage points. Qwen 4B scores 84.0% on BLINK Jigsaw versus 60.0% for 30B-A3B (+24.0 pp, Net = 36). Gemma 3 4B beats 27B by 39.4 pp on ScienceQA Basic Economic Principles (Net = 28), and Gemma 3 12B reaches 95.1% versus 27B at 51.2% on "Particle motion and energy" (+43.9 pp, Net = 18). On InternVL BLINK Forensic Detection, three separate cases have FlipNorm = 0 — no question where the larger MoE model succeeds and the smaller model fails.

• Curves are usually non-monotonic, not simply declining. Of the 171 family–subtask pairs, 87% show non-monotonic accuracy curves rather than consistent degradation. InternVL on MMMU "Mechanics of Materials" goes 27.6% (1B), 48.3% (2B), 44.8% (4B), 31.0% (8B), 51.7% (14B), 58.6% (30B-A3B), 62.1% (38B), then back down to 27.6% (241B-A28B).

• Sometimes the smallest model wins outright. Qwen 2B is the best in its family on MMMU test AI Content Detection (65.0% vs. 45.0% for 235B), and Molmo 4B outperforms 8B on every BLINK subtask showing inverse scaling.

• The effect is not just the biggest model failing. Adjacent-size comparisons alone yield 197 cases across 107 subtask–dataset pairs, including 12 cross-family subtasks, so local regressions between neighbouring sizes are common.

• Replication across independent architectures. 13 subtask–dataset pairs recur in at least three families. BLINK Multi-view Reasoning appears in all four families; Organic Chemistry, Investment and Thermodynamics recur on MMMU; Autonomous Driving recurs on MMT-Bench and Scene on MME; Coarse Perception recurs on MMBench. A further 29 subtasks appear in exactly two families.

• Not a Mixture-of-Experts artifact. MoE variants often appear as the larger member of inverse-scaling pairs, but excluding all MoE models still leaves 193 cases, with 8 cross-family subtasks including BLINK Multi-view Reasoning in all four families.

• The effect shrinks but persists at large size gaps. Restricting to pairs where the larger model has at least twice the total parameters leaves 326 of 375 cases (3.3% of 9,768 comparisons); using active parameters leaves 239 cases (2.9% of 8,214), and both restrictions retain 11 of the 13 cross-family subtasks. At the widest gap in each family (smallest vs. largest), only 9 of 666 comparisons show Net ≥ 5.

• Chain-of-thought prompting attenuates but does not fix it. Re-running both models of every case under CoT (66,698 matched question instances) drops the median Net from +8 to 0, but 96 of 375 cases (25.6%) retain Net ≥ 5, 101 reverse, and the larger model recovers only 43.5% of the questions it had lost. Persistence ranges from 48.6% of MME cases and 42.1% of BLINK cases down to 7.6% on MMMU test. Part of the improvement comes from the smaller model getting worse under CoT (in 251 of 375 cases).

• Most flips are genuine errors, not annotation noise. On 1,300 sampled flips, GPT-5.4 labelled 97.4% as genuine errors and 2.6% as bad annotation or ambiguous, with none as format mismatch. A three-author human majority vote on a 127-question subsample classified 84.3% as genuine and 15.7% as bad annotation or ambiguous; the authors adopt the human figure as the conservative estimate (at least 84% genuine failures). Human–human agreement was Fleiss κ = 0.65; the two LLM judges agreed on 90.8% of questions (Cohen's κ = 0.30).

• Statistics are exploratory. 133 of 375 cases (35.5%) have uncorrected McNemar one-sided p < 0.05, rising from 10.4% of cases with Net 5–7 to 100% with Net ≥ 21, but the authors explicitly caution that these p-values are exploratory because cases were selected on the same data and comparisons overlap.

Methodology in Plain English

The authors picked four openly released VLM families and treated each family as a size ladder. For every pair of sizes within a family, they ran both models on the exact same questions from nine standard benchmark splits, under identical conditions: zero-shot prompting, greedy decoding, thinking modes disabled, no quantization, on NVIDIA H100 GPUs.

For each benchmark subtask (the category labels the benchmarks already provide, such as "Jigsaw" in BLINK or "Organic Chemistry" in MMMU), they compared the two models question by question. Three numbers summarize the comparison: how many questions the smaller model got right while the larger got wrong, how many went the other way, and the difference between them. A large positive difference means the smaller model is winning the disagreements.

To avoid reporting trivial noise, a case is only counted if the subtask has at least 20 shared questions and the net difference is at least 5. The authors check that conclusions hold under a stricter threshold of 10, under dense-only comparisons, and under restrictions to pairs with a twofold size gap. Answer correctness is checked by deterministic extraction and matching, with GPT-5.4 used as a multimodal fallback judge for the 0.4% of responses where rules are inconclusive. To understand why the larger models fail, GPT-5.4 (blinded to model identity) categorized 1,300 sampled flips into failure types, cross-checked against Claude Opus 4.6 and against three human annotators.

Why This Matters

The paper reframes model selection as a task-dependent decision rather than a "pick the biggest" default. Aggregate benchmark tables — including ones published by model providers — can mask subtask-level regressions that matter for a specific deployment.

Real-world applications where the affected subtasks arise:

• Multi-image and spatial tasks: the paper's strongest and most cross-family effects are in BLINK subtasks such as Jigsaw, Multi-view Reasoning, Forensic Detection and Object Localization, which are proxies for comparing images, reading scenes from multiple viewpoints, and spotting inconsistencies.
• Scientific and technical assistance: university-level MMMU subjects (Organic Chemistry, Investment, Thermodynamics) show cross-family inverse scaling, implying that subject-matter accuracy cannot be inferred from general benchmark rank when choosing a model for STEM tutoring or analysis tools.
• Autonomous driving and scene understanding: the MMT-Bench Autonomous Driving subtask and MME Scene subtask both replicate across three families, which is directly relevant to perception-heavy deployment choices.
• Document and diagram-heavy workflows: the paper reports AI2D as comparatively resistant and discusses the MME annotation issues, giving teams a sense of where size-based selection is safer and where it is riskier.

Industry relevance: shipping the largest available checkpoint is expensive. If a mid-size model beats a much larger one on the capabilities an application actually uses, the cost, latency and serving implications are substantial — and the paper's flip analysis is a cheap, reusable diagnostic that teams can run on their own task categories.

Future Directions

• Establishing causality. The authors stress that their results describe released variants, not the causal effect of parameter count: checkpoints under one family name differ in vision encoder, training data, alignment and active parameters. They call for controlled studies training VLMs of several sizes under one fully released recipe.
• Explaining the mechanism. Several hypotheses are offered but untested — localized capability regression, greater susceptibility of larger models to plausible distractor options, MoE routing effects, and training-data composition. Isolating expert routing from active-parameter count is flagged as a specific open problem.
• Testing robustness beyond option order. The strongest BLINK cases survive deterministic reorderings of their answer options, but paraphrases are untested, and size-dependent benchmark contamination cannot be ruled out.
• Diagnosing chain-of-thought "recovery." CoT fixes fewer than half of the individual errors and leaves a quarter of cases intact, and the authors note it is unclear whether this reflects genuine reasoning improvement given recent work on intermediate steps masking vulnerabilities.

Target Audience

Practitioners choosing which VLM to deploy for a specific multimodal task; evaluation engineers who want a lightweight paired-comparison diagnostic beyond aggregate leaderboards; and researchers studying scaling behavior, multimodal robustness, or MoE-versus-dense trade-offs. Readers without a background in benchmarks or paired statistical tests will still follow the main argument, but the dense results tables and significance discussion reward some familiarity with evaluation methodology.

Authors’ abstract

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.