DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning

AI summary

Overview

Research area: Multimodal large language models (MLLMs), reasoning transfer, and efficient fine-tuning. Technical level: Intermediate — familiarity with supervised fine-tuning (SFT), gradients, and model merging helps, but the core ideas are explained conceptually. Scope: The paper proposes a gradient-space method, DRIFT, that transfers reasoning ability from text-only reasoning LLMs into multimodal models using a small amount of multimodal reasoning data, without additional parameters or expensive reinforcement learning.

What This Paper Is About

Multimodal large language models can perceive images well but reason noticeably worse than strong text-only LLMs. The usual fixes — large-scale multimodal chain-of-thought (CoT) supervised fine-tuning or reinforcement learning — are expensive, and the paper shows that the simple alternative of merging a reasoning model's weights into an MLLM is fragile and often degrades performance. DRIFT instead uses the difference between a text-only reasoning expert and a multimodal model as a directional prior that biases gradients during ordinary fine-tuning.

Key Contributions

1. The paper demonstrates that parameter-space merging for reasoning injection is brittle: it helps some model families (LLaVA-Next, Idefics) but degrades Qwen-family models, and the authors quantify this with layer-wise L2 norm and cosine similarity analysis.
2. It introduces DRIFT (Directional Reasoning Injection for Fine-Tuning), which defines a reasoning direction Δ = φ_reason − φ_VL over reasoning-relevant modules (MLP projections, attention projections, and normalization layers) and injects it into gradients during multimodal SFT rather than into weights.
3. It proposes and compares three injection variants — Absolute, Grad-Norm, and Grad-Norm with Adaptive α — showing that gradient-based scaling is stable while the Absolute rule degrades performance.
4. It reports experiments on multimodal reasoning benchmarks (MathVista, MathVision, MathVerse, WeMath, LogicVista) showing DRIFT beats naive merging and standard SFT, and matches or surpasses training-intensive methods with far less data and compute.

Main Findings

• Naive merging is not a "free lunch." When merging text-only reasoning experts into MLLM backbones (Table 1), LLaVA-Next-LLaMA3-8B improved on all three reported benchmarks (MathVista 37.4 → 38.2, MathVision 13.8 → 15.8, MathVerse 16.0 → 17.4), but Qwen2.5-VL-7B degraded on all three (MathVista 67.9 → 65.8, MathVision 25.0 → 22.7, MathVerse 41.4 → 33.2). The paper states BR2V yields up to a 2% improvement on LLaVA-Next and Idefics while often degrading the Qwen series.
• Parameter-space separation explains the failures. Layer-wise L2 norm and cosine similarity analysis (Figure 2) showed LLaMA and Mistral variants remain relatively close in parameter space, while Qwen variants are substantially more dispersed, and multimodal Qwen models' parameter magnitudes diverge sharply from their reasoning counterparts.
• No parameter-merging baseline improved Qwen2.5-VL-7B-Instruct. In Table 2, Task Arithmetic, Layer Swap, TIES, DARE-TIES, and DARE-Linear all scored below the 67.9 / 25.0 / 41.4 / 34.3 / 52.8 / 46.7 baseline across MathVista, MathVision, MathVerse, WeMath strict/loose, and LogicVista (average 44.7). DRIFT scored 69.9 (+2.0), 26.6 (+1.6), 43.9 (+2.5), 38.5 (+4.2), 60.2 (+7.4), 47.2 (+0.5), average 47.7 (+3.0). The paper describes the MathVista gain over Task Arithmetic as +4.5 points.
• DRIFT beats its own SFT baseline on every reported reasoning benchmark. In Table 3, SFT scored 68.7 (MathVista), 25.1 (MathVision), 42.0 (MathVerse), 33.3 (WeMath), 45.6 (LogicVista); DRIFT scored 69.9 (+1.2), 26.6 (+1.6), 43.9 (+1.9), 38.5 (+5.2), 47.2 (+1.6).
• Training-intensive methods remain sometimes higher in absolute score. X-REASONER reached 69.0 on MathVista and 29.6 on MathVision; R1-Onevision-7B reached 64.1, 29.9, 40.0, 61.8 (WeMath not reported). The paper states DRIFT reaches competitive performance with orders of magnitude less reasoning-specific data and training time.
• Efficiency. DRIFT is reported to require 4K multimodal SFT examples versus more than 59K for reasoning-oriented training methods (Figure 1), and to complete in roughly two hours versus more than 1 day for OpenVLThinker-7B and R1-OneVision-7B and more than 2 days for X-REASONER (Table 8).
• Where to inject matters. In Table 4, the Absolute rule on {ATTN, MLP} degraded all benchmarks (MathVista 65.7, MathVerse 39.5, LogicVista 25.9 versus SFT's 68.7/42.0/45.6). Grad-Norm w/ Adaptive α gave the best MathVista score (69.9, +1.2), while Grad-Norm restricted to {ATTN} gave the strongest LogicVista score (49.4, +3.8) and MathVerse of 44.4 (+2.4). Adding normalization layers often diminished performance.
• DRIFT preserves perception. In Table 6, SFT regressed on RealWorldQA (−1.83) and MMStar (−1.90); DRIFT scored 48.79 (HallusionBench), 69.15 (RealWorldQA), and 65.60 (MMStar), versus base scores of 44.39, 68.62, 64.70.
• Generality across pairings. In Table 5, DRIFT improved over both base and SFT for Qwen2.5-VL-7B with Qwen2.5-Math-7B (MathVision 26.00, WeMath strict 36.70, loose 59.30) and with DeepSeek-R1-Distill-7B (26.60, 38.50, 60.20), and for LLaVA-Next-8B with DART-Uniform-8B (MathVision 16.94, WeMath strict 10.57, loose 27.33).
• Robustness to noisy data. In Table 7, adding 4K noisy examples to the 4K high-quality set dropped SFT from 25.10 to 23.20 on MathVision, while DRIFT degraded more gracefully (26.60 to 25.70). The paper notes 4K high-quality data outperformed the larger but noisier 8K set for both methods.

Methodology in Plain English

The method starts from two models sharing a lineage: a text-only reasoning expert (e.g., DeepSeek-R1-Qwen-Distill-7B) and a multimodal model (e.g., Qwen2.5-VL-7B-Instruct). Instead of blending their weights — which can distort multimodal alignment — the authors compute the difference between them, Δ = φ_reason − φ_VL, over specific modules, and treat this difference as a "reasoning direction."

During ordinary multimodal supervised fine-tuning, the weights are left untouched. Instead, during backpropagation the method adds a term to each gradient: g̃ = g + α · scale(g, Δ). Three ways of doing this scaling are tested: adding αΔ directly (Absolute), scaling Δ to match the gradient's magnitude (Grad-Norm), and additionally weighting by how aligned the gradient is with Δ (Grad-Norm w/ Adaptive α).

The reasoning direction is computed once, cached on CPU, and moved to GPU only when needed. No new parameters are added; only the backward pass changes, so it plugs into existing SFT pipelines.

For data, the authors start from the ThinkLiteVL-11K set of 11K image–question pairs, use ThinkLite models to distill CoT annotations, filter out incorrect answers and invalid outputs, and keep 4K examples with reasoning traces wrapped in <think></think> tags. Training uses the LLaMAFactory codebase for three epochs at a learning rate of 1×10⁻⁶ with α = −1 for all variants. Evaluation uses VLMEvalKit on MathVista (testmini), MathVision, MathVerse (vision-only subset), WeMath, and LogicVista.

Why This Matters

The result suggests that a large part of what makes a model reason can be captured as a direction in parameter space and transferred through gradients, rather than paid for with enormous multimodal CoT datasets or reinforcement learning runs. That reframes reasoning transfer as a lightweight optimization-time intervention rather than a data- or compute-intensive training project.

Real-world applications:

• Education and tutoring tools that need to solve math and science problems from photographed homework, diagrams, or charts.
• Document and chart analysis in finance or research, where a model must read a figure and then chain inferences rather than only describe it.
• Assistive and accessibility systems that describe a scene and then reason about it (for example, answering "is this route safe?" from a photo).
• Enterprise visual agents that combine perception with multi-step planning on limited fine-tuning budgets.

Industry relevance: the method needs no additional parameters, caches its prior on CPU, and modifies only the backward pass, which makes it attractive for teams that already run standard SFT and want reasoning gains without building large multimodal CoT pipelines or RL infrastructure. Reported gains span two model families and multiple reasoning experts, suggesting it is not tied to a single vendor's architecture.

Future Directions

• Reducing dependency on a text-only reasoning expert. The limitations section notes DRIFT assumes such an expert is available, which may not hold in every domain.
• Broadening evaluation. The authors suggest testing on commonsense reasoning and embodied perception domains beyond the math-heavy benchmarks used here.
• Lowering training overhead. The paper lists reducing the modest overhead relative to standard SFT as a future direction.
• Improving interpretability. Understanding what the injected reasoning signal actually encodes remains an open question, as does whether better expert–backbone pairings can be selected automatically given the parameter-space dispersion findings.

Target Audience

Researchers and engineers working on multimodal LLM fine-tuning, efficient adaptation, or knowledge transfer between models. It is also relevant to practitioners who already run supervised fine-tuning pipelines and want reasoning gains without large multimodal CoT datasets or reinforcement learning, and to readers interested in the limits of parameter-space model merging.

Authors’ abstract

Multimodal large language models (MLLMs) have made rapid progress, yet their reasoning ability often lags behind strong text-only LLMs. Bridging this gap typically requires large-scale multimodal reasoning data or reinforcement learning, incurring substantial cost. An appealing alternative is parameter-space model merging between reasoning-enhanced LLMs and MLLMs, but we show that naive merging is fragile: its effectiveness varies widely across model families and can significantly degrade performance (e.g., for Qwen-based MLLMs). We propose Directional Reasoning Injection for Fine-Tuning (DRIFT), a lightweight method that transfers reasoning knowledge in the gradient space while preserving multimodal alignment. DRIFT precomputes a reasoning prior from the parameter differences between text-only reasoning experts and multimodal models, and uses it to bias gradients during supervised fine-tuning. This design retains the simplicity of standard SFT pipelines while enabling efficient and stable reasoning transfer. Experiments on multimodal reasoning benchmarks, including MathVista and MathVerse, show that DRIFT consistently outperforms naive merging and standard SFT, and matches or surpasses training-intensive methods with substantially lower data and compute.