Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View

AI summary

Overview

Research area: Multimodal Large Language Model (MLLM) post-training, specifically data sampling and curriculum design for reinforcement-learning-based fine-tuning of vision-language models.

Technical level: Advanced. The paper assumes familiarity with GRPO (Group Relative Policy Optimization), supervised fine-tuning (SFT), Chain-of-Thought reasoning, transformer attention decomposition, and multimodal benchmarks. The core ideas are intuitive but the implementation details are specialized.

Scope: The paper proposes two difficulty-measurement metrics for multimodal training samples and uses them to show that a GRPO-only training pipeline on difficulty-stratified data outperforms the conventional SFT-then-GRPO pipeline.

What This Paper Is About

Most multimodal reasoning post-training methods borrow data-sampling ideas from text-only language models, where "difficulty" can be judged from the wording of a math or code problem. That does not work for image-plus-text samples, because difficulty in these tasks comes from the interaction between the two modalities, not from the text alone. The authors ask two questions: how do you quantify the hardness of a multimodal sample, and what is the best training pipeline once you have done so? Their answer is two complementary difficulty scores plus a finding that reinforcement learning alone, applied to well-chosen samples, beats the standard SFT+GRPO recipe.

Key Contributions

1. Progressive Image Semantic Masking (PISM): A difficulty metric that progressively masks an increasing proportion of an image's pixels and measures at what masking ratio the model's accuracy collapses. Samples that break under light masking are labeled hard; samples that survive heavy masking are labeled easy.

2. Cross-Modality Attention Balance (CMAB): A difficulty metric computed from the model's own attention weights. It measures the ratio of attention paid to image tokens versus text tokens across generated response tokens and transformer layers. Samples that require balanced attention from both modalities are labeled hard.

3. A difficulty-stratified training framework: The authors use the two metrics to partition perception and reasoning datasets into easy / medium / hard / unsolved subsets, then systematically compare GRPO-only training against several SFT+GRPO orderings (medium-then-hard, hard-then-medium, and random-data controls) across six benchmarks.

4. An empirical challenge to the standard pipeline: The paper reports that GRPO-only training on medium+hard samples consistently outperforms SFT+GRPO configurations, implying that supervised fine-tuning can be dropped when sample selection is done well.

Main Findings

• GRPO-only beats SFT+GRPO on difficulty-stratified data. The authors report that models trained with GRPO-only on medium+hard samples outperform all SFT+GRPO variants across visual perception and visual reasoning tasks. On OCRBench (Table 1, visual reasoning via PISM), GRPO-only(mid+hard) scores 77.800, which the paper states surpasses all SFT+GRPO variants by at least 1.3 points.

• Filtering matters more than data volume. On MathVista under PISM (Table 1), GRPO-only(mid+hard) scores 68.300 versus 53.400 for GRPO-only(fullset). The same trend is reported on MMMU (0.547 vs. 0.440) and MMStar (0.639 vs. 0.607). The authors attribute this to easy and unsolved samples diluting the training signal.

• Random sampling underperforms difficulty-aware sampling. GRPO-only(random) on OCRBench in Table 1 scores 77.300 — below the 77.800 of GRPO-only(mid+hard) — which the paper reads as evidence that indiscriminate inclusion of samples where textual cues suffice reduces training efficiency.

• PISM excels at perception, CMAB excels at reasoning. PISM performs particularly well on perception-heavy tasks such as OCRBench and MMVet. CMAB, using mid+hard samples, reaches 69.000 on MathVista (Table 3), exceeding the PISM-based GRPO-only result of 68.300 and all SFT+GRPO configurations in that table. The paper also cites MMStar at 0.639 versus 0.625 for SFT+GRPO as supporting attention balance as a difficulty indicator.

• Reduced hallucination is claimed as a GRPO-only benefit. On HallusionBench (Table 2, visual perception via PISM), GRPO-only(mid+hard) scores 69.716, while the SFT+GRPO variants in that table top out at 68.980. The authors argue SFT's reliance on manually designed reasoning templates encourages surface pattern matching ("Pseudo-CoT") and raises hallucination risk.

• The two metrics classify data very differently. Under PISM (Table A), visual perception data totals 20,633 samples: 7,827 easy, 4,872 medium, 1,454 hard, 6,480 unsolved; visual reasoning totals 27,133: 5,048 easy, 1,061 medium, 1,618 hard, 19,406 unsolved. Under CMAB (Table C), visual perception yields 6,753 easy, 6,029 medium, 1,001 hard, 6,850 unsolved; visual reasoning yields 2,170 easy, 3,604 medium, 2,166 hard, 19,193 unsolved.

• Reference-only baselines. The paper explicitly states that GRPO-only(fullset) and GRPO-only(unsolved) results are presented only as a reference because of their large data volumes, and are not included in the comparison with other training strategies.

Methodology in Plain English

The authors start from a simple intuition: if a question genuinely depends on the image, then damaging the image should hurt the model's answer.

PISM tests this directly. For each image-text pair, the model is run repeatedly with increasing fractions of the image's pixels randomly hidden — masking ratios from 0.0 to 0.9 in steps of 0.1. Because masking is random, each ratio is tested 10 times and the results are averaged into an accuracy estimate. The key number for each sample is the smallest masking ratio at which accuracy falls below a threshold of 0.1. If that happens at or below 0.4, the sample is hard; above 0.7, or never, it is easy; in between it is medium. Samples the model already gets wrong on the unmodified image are labeled unsolved and excluded from difficulty judgment, since their failure reflects model limits rather than sample complexity.

CMAB looks inside the model instead. For every token the model generates, the authors sum up how much attention that token pays to image tokens versus text tokens. The ratio is computed per layer, then combined across layers using a geometric mean that excludes the first and last transformer layers (which they treat as handling input encoding and output decoding rather than high-level reasoning). The per-token ratios are then averaged across the whole response. A ratio below 0.1 means the text dominates; above 1.9 means the image dominates; between 0.4 and 1.6 means the model must genuinely use both modalities — those are the hard samples. Intermediate bands are medium.

With samples sorted, the authors compare training pipelines. One is GRPO-only: reinforcement learning applied directly to a chosen subset. The others are SFT+GRPO: supervised fine-tuning on one subset first (medium, hard, or a random subset of matched size), followed by GRPO on another. The subsets were chosen to have matched sizes so the comparison isolates which data was used rather than how much. All experiments used the Qwen2.5VL-7B base model, with SFT run through LLaMA-Factory and GRPO through the Swift framework, on clusters of NVIDIA A800-SXM4 and H20 GPUs. Evaluation covered MathVista, MMVet, OCRBench, HallusionBench, MMMU, and MMStar, scored through OpenCompass with GPT-4o-mini as the judge.

Why This Matters

Impact on research. The paper pushes back on a widely repeated assumption in the multimodal RL literature — that supervised fine-tuning is a necessary cold-start step before reinforcement learning. It also argues that text-derived difficulty scores (human math ratings, rejection sampling, reward scores, sentence entropy) are the wrong tool for multimodal data, and proposes that difficulty should be measured through cross-modal signals instead. Both claims are testable and, if they hold up, would simplify multimodal post-training pipelines considerably.

Real-world applications (these follow from the benchmark domains the paper evaluates, not from deployment experiments in the paper):

• Document digitization and OCR systems, where the paper's OCRBench results are directly relevant.
• Chart, table, and scientific-figure question answering in analytics and research tools, corresponding to MathVista and MMMU.
• Visual grounding and counting in robotics or industrial inspection, where answers depend on precise visual detail.
• Reducing hallucination in multimodal assistants used for accessibility or customer support, corresponding to HallusionBench.

Industry relevance. The paper is co-authored with researchers at ZTE and reports work done during an internship there, signaling direct industrial interest. The practical payoff is cheaper training: dropping the SFT stage from a post-training pipeline removes a data-annotation burden and a compute stage, while targeted sample selection lets a team train on a small curated subset instead of a full corpus. The code is released at the GitHub link given in the abstract.

Future Directions

• Validating the metrics beyond Qwen2.5VL-7B. All reported experiments use a single 7B foundation model. Whether PISM's accuracy-collapse threshold and CMAB's attention-ratio bands transfer to other architectures and scales is not established in the paper.

• Reconciling the two difficulty definitions. PISM and CMAB produce very different category sizes — for example, 4,872 medium and 1,454 hard perception samples under PISM versus 6,029 medium and 1,001 hard under CMAB. The paper does not report a combined metric, leaving open whether merging the two perspectives would improve results further.

• Understanding the large unsolved fractions. Under both metrics, the unsolved category is the largest for visual reasoning data (19,406 under PISM, 19,193 under CMAB). The paper states these are excluded from difficulty assessment, but does not investigate whether they represent a data-quality problem, a capability ceiling, or a target for a different training approach.

• Cost accounting for the sampling itself. PISM requires running the model 10 times per masking ratio across 10 ratios per sample, and CMAB requires attention extraction on every sample. The paper does not report the compute cost of the selection procedure relative to the savings from the smaller training subsets.

Target Audience

Researchers and engineers working on multimodal model post-training, particularly those applying reinforcement learning with verifiable rewards to vision-language tasks. It is also relevant to practitioners deciding whether to invest in supervised fine-tuning data before RL, and to anyone designing data-curation or curriculum-learning strategies for multimodal systems. Readers without a background in GRPO, transformer attention internals, or multimodal benchmarks will need to consult the cited prior work first.

Authors’ abstract

Recent advances in Multimodal Large Language Models (MLLMs) have spurred significant progress in Chain-of-Thought (CoT) reasoning. Building on the success of Deepseek-R1, researchers extended multimodal reasoning to post-training paradigms based on reinforcement learning (RL), focusing predominantly on mathematical datasets. However, existing post-training paradigms tend to neglect two critical aspects: (1) The lack of quantifiable difficulty metrics capable of strategically screening samples for post-training optimization. (2) Suboptimal post-training paradigms that fail to jointly optimize perception and reasoning capabilities. To address this gap, we propose two novel difficulty-aware sampling strategies: Progressive Image Semantic Masking (PISM) quantifies sample hardness through systematic image degradation, while Cross-Modality Attention Balance (CMAB) assesses cross-modal interaction complexity via attention distribution analysis. Leveraging these metrics, we design a hierarchical training framework that incorporates both GRPO-only and SFT+GRPO hybrid training paradigms, and evaluate them across six benchmark datasets. Experiments demonstrate consistent superiority of GRPO applied to difficulty-stratified samples compared to conventional SFT+GRPO pipelines, indicating that strategic data sampling can obviate the need for supervised fine-tuning while improving model accuracy. Our code will be released at https://github.com/qijianyu277/DifficultySampling.