DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically data curation for visual instruction tuning of Vision-Language Models (VLMs).

Technical level: Advanced. The paper assumes familiarity with vision-language model training pipelines, cross-entropy loss, and dataset selection strategies, though the core intuition is explained in accessible terms.

Scope: The paper proposes a sample-level quality metric (Data Intrinsic Consistency) and an adaptive selection algorithm (DICS) for choosing small, high-value subsets from large visual instruction datasets, validated on LLaVA-1.5-665K, Vision-FLAN-186K, and a newly curated 6M-sample corpus.

What This Paper Is About

Visual instruction tuning requires massive datasets, but training on all of it is expensive and much of it is redundant or low quality. Existing selection methods judge data by distributional diversity, sample relationships, or heuristic rules, which look at the dataset globally and ignore whether the image, instruction, and response inside a single sample actually fit together. This paper defines that internal fit as a measurable score and uses it to pick better training subsets.

Key Contributions

1. Data Intrinsic Consistency (DIC): A self-scoring metric quantifying consistency among the image, instruction, and response within each sample, composed of Visual Information Consistency (VIC) and Response Information Consistency (RIC). DIC is defined as the minimum of the two scores.
2. Data Intrinsic Consistency Selection (DICS): An adaptive sampling strategy that combines DIC ranking, diversity preservation via Farthest Point Sampling (FPS), and budget awareness, behaving differently at low, boundary, and high sampling ratios.
3. DICS-6M: A 6M-sample multimodal instruction corpus assembled entirely from open-source datasets following the InternVL3 collection protocol, used for the largest-scale visual instruction selection study reported in the paper.
4. Extensive validation: Experiments across 12 benchmarks, multiple model architectures (LLaVA-1.5-7B, LLaVA-1.5-13B, Qwen2-VL-7B, InternVL3-8B), two datasets for the small-scale study, and one 6M corpus for the large-scale study.

Main Findings

• Outperforms full-data training at 25% on LLaVA-1.5-665K: DICS reached 101.40% relative performance using a 166K subset, an average improvement of 1.40% across all 12 benchmarks compared with full-dataset training.
• Best scores on several benchmarks: MMBench-CN (55.03), MMMU (36.11), DocVQA (24.64), InfoVQA (23.11), and POPE (84.91).
• Peak performance at 50% sampling ratio: Performance reached 103.15% at a 50% ratio before gradually declining and saturating as more low-DIC samples entered the subset.
• Cross-dataset robustness: On Vision-FLAN-186K, a 25% DICS subset achieved 99.91% relative performance, the best overall among compared methods.
• Cross-architecture transfer: On Qwen2-VL-7B, DICS-selected data reached 100.91% relative performance when Qwen2-VL itself was the selector, and 101.25% when LLaVA-1.5-7B was the selector. LLaVA-1.5-13B reached 101.79% using data selected by the smaller 7B model.
• Large-scale scaling on DICS-6M with InternVL3-8B: Results by sampling ratio were 99.26% at 15% (0.9M), 101.47% at 25% (1.5M), 100.32% at 50% (3.0M), 101.86% at 75% (4.5M), and 103.10% at 85% (5.1M), all relative to the full 6M dataset at 100.00%. The 85% ratio was the optimum, exceeding the full-dataset baseline by 3.10%.
• Comparison with the official model: Against InternVL3-8B-Instruct, which was trained on 21.7M samples and scored 109.08% relative, the DICS model scored 103.10% using 5.1M samples, i.e., 94.52% of the official model's performance with less than 25% of its reported training data.
• Both consistency dimensions are necessary: Ablations on a 25% (166K) subset showed diversity-only FPS at 97.02%, VIC-only at 101.02%, and RIC-only at 97.60%. Fusing the two by sum (98.61%) or product (98.84%) performed worse than the min-based DIC, because a high score in one dimension could compensate for a low score in the other. Removing the part-of-speech token weighting dropped performance from 101.40% to 100.25%.
• FPS is the strongest sampling strategy: DICS with FPS reached 101.40%, versus 101.39% for hierarchical clustering and 100.40% for Top-K. DataTailor and PRISM failed to match the full-dataset baseline under any strategy tested.
• Computational cost: DICS requires only inference-time forward passes with no gradient updates. Selection on LLaVA-1.5-665K took roughly 12 hours on 8 A100 (40GB) GPUs, and selecting a 25% subset reduced total training time to 14 hours, a 1-hour saving versus full-dataset training. The DICS-6M experiment on 32 A100 GPUs completed selection and training in 85 hours, a 5-hour saving over the baseline while gaining 3.10% performance.
• Heuristic baselines trade breadth for narrow gains: Length-based selection maximized HallusionBench by constraining response length but dropped to 88.79% relative performance overall.
• Optimal ratio is not fixed: The paper reports that the best sampling ratio depends on the specific metric and dataset; PRISM and DataTailor both peaked around 75%.
• Qualitative grounding: Attention-map visualizations showed the DICS-trained model localizing the queried object more precisely than other methods.

Methodology in Plain English

The researchers start from a simple hypothesis: a good training sample is one where the image, the question, and the answer genuinely depend on each other.

To measure that, they run a model over each sample twice and compare how surprised it is in each case.

For Visual Information Consistency (VIC), they measure how much the model's loss drops when the image is available versus when it is not. If the answer becomes much easier to predict once the image is shown, the image was carrying real information. They remove the visual signal either by masking out the image tokens or by substituting an unrelated image, and report that both approaches give comparable results.

For Response Information Consistency (RIC), they flip the task around: they ask the model to guess the original question given the image, both with and without the answer. If adding the answer makes the question much easier to reconstruct, the answer was on-topic and informative. This inverse framing is designed to penalize generic, low-information answers that would otherwise score well on standard forward prediction.

Both losses use a part-of-speech weighting scheme, implemented with the spaCy library, that gives full weight (1.0) to content words such as nouns, verbs, adjectives, adverbs, and numbers, and reduced weight (0.1) to functional words such as determiners, prepositions, conjunctions, pronouns, and punctuation. The final DIC score for a sample is the minimum of its VIC and RIC scores, which forces a sample to be strong on both dimensions. Text-only samples are fixed at a DIC of 1.0, and multi-turn dialogues are averaged across turns.

Selection then proceeds in two stages. All samples are scored and ranked, and an adaptive strategy decides how to draw the subset depending on the target ratio. Below 50%, a candidate pool is built from the top 2p fraction and Farthest Point Sampling picks the p fraction from that pool, balancing quality against diversity. At exactly 50%, the top half is taken directly. Above 50%, the top (2p−1) fraction forms a core set and FPS draws the remaining (1−p) fraction from the rest, which effectively prunes redundant low-DIC samples. To keep the scoring stable, the base model is briefly warmed up on a 5% random subset; that warm-up checkpoint is discarded and all final models are retrained from the original pre-trained weights.

Training used LoRA on 8 NVIDIA A100 (40GB) GPUs for LLaVA-1.5 and full fine-tuning on 32 GPUs for InternVL3, one epoch in both cases. Baselines were run following their official implementations.

Why This Matters

The paper argues that intrinsic data consistency matters more than sheer dataset scale. Its practical significance is that a model can be trained on a quarter of the data and still match or exceed full-data training, which directly reduces compute cost, energy use, and the engineering burden of curating ever-larger corpora. It also provides an alternative to methods that depend on expensive external LLMs for filtering, which the paper identifies as unscalable for million-level selections.

Real-world applications:

• Cost-efficient model development: Teams with limited GPU budgets can train competitive VLMs on subsets rather than full corpora, as demonstrated by the 14-hour training run on a 25% LLaVA subset.
• Assistive and accessibility systems: Better visual grounding, shown in the attention-map comparison, supports tasks like describing scenes for visually impaired users, where the model must actually look at the image rather than rely on language priors.
• Document and information workflows: The reported gains on DocVQA and InfoVQA point to document parsing, form understanding, and information extraction from scanned material.
• Hallucination-sensitive deployments: The POPE and HallusionBench results speak to settings such as medical imaging, industrial inspection, or autonomous systems where inventing visual content is costly.

Industry relevance: The cross-architecture transfer results indicate that a small model can score data on behalf of a much larger one, letting organizations use lightweight selectors to curate expensive large-scale training sets. The released DICS-6M corpus and the open-source code at https://github.com/cqu-student/DICS lower the barrier for reproducing and extending the work.

Future Directions

1. Extending beyond image-text: The stated primary limitation is that consistency evaluation currently covers only image-text data, excluding video and audio; the authors plan to extend the framework to broader multimodal contexts.
2. Automatic budget determination: Developing an adaptive strategy to determine the optimal subset size and composition automatically, since the paper observes that the optimal sampling ratio is not fixed and depends on the metric and dataset.
3. Repairing rather than discarding low-DIC samples: Shifting from filtering out inconsistent samples to actively correcting them so they can still contribute to training.
4. Improving sampling efficiency and transfer at larger scales: Given that DIC scores computed by smaller models transfer to larger-model training, an open question is how far this lightweight-selector approach can be pushed as data volumes grow.

Target Audience

This paper is most useful to machine learning researchers and engineers working on vision-language models, dataset curation, and data-efficient training. It will also interest practitioners who need to train multimodal models under compute or budget constraints, and researchers studying data selection, data valuation, or influence-based methods who want a sample-level alternative to distributional and gradient-based approaches. Readers should be comfortable with instruction tuning terminology and standard benchmark names.

Authors’ abstract

Visual instruction tuning is crucial for advancing the vision-language alignment and instruction-following capabilities of Vision-Language Models (VLMs). However, identifying optimal subsets under a fixed ratio constraint from rapidly expanding datasets remains a significant bottleneck. While existing methods largely depend on distribution diversity or heuristic filtering, they often overlook the internal coherence within individual samples. To bridge this gap, we propose Data Intrinsic Consistency (DIC), a self-scoring metric designed to quantify the sample-level inter-component consistency. DIC consists of two modules: Visual Information Consistency (VIC), evaluating the alignment between visual content and instructions, and Response Information Consistency (RIC), assessing response coherence relative to the instruction. Building upon DIC, we introduce Data Intrinsic Consistency Selection (DICS), an adaptive data selection method that optimizes the trade-off between high intra-sample consistency and global distributional diversity under varying data budgets. Extensive experiments demonstrate that DICS consistently outperforms state-of-the-art methods across diverse dataset scales and model architectures, surpassing full-dataset fine-tuning while using only 25% of the LLaVA-1.5-665K data. We further curate DICS-6M, a 6M-sample multi-modal instruction corpus that enables the largest-scale visual instruction selection study to date; remarkably, DICS reaches 94.52\% of the official InternVL3-8B-Instruct performance using less than 25\% of its reported training data. Code can be seen at https://github.com/cqu-student/DICS