CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization

AI summary

Overview

Research area: Multimodal large language models (MLLMs), specifically data-efficient visual instruction tuning and instruction data selection.

Technical level: Advanced. The paper builds on gradient-free feature extraction, spectral clustering, maximum likelihood estimation, and homoscedastic uncertainty weighting, targeting readers familiar with multimodal training pipelines.

One-sentence scope: The paper introduces CoIDO, a framework that trains a lightweight scorer on a small random fraction of a visual instruction dataset to jointly learn sample importance and dataset diversity, then uses that scorer to rank and select a high-quality subset for fine-tuning.

What This Paper Is About

Fine-tuning multimodal models like LLaVA-1.5-7B on large visual instruction datasets such as LLaVA-665K is expensive — the paper notes that one epoch typically requires over 20 GPU hours on 8× A100 40GB GPUs. Existing selection methods try to reduce this cost, but they often process the entire dataset with the target model (costing as much as full training) and treat "which samples are important" and "which samples keep the dataset diverse" as two separate problems solved in two separate stages. CoIDO aims to solve both problems together, using only a small sampled fraction of the data, so that a single lightweight scorer can rank and select the most valuable subset directly.

Key Contributions

1. Coupled importance-diversity optimization. The paper proposes a dual-objective formulation for MLLM visual instruction data in which importance and diversity are optimized jointly during training rather than handled by separate stages or specialized selection algorithms.
2. A lightweight plug-in scorer. Instead of evaluating every sample with the target MLLM, CoIDO uses a scorer (a four-layer MLP in the main experiments) trained on only a small random subset (up to 20%) to learn the distribution of the candidate set, enabling ranking across the full dataset at greatly reduced training cost.
3. A selection stage without a diversity algorithm. Because diversity is baked into the learned score, the selection stage reduces to ranking samples by their CoIDO score and picking the top fraction per downstream task, which the paper says also avoids task bias from global ranking.
4. Efficiency and accuracy results. Training LLaVA-1.5-7B on only 20% of samples selected by CoIDO reaches 98.2% of full-data fine-tuning performance on average across ten downstream tasks, while achieving the lowest total training FLOPs (4.2E ExaFLOPs) among the compared methods.

Main Findings

• Near-full performance at a fraction of the data: On LLaVA-1.5-7B across ten downstream benchmarks, the CoIDO-selected 20% subset reaches a relative score (Rel.) of 98.2% of full-data fine-tuning.
• Best efficiency among compared methods: CoIDO reports the lowest total cost at 4.2E ExaFLOPs, versus 10.2E for Full Data, 31.2E for Self-Filter, 11.7E for TIVE, 12.6E for ICONS, and 4.9E for COINCIDE.
• Reduced MLLM Training Data Cost: CoIDO reports an MLLM Training Data Cost of 20%, whereas the model-involved baselines (Self-Filter, TIVE, ICONS, COINCIDE) report 100%. TIVE additionally uses 8% warm-up data, and ICONS uses 5% warm-up plus 2.2% for benchmark validation sets.
• Competitive with SOTA selection: CoIDO's 98.2% Rel. is comparable to ICONS at 97.1% (its original paper reports 98.6%) and COINCIDE at 97.4%, while TIVE reaches 96.7% and random sampling reaches 95.1%.
• Coupled optimization beats alternatives: Ablations show importance loss alone gives 89.0% Rel., simple summation of ℒ_I + ℒ_D gives 92.0%, learnable weighted combination gives 95.9%, and the proposed homoscedastic uncertainty formulation gives 98.2%.
• The MLP scorer is sufficient: The paper evaluates Transformer-based and attention-only scorer variants (shown in Figure 3, with performance and training-time FLOPs), and attributes the strength of the MLP-based scorer to the expressiveness of the extracted multimodal features.
• Training data ratio plateaus at 20%: Performance is modest when the scorer training fraction p% is below 10%, improves beyond that, and stabilizes after p > 20%, which is why 20% is used as the default.
• Stable across selection ratios: CoIDO is evaluated for selection ratios γ from 5% to 40% and reports the highest Rel. across that range; the paper notes the gap narrows as γ approaches 40%, and that above a 50% selection ratio randomness has a larger effect.
• Generalizes and transfers across datasets: Evaluated on the Vision-Flan dataset (over 200 diverse vision-language tasks) with 20% selection, CoIDO reaches 102.1% Rel. versus 100.0% for full fine-tuning and 97.8% for random selection. A scorer trained on LLaVA-665K and applied to Vision-Flan without retraining reaches 103.7% Rel.

Methodology in Plain English

1. Extract cheap features first. For each sample, CoIDO gathers text quality scores from an LLM-based evaluator (the LLM Score), image quality scores from ImageReward, and image-text features from a pretrained CLIP encoder including a CLIP Score. These are concatenated and serve as the scorer's input, so the scorer never has to run the target model on the whole dataset.
2. Group samples with spectral clustering. The concatenated features are clustered into M classes using spectral clustering instead of K-means, which the paper argues handles complex, non-linear multimodal distributions better. These cluster assignments become the basis for measuring diversity.
3. Train a scorer alongside the target model. On a random fraction p% of the data, each batch does double duty: it fine-tunes the target MLLM with standard instruction learning, and it trains the plug-in scorer (a four-layer MLP) to output a scalar CoIDO score per sample.
4. Define the importance loss. The CoIDO scores are softmax-normalized and used to reweight each sample's cross-entropy loss. Since harder samples produce higher cross-entropy, backpropagation naturally pushes their weights down, so a lower weight marks a more important sample.
5. Define the diversity loss. To stop one cluster from dominating, the method minimizes the variance of average weights across clusters present in the batch. This pushes down over-weighted clusters and pushes up under-weighted ones, while deliberately leaving within-cluster ranking untouched.
6. Balance the two objectives automatically. Rather than hand-tuning a fixed scalar between the two losses, the paper derives the combination from a maximum likelihood view in which each objective has its own learnable uncertainty parameter (σ_I and σ_D). The final objective scales each loss by its inverse uncertainty and adds log terms for those parameters, so the balance adapts during training. At inference, the uncertainty parameters are discarded.
7. Select per task. The trained scorer assigns CoIDO scores to every sample in the full dataset. For each downstream task, the lowest-scoring (most important-and-diverse) top-γ fraction is chosen from that task's candidate pool, keeping task coverage balanced.

Why This Matters

Impact on research. The paper targets a practical bottleneck in multimodal training: selection methods that cost as much as full training defeat their own purpose. By showing that a scorer trained on only 20% of data can select a 20% subset that retains 98.2% of full-data performance, and that the scorer transfers across datasets without retraining, the work argues that importance and diversity should be optimized jointly rather than sequentially.

Real-world applications.

• Reducing the hardware budget needed to fine-tune multimodal assistants for teams without large GPU clusters, given the reported cost of over 20 GPU hours per epoch on 8× A100 40GB GPUs for full LLaVA-1.5-7B fine-tuning.
• Curating domain-specific visual instruction corpora (for example, human-annotated collections such as Vision-Flan) where only a fraction of samples can be afforded during training.
• Reusing a single trained scorer as new in-domain data arrives, since the paper states the scorer learns the data distribution from a small subset and can be applied to new in-domain data without retraining.
• Reducing data storage, annotation review, and preprocessing costs in multimodal data pipelines by prioritizing the most informative samples.

Industry relevance. Organizations deploying vision-language assistants under compute or latency constraints can use the reported efficiency profile — 20% MLLM Training Data Cost and 4.2E total FLOPs — as a template for cheaper iteration. The publicly released code (https://github.com/SuDIS-ZJU/CoIDO) further lowers the barrier to adoption.

Future Directions

• How far does the scorer transfer? The paper demonstrates transfer from LLaVA-665K to Vision-Flan, including a scorer that outperforms an in-distribution one, but it is an open question where transfer breaks down — for example, to strongly dissimilar domains.
• Scorer architecture and feature choices. The paper compares MLP, Transformer-based, and attention-only scorers, and reports that MLP performs strongly because of the extracted features; whether richer features or architectures change the picture at larger scale is not settled here.
• Scaling to larger models and settings. The paper notes additional results on LLaVA-13B and LLaVA-150K are presented in its appendices; extending the conclusions to much larger model sizes and different training regimes remains to be established.
• Behavior at larger selection ratios. The paper reports the performance gap narrows as γ approaches 40% and that randomness matters more above 50%; whether coupled optimization offers similar benefits in the high-budget regime is left open.
• Sensitivity to clustering configuration. The paper references appendices on feature usage and clustering methods, but the main text does not report how sensitive final selection quality is to the number of clusters M or to the choice of spectral clustering.

Target Audience

This paper is most useful to machine learning researchers and engineers working on data-efficient training of multimodal large language models, especially those building or curating visual instruction datasets. Practitioners who need to fine-tune vision-language models under limited GPU budgets will benefit from the efficiency framing and the reported FLOPs and data-cost comparisons. Readers interested in data selection, coreset construction, or multi-task loss balancing (via homoscedastic uncertainty) will also find the coupled optimization formulation relevant, though some familiarity with instruction tuning and clustering is assumed.

Authors’ abstract

Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity. We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling efficient and scalable data selection. In our experiments, we trained the CoIDO scorer using only 20 percent of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20 percent subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2 percent of the performance of full-data fine-tuning, on average.