Importance-Aware Data Selection for Efficient LLM Instruction Tuning

AI summary

Overview

• Research area: Natural Language Processing — instruction tuning and training-data selection for large language models.
• Technical level: Intermediate (requires familiarity with instruction tuning, next-token loss, embeddings, and in-context learning).
• Scope: The paper proposes a training-free data-selection metric called Model Instruction Weakness Value (MIWV), and shows that fine-tuning on the top 1%–15% of instruction samples ranked by MIWV can match or beat fine-tuning on the full dataset.

What This Paper Is About

Instruction tuning works better when the training data is chosen well, but most existing selection methods score data quality in the abstract rather than asking what a specific model actually needs to learn. The authors introduce a metric that measures how much a given instruction sample exposes a weakness in a particular LLM, by comparing the model's loss on a prompt with and without a retrieved one-shot example. The goal is a simple, fully automated way to pick the small subset of data that most improves a given model.

Key Contributions

1. A universal, model-agnostic data selection method based on sample importance that requires no model training and no dependence on an external LLM (such as ChatGPT). The authors describe it as simple, efficient, and fully automated.
2. The Model Instruction Weakness Value (MIWV) metric, which quantifies how important an instruction sample is for improving a given LLM. It is computed as the difference between the prompt loss (instruction plus a retrieved one-shot example) and the plain instruction loss. To the authors' knowledge, this is the first work to build a quantitative evaluation metric out of the model's own in-context learning behaviour.
3. Empirical demonstration that a small, high-quality subset suffices: instruction tuning with the top 1% of samples by MIWV can outperform training on the full dataset, and the selected subsets also beat several advanced selection baselines.
4. Analysis of what high-MIWV data looks like, using t-SNE visualisation of Alpaca instruction embeddings and GPT-4 quality ratings across six dimensions (Complexity, Scope, Depth, Creativity, Expertise, Clarity).

Main Findings

• Top 1% can beat 100%: On the Alpaca dataset, the model trained on the top 1% selected by MIWV (520 samples) outperformed the officially trained Alpaca model on the Vicuna test set, and the 10% selection achieved the highest win rate. On WizardLM, the 1% selection (636 samples) also outperformed the model trained on the full dataset on the Sinstruct test set.
• Pairwise win rates above 1.0 across the board (Table 1): For LLaMA2-7B on Alpaca, win rates were 1.127 (1%), 1.214 (5%), 1.228 (10%), and 1.248 (15%) against the 100% baseline, whose value is 1.000. For LLaMA2-13B on Alpaca: 1.063, 1.160, 1.200, 1.256. For LLaMA2-7B on WizardLM: 1.048, 1.096, 1.114, 1.153. For LLaMA2-13B on WizardLM: 1.043, 1.050, 1.107, 1.118.
• Open LLM Leaderboard improvements: On WizardLM with LLaMA2-7B, the 15% subset raised ARC from 58.61 (100% baseline) to 60.85, and the 1% subset raised it to 60.14. On Alpaca with LLaMA2-7B, the 15% subset raised the Leaderboard average from 55.25 to 57.08; on WizardLM with LLaMA2-7B the average rose from 55.02 to 57.07 at 15%.
• AlpacaEval gains: For LLaMA2-7B on Alpaca, AlpacaEval rose from 27.75 (100% baseline) to 39.50 (1%) and 39.87 (5%). For LLaMA2-13B on Alpaca, it rose from 35.00 to 41.30 (1%) and 48.24 (5%). AlpacaEval values for the 10% and 15% settings are listed as "-" (not reported).
• Best ratios differ by setting: For the LLaMA-7B model, the WizardLM 20% model had the best win rate, while for LLaMA2-7B, WizardLM 15% performed best. The authors note that as the training-data proportion grows, the overall win rate tends to decline, which they attribute to interference or harmful noise among samples.
• Outperforms competing selectors (Table 2, GPT-4 judge, WizardLM test set): MIWV reached win rates of 1.119 (1%), 1.211 (5%), 1.178 (10%), and 1.234 (15%), the highest in the table. For comparison, RECOST scored 1.092/1.138/1.147/1.110; DiverseEvol 1.018/1.142/1.137/1.165; Superfiltering 0.972/1.133/1.101/1.193; IFD Score 0.794/0.853/0.761/0.927. MIWV's selection time was 85 minutes; only Superfiltering was faster at 8 minutes, while DiverseEvol took 300 minutes and Deita 282 minutes.
• Ablations confirm the design choices: Training on randomly selected data, on data with high prompt loss, or on low-MIWV data all performed worse than the official Alpaca model, with low-MIWV selection producing the lowest win rate. Using Bge-en-large, Multilingual-e5-large, or Gte-base-en-v1.5 as the retrieval embedding model all still beat the full-dataset baseline; Bge-en-large was best at 5% and 10%, and Multilingual-e5-large was best at 15%.
• In-context learning itself helps: Replacing the clustering step of the IFD Score method with ICL-based one-shot retrieval improved its overall win rate from 0.939 to 1.017, while MIWV reached 1.140 (Table 3).
• Works across model families: The method was applied to Qwen2.5-7B and Qwen2.5-14B, whose architectures differ from LLaMA/LLaMA2; small amounts of MIWV-filtered data again outperformed training on the complete dataset (LIMA test set, GPT-4 judge).
• What high-MIWV data looks like: t-SNE visualisation of Alpaca embeddings shows top-5% MIWV samples spread uniformly across the instruction space, while the lowest-5% samples cluster in specific regions. Low-MIWV samples tend to be basic tasks such as editing punctuation, words, or simple sentences; high-MIWV samples include both basic and complex tasks such as storytelling and explaining phenomena. GPT-4 ratings of 100 randomly sampled top and least 5% samples showed higher MIWV data winning on Complexity, Scope, Depth, Creativity, and Expertise, with comparable Clarity.
• Case study: On a mathematical question, the LLaMA2-7B model trained on the Alpaca 1% subset gave the correct answer, while the Alpaca-100% model did not.

Methodology in Plain English

The pipeline has three steps:

1. Find a one-shot example for every training sample. Each instruction sample is turned into a single vector by averaging the token embeddings from an embedding model (Bge-en-large in the main experiments). For each sample, the most similar other sample is found using cosine similarity, and that sample's instruction–response pair becomes its one-shot example. If the dataset contains nothing similar, an irrelevant example is used, and a high score in that case still marks the sample as worth selecting, which helps keep the chosen subset diverse.
2. Score each sample. The model's loss is computed twice: once when the instruction is given alone, and once when the retrieved one-shot example is prepended to the prompt (this second value is called the prompt loss). MIWV is simply prompt loss minus plain loss. A large difference means the model does badly when it is shown a related example, i.e., it lacks the underlying ability that this instruction targets — so the sample is valuable for training. No fine-tuning is needed; only inference to compute losses.
3. Select and tune. Samples are ranked by MIWV and the top 1%, 5%, 10%, or 15% are kept as the high-quality subset, which is then used for instruction tuning.

Experimental setup: Training data were Alpaca (52,002 instruction samples) and WizardLM (63,655 samples). Test data were Vicuna, Koala, WizardLM, Self-instruct, and LIMA, totalling 1,030 instruction samples. Models fine-tuned were LLaMA-7B, LLaMA2-7B, and LLaMA2-13B (plus Qwen2.5-7B/14B in the cross-series analysis), using the Alpaca codebase. For LLaMA-7B the setup used the Adam optimizer with learning rate 2×10⁻⁵, batch size 128, three epochs, and maximum input length 512 for Alpaca / 1,024 for WizardLM; LLaMA2-7B and LLaMA2-13B used maximum input length 2,048 for both datasets. Hardware was PyTorch 2.0.1 on a Linux server with 984GB RAM, an Intel Xeon Platinum 836

Authors’ abstract

Instruction tuning plays a critical role in enhancing the performance and efficiency of Large Language Models (LLMs). Its success depends not only on the quality of the instruction data but also on the inherent capabilities of the LLM itself. Some studies suggest that even a small amount of high-quality data can achieve instruction fine-tuning results that are on par with, or even exceed, those from using a full-scale dataset. However, rather than focusing solely on calculating data quality scores to evaluate instruction data, there is a growing need to select high-quality data that maximally enhances the performance of instruction tuning for a given LLM. In this paper, we propose the Model Instruction Weakness Value (MIWV) as a novel metric to quantify the importance of instruction data in enhancing model's capabilities. The MIWV metric is derived from the discrepancies in the model's responses when using In-Context Learning (ICL), helping identify the most beneficial data for enhancing instruction tuning performance. Our experimental results demonstrate that selecting only the top 1\% of data based on MIWV can outperform training on the full dataset. Furthermore, this approach extends beyond existing research that focuses on data quality scoring for data selection, offering strong empirical evidence supporting the effectiveness of our proposed method.