A Unified Moral-Value Dataset for Instruction Tuning

AI summary

Overview

• Research area: Natural Language Processing — value alignment and instruction tuning for large language models.
• Technical level: Intermediate (requires familiarity with instruction tuning, supervised fine-tuning, and classification metrics, but the pipeline is described step by step).
• Scope: The paper merges three existing moral-value datasets into a single instruction–response corpus, trains models to fill missing value annotations, and tests how mixing that corpus with general instruction data affects both general-task and value-alignment performance.

What This Paper Is About

Large language models can be steered by instruction tuning, but almost no instruction-tuning datasets are designed around moral scenarios and behaviors, leaving a gap between flexible training methods and available value-oriented data. The authors build a unified moral-value dataset by collecting three existing moral datasets, converting them into a common schema, generating the missing annotations, and wrapping everything into instruction–response format. They then fine-tune a language model on mixtures of this dataset and a general instruction dataset to see whether value-oriented ability improves without losing general-task performance.

Key Contributions

1. The authors review and collect three moral-value datasets (ETHICS, UNIMORAL, and SOCIAL-CHEM-101) and integrate them into a unified dataset with a consistent format and schema, then train a ModernBERT model to impute missing values.
2. They design prompting strategies to generate instruction–response templates and combine them with the unified dataset to construct a moral-value instruction-tuning dataset.
3. They fine-tune large language models on this dataset under different mixing ratios with a general dataset, observing that general-task performance is preserved across configurations and that the mixing ratio influences value-alignment performance.
4. They release the dataset publicly at https://huggingface.co/datasets/teohzzh/value-for-instruction-tuning.

Main Findings

• Ethics-label generation works well overall: The Normative Ethics generator (ModernBERT-large, threshold t = 0.5) reaches an overall F1 of 0.9093, Average Precision of 0.9540, and Expected Calibration Error of 0.0608 across 16,440 instances.
• Performance varies by ethical category: The strongest category is Virtue (F1 = 0.9738), followed by Deontology (0.9212) and Utilitarianism (0.9168); Commonsense is weakest (F1 = 0.7946) and shows the highest calibration error (0.1143).
• Ordinal modeling helps the moral-foundations generator: The ordinal configuration E2.3 achieves the best weighted F1 (0.5334), and all ordinal configurations (E2.3, E2.6, E2.7) beat the baseline classification model (E2.2, F1 = 0.4895), which the authors take as evidence that the labels are inherently ordered.
• Focal loss gives the best ordinal consistency: Configuration E2.4 (focal loss, gamma = 2.0) reaches the highest Quadratic Weighted Kappa (0.3733); all models clear the 0.2 threshold for fair agreement, and within-one accuracy stays above 0.94 across configurations, meaning most errors fall between adjacent classes.
• General-task performance is preserved: On the OLMES benchmark, all trained models outperform the base model (68.5 average), with configurations landing in the range of 70.8 to 71.2 — a range the authors describe as showing no substantial differences across mixing ratios.
• Instruction signals, not dataset composition, drive gains at this scale: Even training solely on the value dataset produces gains comparable to mixtures containing general task data, which the authors attribute to the instruction-tuning paradigm itself for a model of this size.
• Value-alignment results are non-monotonic: Value-action gap F1 rises slightly from 0.8485 at 0% value data to a peak of 0.8521 at 10%, declines to 0.8486 at 40%, and reaches its lowest point of 0.8378 at 70%, with a minor recovery afterward that remains below the 0% baseline.
• Format-following depends on general data: The base model and the model trained on 100% value data could not be evaluated because neither produced correctly formatted outputs for Task 1, which the authors read as evidence that general-task data is needed for the model to understand task structure and output format.
• Best balance is a small value-data share: A mixture with roughly 10% value data combined with a larger amount of general task data yields the best value-alignment performance for Qwen3-1.7B-Base, though the absolute differences between non-degenerate mixing ratios are modest.

Methodology in Plain English

The work proceeds in three stages. In Stage 1, the authors filter three publicly available moral-value datasets and convert them into a common format of scenario, value framework, and label. Because the source datasets use different value frameworks — normative ethics (ETHICS, UNIMORAL) and Moral Foundations Theory (SOCIAL-CHEM-101) — the authors train two annotation models to fill in what is missing. The Normative Ethics generator is a ModernBERT model framed as binary classification over a scenario paired with one of six ethics types (Deontology, Utilitarianism, Justice, Virtue, Commonsense, Right-based), formatted in an NLI-style premise–hypothesis setup. The Moral Foundations generator uses the same backbone but predicts a rating from 0 to 4 for one of five foundation types (care–harm, fairness–cheating, loyalty–betrayal, authority–subversion, sanctity–degradation); the authors compare regression, classification, and ordinal formulations, using MSE, cross-entropy, focal loss, CORN loss, and class weights [2.75, 1.29, 0.46, 0.22, 0.28].

In Stage 2, a prompt is designed to have GPT-5.2 generate instruction–response templates in JSON format, encoding prompt-engineering principles as constraints. An instruction-tuning wrapper randomly combines these templates with the data, and the resulting dataset is fine-tuned using the same open-instruction pipeline used for the Tulu3 model, with supervised fine-tuning chosen as the primary approach. A small validation split is carved out to compute loss at specific training steps. To guard against degradation on general tasks, the value dataset is mixed with the TULU-3 SFT dataset at varying ratios.

In Stage 3, models are evaluated two ways. General tasks use the full OLMES benchmark (ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, MMLU, OpenBookQA, PIQA, Social IQA, WinoGrande). Value behavior uses the Value-Action Gap pipeline, which runs two tasks: stating how strongly the model agrees with a value, and choosing between two actions where one aligns with that value and one does not. The authors augment this pipeline with vLLM for structured output constraints and batch inference, addressing formatting failures and inference latency in the original implementation.

Why This Matters

The paper addresses a concrete gap: instruction tuning is a flexible and widely used training paradigm, but no instruction-tuning dataset had been specifically designed around moral scenarios and behaviors, leaving value-oriented resources hard to use in practice. By unifying heterogeneous moral datasets and releasing the result, the work lowers the barrier to running alignment experiments and provides a test bed for studying how training data composition affects both stated values and actual behavioral choices.

Real-world applications:

• Model alignment and safety training: Teams can fine-tune assistants on a ready-made instruction–response corpus instead of assembling moral data from scratch.
• Auditing deployed systems: The value-action gap setup lets developers measure whether a model's stated values match the actions it selects, a form of consistency check for consumer-facing chatbots.
• Content moderation and policy enforcement: Value-labeled scenarios could inform how models judge whether a behavior is appropriate or inappropriate in everyday social situations.
• Data engineering for alignment pipelines: The annotation-imputation approach, using ModernBERT classifiers to fill missing value labels, is a reusable technique for integrating datasets built on incompatible annotation schemes.

Industry relevance: The finding that instruction tuning itself, rather than dataset composition, drove most gains for a small model is practical for teams with limited compute, and it suggests a small share of value data (around 10%) is enough to improve value alignment without the cost of a fully value-focused training run.

Future Directions

• Broaden framework coverage: The dataset is limited to normative ethics and Moral Foundations Theory; the authors propose extending the pipeline to frameworks such as valueML, which is based on Schwarz Value Theory.
• Improve language and cultural coverage: The source datasets are primarily written in English, so the unified dataset has limited coverage of other languages and cultural perspectives.
• Reduce annotation noise: Labels produced by the data generators may introduce errors into the unified dataset, which the authors identify as a source of noise worth addressing.
• Test larger models and more runs: The downstream results come from a single model family and a single run per configuration, so whether the mixing-ratio effects hold for larger models and across random seeds remains an open question, and the authors explicitly describe their initial experimental results as not conclusive.

Target Audience

Researchers and practitioners working on value alignment, instruction tuning, and dataset construction for large language models, particularly those interested in how training data composition affects moral reasoning and behavioral consistency. It is also useful for engineers building alignment pipelines who need a ready-to-use moral-value instruction dataset, and for readers who want a concrete worked example of merging heterogeneous annotated datasets through trained annotation models.

Authors’ abstract

Large language models (LLMs) have developed rapidly and become valuable tools in everyday life. However, how to align LLMs to a particular set of human values is still an open problem. Recent studies show that instruction tuning has strong potential for zero-shot tasks and may serve as an effective approach to addressing value alignment. Nevertheless, although many datasets for instruction tuning already exist, they are not specifically designed around moral scenarios and behaviors. We construct a unified moral-value dataset that can be directly used for instruction tuning. This dataset is built upon existing moral-value datasets by merging them into a unified corpus and converting them into an instruction-response format. We show that training on a mixed dataset combining general task datasets with our dataset preserves general-task performance, and we report preliminary observations on how the mixing ratio affects value-oriented task performance. Our work provides a moral-value dataset for instruction tuning and offers a useful resource for further alignment research. The dataset is available at https://huggingface.co/datasets/teohzzh/value-for-instruction-tuning.