UniData: Universal Multimodal Instruction Generation Pipeline

AI summary

Overview

Research area: Multimodal large language models (MLLMs), synthetic instruction-data generation, and any-to-any multimodal modeling.

Technical level: Intermediate. The paper assumes familiarity with instruction tuning, tokenizers, projection layers, and multimodal benchmarks, but its central idea (a modular generation pipeline) is easy to follow.

Scope: This paper introduces UniData, a pipeline that turns a user's short keyword requirements into large-scale multi-round, nine-modality instruction data, plus UniDataset, a 20,000-entry dataset built with that pipeline.

What This Paper Is About

Training Multimodal Large Language Models requires large amounts of high-quality instruction data, but collecting and annotating such data by hand is expensive. Existing automatic generation methods, such as Self-Instruct, VIGC, and Multimodal Self-Instruct, support only limited modalities and offer weak support for multi-round dialogue. The paper's goal is a universal pipeline that expands a few user keywords into diverse events and then generates multi-round instructions spanning nine modalities, while filtering out repetitive and irrelevant generations during inference.

Key Contributions

1. UniData, a universal multimodal instruction generation pipeline. From a single simple user requirement, UniData produces multi-round multimodal instructions, in contrast to prior frameworks restricted to language (Self-Instruct), language plus visual input (VIGC), or abstract images (Multimodal Self-Instruct).

2. UniDataset, a nine-modality instruction dataset. The dataset contains 20,000 instructions with nine distinct modalities on both the input and output sides: <language>, <image>, <music>, <emoji>, <code>, <map>, <link>, <math>, and <QR code>. The authors position it as addressing limitations in data quantity, quality, and copyright status, and as a foundation for training UniData or other any-to-any models.

3. A three-part generation design: a Diverse Event Generator (DEG) that expands keywords into diverse events, a Multimodal Instruction Generator (MIG) built on an any-to-any large model, and an instruction-flow correction step that uses correlations between rounds to suppress irrelevant and redundant content.

4. Empirical demonstration that the data is useful downstream. UniData is reported to reach state-of-the-art data quality on four GPT-guided metrics (reasonableness, clarity, detail, relevance), and the generated instructions improve the understanding and generation capabilities of other multimodal models after fine-tuning.

Main Findings

• Data quality beats prior pipelines on all four evaluated dimensions. On GPT-Guided metrics, UniData (LLaMA-3 7B backbone) scores 0.661 reasonableness, 0.527 clarity, 0.667 detail, and 0.682 relevance, versus Self-Instruct (GPT-4) at 0.624 / 0.444 / 0.594 / 0.520 and VIGC (Vicuna 7B) at 0.460 / 0.359 / 0.534 / 0.569. The largest margins are on clarity and relevance.

• UniData produces far more multimodal content and much longer dialogues. Average multimodal counts per instruction are 1.92 images, 0.74 music, and 7.37 other modalities, compared with no reported image/music/other content for the two baselines. Average rounds are approximately 17.5 for UniData, approximately 3 for Self-Instruct, and approximately 1 for VIGC.

• Generated data improves music understanding. Fine-tuning MuMu-LLaMA raises BLEU from 0.2675 to 0.2681, BLEU-4 from 0.2019 to 0.2076, METEOR from 0.3232 to 0.3330, ROUGE from 0.3376 to 0.3496, and BERT from 0.8932 to 0.8936.

• Generated data improves music generation. FAD drops from 16.69 to 16.51, CLAP rises from 0.3138 to 0.3192, and KL drops from 2.1358 to 2.1038.

• Generated data improves code, math, and text-based capabilities. On MBPP (with LLaMA 2), accuracy goes from 20.8 to 21.3. With Ovis2-1B, MathVista goes from 59.4 to 60.1, MathVision from 12.8 to 13.0, and MMLU-Pro from 20.3 to 21.9.

• The largest image-side gain is on interleaved understanding. Using SEED-X and SEED-Bench-2, single-image understanding goes from 26.19% to 27.15%, multi-image from 37.73% to 39.09%, and interleaved from 19.42% to 35.25%; image generation goes from 24.43% to 25.90%.

• Visual understanding benchmarks also improve. With Ovis2-1B, MMMU rises from 35.8 to 36.4, ChartQA from 17.4 to 17.9, and TextVQA from 58.9 to 60.1.

• Both pipeline components matter in ablation. Removing event expansion drops scores to 0.557 reasonableness, 0.518 clarity, 0.607 detail, 0.550 relevance, and yields only k instructions rather than N × k. Removing flow correction gives 0.648 / 0.489 / 0.610 / 0.654 while still generating N × k instructions. The full method reaches 0.661 / 0.527 / 0.667 / 0.682.

• UniDataset is broader than comparable datasets. Compared against LLaVA, VideoChat, FIRE, LAMM, mPLUG-DocOwl, GenHowTo, and MosIT, UniDataset supports nine input and nine output modalities and approximately 17.5 rounds on average, against at most approximately 4.8 rounds for prior work, with 20K instructions.

Methodology in Plain English

The pipeline is built around the idea that a user should be able to type a few keywords and receive a full instruction dataset, so it deliberately inserts randomness and modular tools between the keywords and the final data.

Step 1 — Turn keywords into diverse events. The authors curate a topic pool of 90 domains spanning e-commerce, lifestyle, sports, science, the arts, and more. Two topics are randomly paired and fed to GPT-4o to produce an event. Pairing gives (90 choose 2) = 4,005 unique combinations, which is the main source of diversity. Random emotion proportions (of seven distinct emotions) and a random occasion (formal or informal) are also added.

Step 2 — Write multi-round instructions. Events are used to prompt GPT-4o into generating multi-round instructions, with the number of rounds chosen randomly between 5 and 30. This produces the 20,000 multi-round instructions.

Step 3 — Predict where other modalities belong. Because the dialogues are text-only at this point, GPT-4o predicts which parts of the dialogue should carry image, music, emoji, code, map, link, math, or QR code content, and writes the appropriate prompt, label, query, or content string for each.

Step 4 — Realize the content with external tools. A toolkit generates the actual modality content: GPT-4o for language, Google Search for links, Google Maps for maps, DALL·E 3 for images, MusicGen for music, and the Apple Emoji Archive for emoji.

Model training within the pipeline. Text and text-representable modalities are tokenized with the LLaMA Tokenizer; images use the visual tokenizer and projection from Ovis-1.5; music uses a pre-trained Whisper tokenizer followed by an MLP projection. Special tokens <|modality|> and <|modality_end|> mark where each modality begins and ends. LLaMA-3 is the backbone, trained with causal attention. Because images and music have no natural text label, the prompts used to generate them serve as discrete training labels. The model is pre-trained following Ovis-1.5's training strategy and dataset, then fine-tuned with visual and music tokenizers frozen on a random 90% of UniDataset, holding out 10% for testing.

Step 5 — Correct the instruction flow during inference. After each new round is generated, its cosine similarity to the previous k rounds is computed. If similarity is below α1 = 0.05, the round is treated as irrelevant and the temperature is raised to sample more alternatives. If similarity is above α2 = 0.4, the round is treated as repetitive and the repetition penalty is raised. Anything in between is accepted as high quality and generation proceeds. Users can adjust these thresholds.

Detokenization. Text detokenizes directly for code, map, link, and math. Stable Diffusion 3 turns text into images, MusicGen turns text into music, emoji are retrieved by similarity search against an archive, and QR code content is converted directly using the qrcode Python package.

Why This Matters

Impact on research. The paper targets a concrete bottleneck in multimodal research: the expense and copyright difficulty of obtaining instruction data. It also frames its approach as a data foundation for any-to-any models, noting that NExT-GPT cannot release its training data due to copyright constraints and that Show-o is limited to end-to-end training for lack of a suitable dataset. UniData's modular four-stage design is also intended to accept new modalities without architectural changes, since the planner emits modality-agnostic slots.

Real-world applications.

• Bootstrapping domain-specific multimodal assistants in areas matched by the 90-topic pool, such as e-commerce, travel, cooking, fitness, or education, where a few keywords could seed a task-specific dataset.
• Generating training data for low-resource modalities like music or emoji, where annotated instruction data is scarce and the pipeline can delegate realization to MusicGen and the Apple Emoji Archive.
• Producing evaluation and instruction material that mixes formats, including maps, links, QR codes, math, and code, which is useful for benchmarking models on interleaved tasks — the setting where the largest reported gain appeared (19.42% to 35.25%).
• Cost reduction in dataset construction, since generation relies on public APIs and open-source models rather than manual annotation.

Industry relevance. The use of LLaMA-3, Ovis-1.5, Whisper, Stable Diffusion 3, MusicGen, DALL·E 3, OpenAI's API, Google Search, and Google Maps means the pipeline is assembled from widely available components, which lowers the barrier to reproducing and extending it inside product teams. The authors also insert human oversight at two points — hand-selecting the 90 seed topics and filtering implausible or near-duplicate events after expansion — to limit propagation of LLM-specific stylistic bias.

Future Directions

• Extending beyond nine modalities. The authors explicitly list video, 3D, motion, tables, biological data, and spoken input as modalities the current coverage omits, and the pipeline is designed so that adding a modality does not require changing the planner.

• Replacing rule-based correction with stronger reasoning. The limitations section states that long-horizon structured reasoning is handled only by simple correction rules based on cosine similarity thresholds, which suggests learned or more sophisticated flow control as an open direction.

• Reducing dependence on external tools. Because UniData is described as a text-centered orchestration pipeline rather than a native any-to-any model, per-modality quality is bounded by the external tools that realize each modality.

• Systematic bias auditing. The paper states that UniData inherits biases from its LLM backbones, search APIs, and modality generators, that these may enter UniDataset and be amplified downstream, and that systematic bias audits remain future work.

The paper's appendix also promises a user study, out-of-distribution results, ablations on multimodal balance and user-conditioned generation, and a scope comparison with recent instruction-data pipelines, but the truncated content available here does not report those results.

Target Audience

Researchers and engineers working on multimodal large language models, instruction tuning, and synthetic data generation will benefit most, particularly those who need scalable ways to produce instruction data across more than text and images. It is also relevant to practitioners building data pipelines with off-the-shelf APIs and open-source generators, and to readers interested in how inference-time correction rules can be used to maintain data quality over long multi-round generation.

Authors’ abstract

Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.