VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically automated construction of instruction-tuning data for multimodal large language models (MLLMs) and unified multimodal models (UMMs).

Technical level: Advanced. The paper assumes familiarity with vision-language models, diffusion/autoregressive image generation, LoRA fine-tuning, and benchmark suites such as MME and GEdit-Bench.

Scope (one sentence): The paper describes VidUEU-Agent, a four-stage LLM-driven pipeline that turns raw web and open-dataset videos into 100K instruction samples spanning multimodal understanding, editing, and unified tasks, and reports fine-tuning results across six benchmarks. A truncated version of the paper content was provided, so the appendix sections (prompt templates, full statistics, and the feedback-free evaluations on MMBench, MMStar, and VSI-Bench) are only partially visible.

What This Paper Is About

Progress in multimodal models is limited less by architecture than by the scarcity and cost of high-quality instruction data. Raw video is an abundant source of visual knowledge with real temporal dynamics, but it is redundant, noisy, and unstructured, so it does not directly yield task-oriented instructions. The paper's goal is to build an agent that automatically converts large collections of raw videos into instruction data that jointly supports understanding, editing, and unified (interleaved understanding-plus-generation) capabilities, and to show that fine-tuning on this data improves downstream benchmark scores.

Key Contributions

1. VidUEU-Agent, an agentic data-synthesis pipeline. A four-stage framework — global perception, keyframe mining, metadata construction, and instruction data construction — that uses a planning agent with tool access to convert raw videos into instruction samples for understanding, editing, and unified tasks.

2. Autonomous data iteration. The agent consumes downstream training results and updates its target data distribution over capabilities, domains, and task types, adjusting video-type sampling, keyframe budgets and scoring weights, and task-routing quotas between iterations. Each iteration trains on 30K samples; the final 100K dataset is scaled from Iteration 4's target distribution.

3. VidUEU-100K dataset. Built from 8,268 raw source videos yielding approximately 8,505 semantic segments, the dataset contains 100K instruction-tuning samples (40K understanding, 30K editing, 30K unified) plus nearly 20k dense metadata annotations, spanning nine task categories and five scenario categories.

4. Empirical validation. LoRA fine-tuning of InternVL3-8B (understanding subset) and Bagel-7B-MoT (each of the three capability subsets) produces consistent gains on MMT-Bench, MME, LLaVA-Bench, MMDU, GEdit-Bench-En, and WEAVEbench, supported by ablations, case studies, and iteration analysis.

Main Findings

• Understanding improves for both model families. Fine-tuned InternVL3-8B reaches 0.596 on MMT-Bench, 2369.3 on MME, 70.8 on LLaVA-Bench, and 61.1 on MMDU, versus 0.546, 2308.0, 69.0, and 53.1 for the InternVL3-8B baseline. Fine-tuned Bagel scores 0.651 on MMT-Bench (baseline Bagel: 0.645), 1900.2 on MME (baseline: 1573.6), 57.9 on LLaVA-Bench (baseline: 47.5), and 33.2 on MMDU (baseline: 28.5).

• Editing gains are consistent and larger on perceptual quality. On GEdit-Bench-En, fine-tuned Bagel reaches 7.565 G_SC, 7.328 G_PQ, and 6.951 G_O, compared with 7.358, 6.831, and 6.527 for Bagel — described as the best results among the compared models, including Step1X-Edit, Gemini-2.0-flash, and OmniGen.

• Unified performance rises but trails the strongest baseline. On WEAVEbench, fine-tuned Bagel improves Overall from 0.446 to 0.501, with 0.480 on Logic and 0.419 on Game, while OmniGen2 remains higher at 0.609 Overall. The authors attribute the modest Creation gain (0.475 to 0.501) to this supervision being better suited to structured transitions than open-ended creative generation.

• Iterative refinement trends upward. Across Iterations 1–4 the scores move from 29.7 to 32.8 (MMDU), 6.541 to 6.901 (GEdit), and 0.448 to 0.499 (WEAVE); Iteration 2 dips slightly on MMDU (29.2) while improving on the other two, which the authors read as early distribution updates benefiting generation first. The Final 100K run reaches 33.0, 7.065, and 0.526.

• Frame filtering and the agent structure matter most. In the ablation (Bagel trained on 10K samples per capability), removing Gate B frame-quality filtering causes the largest single-component drop: 24.7 MMDU, 5.175 GEdit, 0.404 WEAVE, versus 33.2, 6.951, and 0.501 for the full method. The no-agent variant, which uses fixed-interval frames and direct VLM question–answer generation, is worst overall at 22.0, 5.041, and 0.393.

• Long interleaved interaction remains unsolved. Case studies show improved instruction alignment and local-edit preservation, but errors in early turns of long image–text chains can propagate to later turns.

Methodology in Plain English

The agent processes each raw video in four stages, with three quality gates in between.

Stage 1 — Global perception. Videos are split into shots with PySceneDetect. A vision-language model summarizes each shot from its middle frame, the agent groups adjacent shots into chapters with chapter summaries, and then produces a whole-video summary. Gate A then removes shots judged uninformative or visually poor.

Stage 2 — Keyframe mining. Frames are sampled at a fixed interval from surviving shots, and Gate B combines tool-based quality statistics with a VLM check to discard bad frames. The agent plans which perception tools to run at the chapter level and then at the shot level, applies them to retained frames, and verbalizes the raw tool outputs into plain-language descriptions. It then scores every frame from 0 to 1 using the descriptions, captions, and the three-level summaries. A keyframe budget is set from video duration and distributed across shots, and within each shot the top candidates are selected by weighted facility-location greedy selection over image-embedding cosine similarities, which keeps keyframes informative and non-redundant.

Stage 3 — Metadata construction. Signals attached to each keyframe are checked against predefined rules and filtered. The agent then writes transitions describing changes between consecutive keyframes and a clip summary, verifies this text against the retained signals, and binds keyframes, captions, signals, transitions, and summary into a metadata record.

Stage 4 — Instruction data construction. Gate C scores each metadata entry for information content, logical coherence, and overall quality, and discards unsuitable entries. The agent then routes the entry to one or more task types (understanding, editing, unified), selects matching instruction templates, and requires the generator to cite the signal IDs it used so each sample is traceable to evidence. The prompts note that rejected cases include global rejections, quality failures, generation-unsuitable frames, and task mismatches.

Self-iteration. After each round of training and evaluation on 30K samples, the agent reflects on weaknesses in the data, compares against baselines and earlier iterations, and rewrites its target distribution — up-weighting underrepresented video types, changing sampling density and keyframe budgets, and down-sampling over-represented task types. Implementation uses DeepSeek-V3 as the agent LLM and Qwen3.5-9B as the VLM. Video sources include OpenVid-1M, Panda-70M, Wikimedia, YouTube, FineGym, BDD100K, and CodeVidCL.

Why This Matters

The work reframes multimodal data curation as a planning-and-verification problem rather than a prompting problem, and it argues that video — with its natural state transitions — is a better substrate for editing and interleaved supervision than static image–text pairs. It also contributes a concrete, reproducible-scale artifact (100K samples from 8,268 videos) and a public code release.

Real-world applications:

• Instruction-based photo editing tools. Adjacent keyframes share most scene content but differ in specific objects or attributes, which mirrors the "edit this, preserve that" structure of consumer editing assistants.
• Video-derived training data for robotics and procedural domains. The pipeline uses FineGym, BDD100K, and CodeVidCL as domain sources, indicating applicability to sports analysis, driving scenes, and procedural/tutorial content.
• Agentic data factories for model teams. The gate-and-route structure is a template for automating dataset construction where human annotation is the bottleneck.
• Interleaved assistants. Unified samples target models that must recall visual context, infer state changes, generate target states, and select consistent images across turns — the skill set behind multi-step visual dialogue.

Industry relevance: The pipeline is built from off-the-shelf components (PySceneDetect, open VLMs, tool-based detectors) and runs at scale, which is the shape industry data engines take. The iteration loop, which turns benchmark feedback into data-distribution policy, is directly transferable to production data flywheels.

Future Directions

• Long-horizon interleaved reasoning. The authors identify error propagation across long image–text chains as an open challenge; mitigating early-turn mistakes is an obvious next target.
• Stronger creative generation supervision. The small Creation gain (0.475 to 0.501) suggests current transition-based instructions do little for open-ended generation, motivating supervision designed for free-form synthesis.
• Closing the gap to specialist unified models. Fine-tuned Bagel reaches 0.501 Overall on WEAVEbench while OmniGen2 reaches 0.609, leaving headroom in capability balance and scale.
• Further iteration rounds and broader source coverage. Only four iterations plus a scaled final run are reported, and the paper's gate ablations indicate that source-level curation already removes many useless shots — suggesting both more feedback cycles and more aggressive source selection as directions.

Target Audience

Researchers and engineers building multimodal training data pipelines or unified multimodal models; dataset-construction teams interested in agentic, tool-grounded synthesis; and practitioners applying LLM agents to large-scale content processing. Readers wanting implementation details will need the appendix, since the main text supplies high-level stage descriptions and benchmark tables rather than full hyperparameters — the settings are deferred to Appendix G and are not visible in the provided excerpt.

Authors’ abstract

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.