MotionEdit: Benchmarking and Learning Motion-Centric Image Editing

AI summary

Overview

Research area: Computer Vision — specifically instruction-guided image editing with diffusion / flow-matching generative models, combined with reinforcement-learning-style post-training and optical-flow motion estimation.

Technical level: Advanced. The paper assumes familiarity with flow-matching models, DiffusionNFT-style policy optimization, optical flow estimation, and MLLM-as-judge evaluation.

Scope: One sentence — the paper defines the new task of motion-centric image editing, releases the MotionEdit dataset and MotionEdit-Bench benchmark for it, and proposes MotionNFT, a motion-guided fine-tuning framework that improves two existing editing models on this task.

What This Paper Is About

Existing text-guided image editing models handle static appearance changes (color, texture, object presence) well but often fail when the instruction requires changing how a subject moves, poses, or interacts with something. The paper argues the root cause is data: existing editing datasets and benchmarks either contain no motion edits at all or only small amounts of low-quality ones (unfaithful targets, artifacts, shifted viewpoint or scale). The goal is to build a high-quality motion-editing dataset and benchmark, and to train models to actually execute the instructed motion while preserving identity, structure, and physical plausibility.

Key Contributions

1. A formal definition of the new task. Motion image editing is defined as editing that modifies the action, pose, or interaction of subjects and objects per a textual instruction, while (1) faithfully reflecting the intended action, (2) remaining physically plausible under articulated constraints (e.g., "slightly open his eyes"), and (3) keeping appearance, background, and viewpoint consistent.

2. The MotionEdit dataset and MotionEdit-Bench benchmark. 10,157 motion-editable frame pairs extracted and validated from continuous video frames rather than synthesized: 6,006 samples from Veo-3 and 4,151 from KlingAI. A random 90/10 split yields 9,142 training samples and 1,015 evaluation samples that constitute MotionEdit-Bench.

3. A motion-centric evaluation suite. Optical-flow-based Motion Alignment Score (MAS) for discriminative evaluation, MLLM-based generative metrics (Overall, Fidelity, Preservation, Coherence on a 0–5 scale), and pairwise Win Rate.

4. MotionNFT (Motion-guided Negative-aware FineTuning). A post-training framework that extends DiffusionNFT with optical-flow-based motion rewards, guiding both motion magnitude and direction.

Main Findings

• Motion editing is hard for current models. Across 9 open-source models benchmarked on MotionEdit-Bench, the strongest prior methods were Step1X-Edit (Overall 4.02, MAS 52.98, Win Rate 61.14) and BAGEL (4.10, 51.83, 61.46). Diffusion-based editors performed worst: Instruct-P2P (1.30, 34.15, 16.09), AnyEdit (1.31, 35.11, 16.88), MagicBrush (1.50, 44.24, 19.51), UltraEdit (2.42, 47.18, 28.33), UniWorld-V1 (2.87, 55.37, 41.14).

• MotionEdit contains far larger motion than prior datasets. On 100 randomly sampled pairs per dataset, prior datasets (MagicBrush, AnyEdit, InstructPix2Pix, UltraEdit, OmniEdit) show motion magnitude typically around 0.05, while MotionEdit shows 0.19 — 5.8× greater motion than MagicBrush and OmniEdit, and 3× that of UltraEdit.

• MotionNFT improves generative quality on FLUX.1 Kontext. Overall rose from 3.84 to 4.25 (+10.68%), with Fidelity up +0.44 and Coherence up +0.42. Fidelity went 3.89 → 4.33 and Preservation 3.79 → 4.16.

• MotionNFT improves generative quality on Qwen-Image-Edit. Overall rose from 4.65 to 4.72, Fidelity from 4.70 to 4.79, Preservation from 4.59 to 4.63, Coherence from 4.66 to 4.74.

• Motion alignment improves on both base models. MAS rose from 53.73 to 55.45 on FLUX.1 Kontext and from 56.46 to 57.23 on Qwen-Image-Edit.

• Pairwise preference improves. The paper's text reports FLUX.1 Kontext win rate rising from 57.97% to 65.16% (+12.40%), and Qwen-Image-Edit from 72.99% to 73.87%. Note that Table 1 lists 57.71 → 64.95 for FLUX.1 Kontext and 72.80 → 73.67 for Qwen-Image-Edit, and Table 3 lists the same Table 1 values; the text and tables do not fully agree.

• General editing ability is preserved. On ImgEdit-Bench (8 subtasks), FLUX.1 Kontext's Overall improved from 3.26 to 3.50 and Qwen-Image-Edit's from 4.08 to 4.20 after MotionNFT training.

• Motion reward beats MLLM-only reward. Compared to training with the UniWorld-V2 MLLM-only reward, MotionNFT gave higher Overall, MAS, and Win Rate on both base models (e.g., FLUX.1 Kontext: 4.25 / 55.45 / 64.95 versus 4.20 / 54.58 / 63.76 for UniWorld-V2).

• Qualitative failure modes persist in baselines. The paper reports that even FLUX.1 Kontext and Qwen-Image-Edit often preserve the original pose or make only superficial appearance changes; UniWorld-V1 fails to edit subject postures and FLUX.1 Kontext produces severe identity distortions.

Methodology in Plain English

Building the data. Instead of asking a model to synthesize edited targets, the authors mine real motion from video. They start from human action datasets (HAA500, K400) but reject them for low resolution, motion blur, and rapid viewpoint shifts, and instead draw on two publicly released text-to-video collections — ShareVeo3 and the KlingAI Video Dataset. Each video is cut into 3-second windows, and the first and last frame of each window form a candidate "before/after" pair. Google's Gemini model then acts as an automatic quality judge, scoring each pair on three things: whether background, viewpoint, and lighting stayed stable; whether the motion or interaction change is real and non-trivial; and whether subjects are present, identifiable, and artifact-free. Only pairs passing all criteria are kept. A separate MLLM "rewrite" step converts the model's motion summary into a clean, imperative user-style instruction (e.g., "Make the woman turn her head toward the dog."), following the prompt refinement procedure of Wu et al. The kept pairs are then grouped into six motion types: Pose/Posture, Locomotion/Distance, Object State/Formation, Orientation/Viewpoint, Subject-Object Interaction, and Inter-Subject Interaction.

Measuring motion. For any input/edited/target triplet, a pretrained optical flow estimator (UniMatch) produces a per-pixel displacement field between input and edited image, and between input and ground-truth image. The two flow fields are normalized by the image diagonal so resolutions are comparable.

Training. MotionNFT builds on DiffusionNFT, which trains a flow-matching model with both a "positive velocity" it should move toward and a "negative velocity" it should avoid, weighted by a reward. The novelty is the reward itself. Three terms are combined: a motion magnitude consistency term measuring how far predicted displacement magnitudes deviate from ground-truth ones (using a robust L1-style distance with an outlier-suppressing exponent q in (0,1)); a motion direction consistency term measuring the cosine-based angular error between flow vectors, weighted by ground-truth motion magnitude per pixel; and a movement regularization term that penalizes near-static edits by comparing mean predicted and ground-truth magnitudes. These are summed with balancing constants into a composite distance, normalized and clipped to [0,1], converted to a continuous reward (1 minus the distance), and finally quantized into 6 discrete reward levels: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. That scalar feeds the standard optimality-reward normalization of DiffusionNFT.

Evaluation. MAS reuses the magnitude and direction consistency terms as a combined 0–100 score, and assigns MAS = 0 if predicted motion is nearly static relative to ground truth (below a threshold). Generative metrics use Gemini as an MLLM judge with prompts adapted from the "action" category of Luo et al. Win Rate comes from head-to-head pairwise comparisons.

Training setup. Base models are FLUX.1 Kontext [Dev] and Qwen-Image-Edit. Fully Sharded Data Parallelism (FSDP) is used for the text encoder plus gradient checkpointing. The reward is a 50/50 weighted combination of the optical-flow motion reward and the MLLM reward from Lin et al. A Qwen2.5-VL-32B-Instruct model served via vLLM on a separate node handles online MLLM scoring, while a lightweight UniMatch model (335.6M parameters) runs directly on the training nodes for the flow-based component.

Why This Matters

Impact on research. The paper reframes motion editing as a distinct task with its own supervision signal, rather than treating it as a subtype of appearance editing. It also provides a deterministic, flow-based metric (MAS) that complements subjective MLLM judging, and demonstrates that task-specific reward design during post-training matters — an optical-flow reward outperformed a general MLLM reward on the same base models.

Real-world applications:

• Frame-controlled video synthesis, where an edit defines a start and end frame that a video model must interpolate.
• Character animation, where a subject's pose or orientation must change while identity stays fixed.
• Interactive content creation and image retouching pipelines where users issue natural-language action changes.
• Data generation for downstream action- and interaction-recognition research, since the dataset is built from verified video-derived motion transitions.

Industry relevance. MotionNFT is presented as a post-training add-on that boosts motion fidelity on two strong open-source base models without degrading general editing ability — the kind of incremental training recipe that product teams can layer onto an existing editing model rather than replacing it. The project page is https://motion-edit.github.io, and the authors state that dataset, code, and evaluation toolkit will be released upon acceptance.

Future Directions

• Scaling the data pipeline. The authors state their video-driven construction pipeline can be extended to much larger video corpora; the current dataset is 10,157 pairs drawn from two T2V collections.

• Better motion editing generally. Motion editing "remains highly challenging" for existing state-of-the-art diffusion-based editing models even after MotionNFT, which suggests room for stronger motion representations or architectures.

• Comparing against closed-source models. The supplementary material mentions qualitative examples against both open-source and closed-source commercial models, highlighting failure cases and pointing toward future research; the full comparison results are in the appendix section not fully included in the provided content.

• Validating and refining the metrics. The appendix mentions human validation of the metrics and additional ablation experiments on MotionNFT components; those details are referenced but the corresponding numbers are not present in the provided truncated content.

• Reward design. Since the paper shows a flow-based reward beats an MLLM-only reward, an open question is whether richer motion-aware or hybrid reward signals could push alignment further.

Target Audience

Researchers and engineers working on text-guided image editing, diffusion / flow-matching generative models, and reinforcement-learning-style post-training for generative systems. It is also relevant to teams building video synthesis or character animation pipelines, and to dataset builders interested in video-mined, MLLM-filtered data construction. Readers without background in flow matching, optical flow, or reward-based fine-tuning will find the training sections dense.

Authors’ abstract

We introduce MotionEdit, a novel dataset for motion-centric image editing-the task of modifying subject actions and interactions while preserving identity, structure, and physical plausibility. Unlike existing image editing datasets that focus on static appearance changes or contain only sparse, low-quality motion edits, MotionEdit provides high-fidelity image pairs depicting realistic motion transformations extracted and verified from continuous videos. This new task is not only scientifically challenging but also practically significant, powering downstream applications such as frame-controlled video synthesis and animation. To evaluate model performance on the novel task, we introduce MotionEdit-Bench, a benchmark that challenges models on motion-centric edits and measures model performance with generative, discriminative, and preference-based metrics. Benchmark results reveal that motion editing remains highly challenging for existing state-of-the-art diffusion-based editing models. To address this gap, we propose MotionNFT (Motion-guided Negative-aware Fine Tuning), a post-training framework that computes motion alignment rewards based on how well the motion flow between input and model-edited images matches the ground-truth motion, guiding models toward accurate motion transformations. Extensive experiments on FLUX.1 Kontext and Qwen-Image-Edit show that MotionNFT consistently improves editing quality and motion fidelity of both base models on the motion editing task without sacrificing general editing ability, demonstrating its effectiveness. Our code is at https://github.com/elainew728/motion-edit/.