ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

AI summary

Overview

• Research area: Computer vision, specifically video generation and editing — first-frame-guided object insertion, with a vision-language model (VLM) supplying temporal interaction guidance.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video models, LoRA fine-tuning, cross-attention conditioning, and VLM-based evaluation.
• One-sentence scope: ALIVE is a combined dataset (35,800 editing pairs), diffusion editor, VLM guidance predictor, and benchmark for inserting objects that respond coherently to the actions already happening in a source video.

The work is by Zhenghong Zhou (University of Rochester), Zhe Lin (Adobe Research), Jiebo Luo (University of Rochester), and Yuqian Zhou (Adobe Research). It is listed as arXiv:2610.08779v1 [cs.CV], 06 Oct 2026.

What This Paper Is About

Existing video editors can add an object to a clip, but the object often just sits there instead of participating in what is happening around it — a mug should rise with the hand that lifts it, and dough should split as a knife cuts it. The core problem is twofold: interaction editing pairs are scarce, and the source video shows the action but not how the newly added object should respond to it.

ALIVE's goal is to make inserted objects "alive" using only three user inputs — a source video, an edited first frame that fixes the object's appearance and initial placement, and an object-only instruction that merely names the object. Everything about the object's subsequent motion or state change must be inferred.

Key Contributions

1. An interaction-focused dataset. 35,800 video editing pairs combining interaction-focused sources (3D-rendered, model-generated, real-world) with general insertion pairs from ROSE, plus multi-level prompt annotations (P0–P4) for studying which guidance helps editing.
2. A diffusion editing framework with VLM-predicted guidance. A first-frame-guided editor conditions on the source video, the edited first frame, and text, while a separately fine-tuned VLM predicts temporally localized interaction descriptions from the same inputs and aligns them to the corresponding video intervals.
3. The ALIVE-interaction benchmark. A 128-case benchmark for assessing interaction fidelity, source preservation, and visual coherence under a unified VLM-judge protocol, alongside evaluation on a general video object insertion benchmark of 103 cases.
4. Ablations on data composition, prompt level, training budget, and backbone. These isolate the contributions of interaction-focused training data, prompt detail, temporal prompt routing, and model choice.

Main Findings

• Large gains on interaction-focused insertion. Without VLM guidance, ALIVE scores 86.71 Overall on the ALIVE-interaction benchmark versus 60.24 for the strongest evaluated baseline (Señorita) — a 26.46-point gap, described in the abstract as a 43.9% improvement. Its motion/propagation score reaches 86.13 against 36.52 for Señorita.
• General insertion quality is retained. On the 103-case general benchmark, ALIVE scores 92.28 Overall versus 88.39 for Señorita, a 3.89-point (4.4%) improvement in the abstract's framing.
• VLM-predicted P4 guidance adds a further gain. Predicted P4 raises the ALIVE-interaction Overall score from 86.71 to 87.66 (a 0.95-point improvement) with no additional user input, and the paper reports gains in temporal consistency and background preservation as well as motion/propagation.
• The gain does not transfer to general insertion. On the general benchmark, P1 (92.28) stays slightly ahead of predicted P4 (91.71), though both beat the evaluated baselines.
• Prompt form matters. On the interaction benchmark, P0 (no text) scores 85.38 and P1 (object identity only) scores 86.71, showing the editor can infer some interaction from visual inputs alone. Detailed P3 reaches 87.37 and temporally chunked P4 reaches 88.69 — the best annotated format. P2 (86.68) provides little benefit over P1. Among VLM-predicted variants, predicted P4 is highest at 87.66 (VLM P2 at 87.33, VLM P3 at 87.17).
• Interaction-focused data is necessary. Training on ROSE general editing pairs alone yields only 60.12 on the interaction benchmark, versus 82.35 for the four-source mixture. The mixture stays within 0.46 points of model-generated-only training on interaction while improving general insertion by 7.61 points (84.36 to 86.93).
• Different sources favor different capabilities. Model-generated pairs are strongest for interaction (82.81 interaction Overall), while ROSE is strongest among individual sources for general insertion (84.36). The ROSE-only interaction score is 60.12.
• Longer training helps. Going from 5k to 20k steps with the P1-only LTX-2.5 editor raises interaction Overall from 82.35 to 86.46 and general insertion Overall from 86.93 to 89.20.
• Backbone trade-off. At 5k P1-only steps, Wan 2.2 I2V (14B) scores 89.82 interaction / 89.44 general, above LTX-2.5 (22B) at 82.35 / 86.93, and above Wan 2.2 TI2V (5B) at 71.85 / 81.54. LTX-2.5 is far cheaper to train: 4.95 s per update on 16 A100 80GB GPUs with a 5k-step run at 8.1 hours, versus roughly 45.4 s per expert update for Wan 14B on 32 A100 80GB GPUs (an extrapolated ~63 hours for 5k updates). LTX uses 8×32×32 VAE compression versus 4×8×8 for Wan, shortening the video-token sequence.
• Inference cost is reported per method. Measured on one A100 80GB per video: LTX 22B with P1 runs 81 frames at 768×512 in 154.16 s with 39.43 GiB peak; with predicted P4 it takes 184.50 s at the same peak; the VLM P4 predictor takes 22.75 s and 52.27 GiB, giving an estimated 207.25 s per edit when the separately measured stages are summed.
• Qualitative behavior. Figure 4 shows ALIVE + VLM moving inserted objects with hands in book-handling and basket-carrying samples, where baselines can leave objects stationary, omit them, or distort their shape. Figure 5 shows predicted P4 ("lower it onto the head", then adjusting straps) avoiding strap artifacts around the eyes that appear with P1, and better seating a seedling in soil.
• Evaluation is judge-based. A reference-aware VLM judge (gpt-5.6-sol, high reasoning effort) scores six weighted dimensions on 0–100: task fidelity (0.25), object identity (0.15), motion/propagation (0.20), background preservation (0.15), temporal consistency (0.15), and visual quality (0.10). Automatic metrics following EditVerse (PickScore, frame-level CLIP, video-level ViCLIP, TC-CLIP, TC-DINO) are also reported but do not enter the VLM Overall score. The paper does not report a human perceptual study.

Methodology in Plain English

Building the data. The authors needed paired videos that differ only in whether the target object is present, while the surrounding action stays the same. They assembled four sources. For 3D-rendered content, they selected interaction segments from SpaceTimePilot assets and rendered each scene from multiple first- and third-person viewpoints, with and without the object, keeping the scene, camera trajectory, and animation aligned — yielding exact pairs and masks. For model-generated content, they used GPT to design prompts covering common interaction types and synthesized object-present videos with Wan2.2, then used SAM3 to segment and track the object and UnderEraser to remove it and create the paired source video. For real-world content, they curated interaction clips from ARCTIC, HOCap, HOI4D, and HOT3D, using provided segmentations or projected object geometry as masks and UnderEraser for removal. They also reversed ROSE removal pairs so the object-absent result becomes the source and the original video becomes the target, covering cases where the object should simply stay put or cast shadows and reflections. Automated quality checks with Qwen3.6-27B and GPT-5.6 Luna/Sol filtered out clips with no visible interaction, poor quality, or visibly incorrect removal.

Annotating guidance levels. Each pair carries prompt annotations at several levels of detail. P0 uses no text. P1 names only the added object. P2 describes the principal interaction; P3 details its sequence and outcome; P4 describes temporal chunks, each with a description and an inclusive frame range, and can span the whole clip when no decomposition is needed. P0 and P1 give the editor no interaction description at all, so it must infer behavior from the source video and edited first frame. Qwen3.6-27B assisted annotation using ordered target frames, masks, and object/action metadata, with selective GPT verification and refinement.

The editor. ALIVE adapts LTX-2.5. A frozen video encoder maps source and target videos to aligned latent grids. Clean source latents are channel-concatenated with noisy target latents at matching spatiotemporal positions, so source context is supplied without adding video tokens; the target half of the input projection is initialized from the pretrained weights and the source half to zero. The encoded edited first frame anchors the first target frame and its tokens stay clean throughout denoising. The backbone's standard text path handles P0–P3 as one shared context. For P4, each chunk description is encoded independently, and video-to-text cross-attention is weighted by each video token's normalized temporal overlap with that chunk's interval, with a length normalization so longer descriptions do not gain attention mass merely by being longer. Zero overlap masks the context entirely. All video chunks are denoised jointly; only cross-attention routing changes. Training uses rank-128 LoRA on the 35,800 pairs with 81-frame clips at 832×480, AdamW at 1e-4 with linear decay, global batch size 16, 20,000 updates, and 16 GPUs (A100 80GB, H100, or H200). Each sample draws a prompt level with probabilities 0.1/0.3/0.2/0.2/0.2 for P0–P4 and then a pair with a valid annotation at that level, optimized with the native flow-matching objective over unconditioned target tokens. Inference generates 81 frames at 768×512 in 30 denoising steps.

The VLM predictor. Qwen3.6-27B is LoRA-adapted (rank 64, alpha 128, dropout 0.05) with separate predictors for P2, P3, and P4. Inputs are 21 uniformly sampled source frames, the edited first frame, and P1; training uses cross-entropy on response tokens only, with the 2,048-update checkpoint used at inference. Annotation may consult the target video, but prediction never does. The predicted P4 text replaces P1 as the editor's text condition and is routed to its predicted intervals, so users supply no interaction description, masks, or trajectories.

How it is judged. Every method is scored by the same VLM judge on 16 uniformly sampled output frames including both endpoints, with source, reference, candidate, the edited first frame, and an object-addition instruction presented side by side. Scores are integers 0–4 per dimension, converted to a 0–100 scale using the fixed weights. The authors note that plausible interactions differing spatially or temporally from the reference are allowed, while missing interactions, broken contact, impossible motion, and source corruption remain errors. Mask-guided configurations (VACE, LoRA-Edit) are excluded because per-frame masks would supply the object states the editor is supposed to infer, and instruction-only editors are excluded because text specifies initial placement less precisely than the edited first frame.

Why This Matters

Impact on research. The paper reframes object insertion as an interaction-modeling problem rather than a compositing problem. Its most transferable idea is the decoupling of interaction understanding from video synthesis: a VLM predicts what should happen and when, and the diffusion editor is conditioned on that prediction through temporally routed cross-attention. The multi-level P0–P4 annotation scheme also provides a controlled way to ask how much textual specification an editor actually needs, which prior insertion work had not isolated. The released benchmark and the ablation across data sources give the field a way to separate "can it insert?" from "can it interact?"

Real-world applications:

• Advertising and product video, where a product must be shown being picked up, worn, opened, or used in footage where it originally was absent.
• Film and post-production visual effects, adding props into existing plates and having them react to actors' hands and tools.
• E-commerce and virtual try-on, inserting garments or accessories that respond plausibly to body motion.
• Robotics and embodied-AI data generation, synthesizing training video where objects respond to manipulation actions.

Industry relevance. The authors are affiliated with Adobe Research, and the work sits squarely in the professional video-editing tooling space, where the practical constraint is that users want an interaction-aware result without labeling masks or drawing trajectories. The paper's efficiency measurements matter here: LTX-2.5 trains roughly an order of magnitude faster per update than Wan 14B in their setup, even though Wan 14B scores higher at the same 5k-step budget, and the full edit pipeline is estimated at 207.25 s per edit excluding model-switching overhead. The authors also note that combining ALIVE with Self Forcing or EditStream could enable real-time streaming object insertion.

Future Directions

• Resolve ambiguity from underspecification. The source video and edited first frame do not fully determine geometry, contact conditions, or forces, so multiple object responses are plausible — a soft object may deform differently depending on stiffness, and a partially occluded grasp may leave a cup upright or tilting during lifting. The authors suggest incorporating explicit physical information for greater accuracy and control.
• Improve guidance reliability. Because the editor depends on VLM-predicted prompts, guidance errors can cause incorrect behavior. Making the predictor more reliable is stated as future work.
• Strengthen the editor on hard interactions. The paper reports that the editor still struggles with complex cloth deformations and rapid, large-angle rotations.
• Move toward real-time streaming. Combining ALIVE with Self Forcing or EditStream is proposed to enable real-time streaming object insertion.
• Open question the paper leaves: whether a stronger backbone closes the gap between LTX-2.5 and Wan 14B on interaction scores without sacrificing LTX-2.5's training efficiency, and why predicted P4 helps interaction editing but not general insertion.

Target Audience

Researchers and engineers working on video generation, video editing, and controllable diffusion models, particularly those interested in object insertion, first-frame-guided propagation, or VLM-in-the-loop generation. It is also relevant to practitioners building editing tools who need to know what conditioning signals actually change object behavior, and to benchmark designers interested in VLM-as-judge protocols with weighted, multi-dimension rubrics. Readers without background in diffusion models, LoRA adaptation, or cross-attention conditioning will find the method sections dense, though the task framing and results are accessible.

Authors’ abstract

Current video editors can insert objects but often struggle to make them participate in interactions such as being picked up or manipulated. We introduce ALIVE, a framework that makes inserted objects "alive" through coherent interactions with the source video's contents, using an edited first frame and an instruction naming only the added object. We curate 35,800 editing pairs combining 3D-rendered, model-generated, and real-world videos with general editing pairs from ROSE. Each pair differs in the target object's presence while preserving the surrounding action, teaching editors coordinated object behavior and source preservation. We further train a vision-language model (VLM) to predict interaction guidance from the same inputs. We introduce the ALIVE-interaction benchmark to assess interaction fidelity, source preservation, and visual coherence using a unified VLM-based protocol, and evaluate on the general video object insertion benchmark. Without VLM guidance, ALIVE improves Overall over the strongest evaluated baseline by 43.9% and 4.4% on the two benchmarks, respectively. VLM-predicted guidance further improves the ALIVE-interaction score by 0.95 points without additional user inputs.