Composing Concepts from Images and Videos via Concept-prompt Binding

AI summary

Overview

Research area: Computer Vision, specifically diffusion-based text-to-video generation, personalization, and visual concept composition.

Technical level: Advanced. The paper assumes familiarity with Diffusion Transformers (DiT), cross-attention conditioning, LoRA-style adaptation, and one-shot personalization.

Scope: This paper proposes Bind & Compose (BiCo), a one-shot method that binds visual concepts from images and videos to individual prompt tokens and then recombines those tokens to generate a new composed video.

What This Paper Is About

Existing methods for concept-driven video generation can struggle to extract complex concepts from visual inputs — especially overlapping objects, motion, and non-object attributes such as style — and are usually limited to animating an object from a single image using motion from a single video. The paper's goal is a single framework that decomposes concepts from both images and videos, lets a user selectively recombine those concepts through the text prompt alone (no masks), and produces a coherent video with good concept consistency, prompt fidelity, and motion quality.

Key Contributions

1. Hierarchical binder structure: A global binder plus per-block binders attached to the cross-attention layers of DiT blocks encode visual concepts directly into the prompt tokens that already correspond to them, enabling text-conditioned concept composition without explicit mask input.
2. Diversify-and-Absorb Mechanism (DAM): A Vision-language Model (Qwen2.5-VL) diversifies the one-shot training prompts while keeping key concept words fixed, and an extra learnable absorbent token absorbs concept-irrelevant visual details during binding (the absorbent token is discarded at composition time).
3. Temporal Disentanglement Strategy (TDS): Video concept training is split into two stages — first training on individual frames with no temporal concepts (matching the image-concept setting), then training on full videos with a dual-branch binder consisting of spatial and temporal MLPs fused by a learnable gate — to improve compatibility between image and video concepts.
4. Two-stage inverted training strategy: An initial stage trains the global binder with emphasis on high noise levels (above a threshold alpha) before full hierarchical training, described as giving better initialization and more stable optimization.

Main Findings

• Quantitative comparison: On 40 test cases using images and videos from the DAVIS dataset and the Internet, BiCo scored CLIP-T 32.66 and DINO-I 38.04, versus 31.60 and 32.78 for the prior-art baseline DualReal, 30.25 and 27.74 for DB-LoRA, 27.43 and 24.15 for DreamVideo, and 25.96 and 20.47 for Textual Inversion (the latter two methods were re-implemented on the same Wan2.1 base model).
• Human evaluation: With 28 volunteers rating on a 5-point Likert scale, BiCo received Concept Preservation 4.71, Prompt Fidelity 4.76, Motion Quality 4.46, and Overall Quality 4.64, compared to 3.10, 3.11, 2.78, and 3.00 for DualReal. The paper reports this as a +54.67% improvement in subjective Overall Quality over DualReal.
• Ablation of components: A separate human study with 24 volunteers showed Overall Quality rising from 2.34 for a global-binder-only baseline, to 2.81 with the hierarchical design, 3.26 after adding prompt diversification, 3.47 after adding the absorbent token, and 3.82 after adding TDS. Removing the two-stage inverted training strategy dropped Overall Quality to 2.58.
• Ablation of training strategy: In an additional ablation, training the global binder first (Two-stage) alone reached Overall Quality 3.61, and combining it with the inverted noise-level emphasis reached 4.40.
• Capacity beyond object swapping: BiCo supports extraction of non-object concepts such as style and motion, learning multiple concepts from a single input, and composing concepts from arbitrary image/video input types — capabilities the paper states previous methods lack.
• Additional applications: Beyond composition, BiCo can decompose concepts (for example, keeping only dog-related tokens while discarding cat-related ones) and perform text-guided visual editing by passing unchanged tokens through the binder and using edited tokens directly.
• Known failure modes: The paper reports that BiCo treats each token equally, so concepts that deviate strongly from the "average looking" of a text token can drift (a colorful whimsical hat was not reproduced accurately), and composition requiring common-sense reasoning fails (a Doberman was given an additional fifth leg to hold a gun rather than raising an existing leg).

Methodology in Plain English

BiCo builds on Wan2.1-T2V-1.3B, a text-to-video diffusion model whose transformer blocks inject text through cross-attention, where latent tokens act as queries and prompt tokens act as keys and values.

The core idea is that each prompt token already has a learned association with a visual concept. BiCo attaches small "binder" modules — an MLP with residual connections and a zero-initialized learnable scaling factor, expressed as f(p) = p + gamma * MLP(p) — to the cross-attention layers. During one-shot training on a single image or video and its prompt, the binders update the prompt tokens so they carry the specific visual concept from that input. A global binder performs a coarse update, and per-block binders refine it inside each transformer block.

To keep the binding accurate from only one example, a vision-language model extracts key concepts from the input and rewrites them into many varied prompts, while a special absorbent token is appended to the prompt sequence and learns to soak up visual details that the varied prompts do not cover. At composition time, the user writes a target prompt, the concept-related words are routed through their corresponding trained binders, the absorbent tokens are dropped, and the resulting updated tokens condition the model.

For video inputs, training happens in two phases: first on individual frames with only spatial concepts, using the same setup as images, then on whole videos with temporal concepts, using a binder that adds a temporal MLP branch combined with the spatial branch through a gating module whose weights start at zero.

Training specifics: learning rate 1.0 × 10⁻⁴, 2400 iterations per stage, noise-level threshold alpha set to 0.875, 81 frames generated at inference, all on NVIDIA RTX 4090 GPUs. The absorbent token and both training strategies are validated by ablations and by visualizations of cross-attention maps, which show the absorbent token attending to irrelevant details such as plants.

Why This Matters

This work pushes concept personalization for video beyond "animate this subject with that motion" toward a general token-level composition interface, where attributes like style, lighting, and motion can be mixed across image and video sources using text alone.

• Filmmaking and previsualization: creators can sketch a shot by combining a character from one image with motion and style from another video, without training separate models per element.
• Content creation and advertising: brand styles, product appearances, and reference motions could be recombined into new videos from very few inputs.
• Editing and asset separation: the same binding mechanism supports isolating concepts (dogs versus cats in one clip) and performing text-guided edits while preserving untouched elements.
• Pipelines built on existing T2V models: because BiCo is a lightweight binder attached to a DiT-based model, the approach is relevant to anyone already building on Wan-style backbones rather than requiring a new generative model.

Industrial relevance comes from the one-shot nature of the method: the paper reports no per-concept dataset collection or mask annotation, which lowers the cost of personalizing a video generator for a specific subject or style, although the evaluation is limited to 40 test cases and human ratings from 28 and 24 volunteers.

Future Directions

• Adaptive token weighting: the authors note that token importance for T2V generation is unevenly distributed and propose integrating adaptive designs that highlight critical tokens, such as subjects and motions, over function words.
• Stronger representation for atypical concepts: the paper observes that when a concept departs strongly from the average appearance of its text token, a single binder may lack the capacity to hold all the visual information, motivating more expressive binders.
• Common-sense-aware composition: failure cases such as the five-legged dog suggest combining BiCo with the reasoning abilities of vision-language models for a more comprehensive captioning and composing paradigm.
• Scaling the evaluation: the current comparisons are restricted to one image plus one video for fairness, and the paper leaves open broader tests over more sources and concept counts.

Target Audience

Researchers and practitioners in generative video, diffusion model personalization, and multimodal conditioning will benefit most, particularly those already working with Diffusion Transformers and cross-attention control. The paper is also useful for creators and technical artists who want to understand what a token-level concept composition interface makes possible, and for engineers evaluating whether personalization for video can be done with lightweight, one-shot adaptation rather than full fine-tuning. Readers without a background in diffusion models or attention mechanisms will find the methodology section demanding.

Authors’ abstract

Visual concept composition, which aims to integrate different elements from images and videos into a single, coherent visual output, still falls short in accurately extracting complex concepts from visual inputs and flexibly combining concepts from both images and videos. We introduce Bind & Compose, a one-shot method that enables flexible visual concept composition by binding visual concepts with corresponding prompt tokens and composing the target prompt with bound tokens from various sources. It adopts a hierarchical binder structure for cross-attention conditioning in Diffusion Transformers to encode visual concepts into corresponding prompt tokens for accurate decomposition of complex visual concepts. To improve concept-token binding accuracy, we design a Diversify-and-Absorb Mechanism that uses an extra absorbent token to eliminate the impact of concept-irrelevant details when training with diversified prompts. To enhance the compatibility between image and video concepts, we present a Temporal Disentanglement Strategy that decouples the training process of video concepts into two stages with a dual-branch binder structure for temporal modeling. Evaluations demonstrate that our method achieves superior concept consistency, prompt fidelity, and motion quality over existing approaches, opening up new possibilities for visual creativity.