FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation

AI summary

Overview

Research area: Computer Vision — unified conditional image generation, video foundation model adaptation, parameter-efficient fine-tuning (LoRA) of diffusion transformers.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, 3D rotary positional embeddings (3D-RoPE), mixture-of-experts routing, and low-rank adaptation.

Scope: FrameForge adapts a frozen pretrained video diffusion model (HunyuanVideo1.5) with a single shared LoRA to perform subject-driven generation, canny-to-image generation, and instruction-guided image editing through input-adaptive "virtual temporal" positioning plus a latent reasoning chain.

What This Paper Is About

Unified conditional image generation asks one model to handle very different inputs — a reference subject, an edge map, or an editing instruction — while producing an image that follows text. Pretrained video models are attractive foundations because they already encode how visual content changes over time, but existing methods always place the condition and target at fixed temporal positions, giving every task the same positional relation. The authors argue that different tasks need different "distances" between condition and target, and propose FrameForge to choose that distance per input and denoising step, plus an intermediate latent that models how the condition turns into the target.

Key Contributions

1. A reframing of heterogeneous conditional generation as an input-adaptive virtual temporal relation. The authors measure LPIPS distances between frame pairs in natural videos at different temporal intervals and compare them with condition-to-target LPIPS distances for editing, subject-driven generation, and canny-to-image generation. They show the three tasks have distinct distributions whose ranges overlap with natural video transitions, motivating per-input rather than fixed temporal offsets.

2. Mixture of Temporal Experts (MoTE). Multiple temporal RoPE experts correspond to different virtual intervals; a lightweight router selects one expert per sample (Top-1 with straight-through gradients) from text, condition, and noisy target representations, so the virtual gap adapts across tasks, samples, and denoising steps. A batch-level load-balancing loss prevents routing collapse.

3. Chain of Temporal Reasoning (CoTR). A single noise-initialized reasoning latent forms a condition–reasoning–target chain, and a stochastic reasoning-bottleneck attention mask (applied with probability 0.5 during the intermediate training stage) blocks direct target-to-text/condition attention, forcing information to flow through the latent pathway. No intermediate-frame supervision is used.

4. A single shared adapter for three tasks. The video backbone and VAE encoder stay frozen; only a LoRA (rank and alpha 32, 182M trainable parameters) and the router are trained jointly across subject-driven generation, canny-to-image, and editing, with a three-stage schedule.

Main Findings

• Best overall DreamBench score among compared methods: FrameForge reaches O = 0.739, improving over ChronoEdit by 1.37% and outperforming OminiControl (0.719), ACE++ (0.688), and EasyControl (0.679). Its SC of 0.739 ranks second to ChronoEdit's 0.757; its PQ is 0.783.
• Strongest structural fidelity on COCO2017 canny-to-image: SSIM of 0.447, a 7.2% relative improvement over the second-best EasyControl (0.417). CLIP-T is 0.323, comparable to ControlNet and RealGeneral; FID is 23.47, comparable to DRA-Ctrl (23.27) and RealGeneral (24.63). ControlNet has the lowest FID (20.42), which the authors attribute partly to backbone-specific appearance statistics and note that FID is known to disagree with human judgments.
• Large gains on instruction-guided editing (GEdit-Bench): Overall G-O rises from ChronoEdit's 5.08 to 6.12, and Motion Change G-O from 5.76 to 6.84, both the best among open-source methods. The Motion Change score surpasses Gemini 2.5 (6.72) and Seedream 4.0 (5.53) and approaches GPT-4o (7.81), though overall GEdit remains below closed-source systems (GPT-4o 8.01).
• Routing is task- and timestep-dependent: Over 50 denoising steps averaged across 500 cases, canny-to-image (largest condition-target distance) predominantly selects the long-interval expert, especially early on; subject-driven generation and editing start with the medium-interval expert and shift toward the short-interval expert as target content emerges, with editing transitioning earlier and more strongly.
• MoTE and CoTR are complementary: Shared LoRA alone gives 0.722 / 25.78 / 4.94 (subject / canny FID / editing). Adding MoTE gives 0.719 / 25.62 / 5.77; adding CoTR gives 0.726 / 25.94 / 5.56; both together give 0.739 / 23.47 / 6.12. Neither module alone recovers the task-specific editing score of 5.99, but the combination exceeds it.
• Three experts is the best trade-off: 1 expert gives 0.739 / 25.90 / 5.90; 3 gives 0.739 / 23.47 / 6.12; 5 gives 0.743 / 24.24 / 5.96; 7 gives 0.743 / 24.55 / 5.67 — more experts slightly help subject-driven generation but degrade the other tasks, which the authors attribute to sparser supervision per expert.
• One reasoning latent is optimal: 0 latents gives 0.719 / 25.62 / 5.77; 1 gives 0.739 / 23.47 / 6.12; 2 gives 0.709 / 25.82 / 5.62; 3 gives 0.713 / 24.55 / 5.09.
• Interval configuration matters: Short spans {17, 25, 33} give 0.741 / 24.84 / 5.71; medium {13, 37, 61} give 0.716 / 23.19 / 5.42; long {17, 69, 121} give 0.739 / 23.47 / 6.12 — the long/wide configuration is chosen because its subject score is only 0.002 below the best while delivering the best editing result.
• Transfers to another video backbone: On HunyuanVideo1.0, FrameForge improves over a unified shared-LoRA baseline on all three tasks — subject 0.699 to 0.736, canny FID 24.52 to 23.07, editing 4.38 to 5.00.

Methodology in Plain English

The starting point is a video generator that denoises compressed video latents and uses 3D rotary position embeddings to tell tokens where they sit in time and space. FrameForge treats a conditional image task as generating the last frame of a tiny two-frame (or three-slot) video: the condition is the first frame, the target is the last, and the gap between them sets how strongly the model is told the content should change.

• Choosing the gap (MoTE): Instead of hard-coding that gap, the model keeps several rotary-position "experts," each built for a different virtual span — in the main setting latent spans {5, 18, 31}, equivalent to video-frame spans {17, 69, 121} with a temporal compression factor of 4. A small router looks at the text, the condition, and the noisy target, averages its per-token logits into one sample-level score, and picks one expert. Because the pick is discrete, gradients are passed through a softmax relaxation with a straight-through estimator; a load-balancing penalty (weight 0.01) keeps all experts in use.
• Modeling the change itself (CoTR): A single extra latent, initialized from Gaussian noise at an intermediate temporal position, sits between condition and target. It is never supervised or decoded; it is refined by attention. During the middle training stage, a mask applied with probability 0.5 cuts direct attention between target tokens and text/condition tokens so that target information must route through this latent — making it a genuine bottleneck rather than an optional detour.
• Training: The video backbone and VAE are frozen; one LoRA adapter (rank and alpha 32) is inserted into the transformer's attention and feed-forward projections, giving 182M trainable parameters. Stage 1 trains only the router without the reasoning latent; stage 2 adds the latent, trains router and LoRA jointly, and applies the stochastic mask; stage 3 keeps both with full attention, matching inference. Training runs 12,000 iterations on two NVIDIA A800 GPUs with batch size 60 and learning rate 3×10⁻⁴ for about 6 days, on 170K samples per task (510K total) at 512×512.
• Evaluation setup: DreamBench (30 subjects, 25 prompts each, VIEScore with GPT-4o), the 5,000-image COCO2017 validation set with GPT-4o captions, and the 606-sample English subset of GEdit-Bench across 11 categories scored with GPT-4.1. Inference uses 50 denoising steps and classifier-free guidance of 6 for subject-driven generation and editing, 2 for canny-to-image.

Why This Matters

Impact on research. The paper offers a concrete diagnosis of why a single shared adapter struggles across conditional tasks — fixed temporal positioning ignores how much change each task requires — and a measurable fix. It also provides evidence, via the LPIPS analysis and the routing curves, that a video model's temporal prior is a continuum that can be indexed per input and per denoising step, rather than a binary "condition frame vs. target frame" convention. The ablation showing that neither MoTE nor CoTR alone beats task-specific editing (5.99) while the pair reaches 6.12 is a useful data point on task interference in unified models.

Real-world applications:

• Product and e-commerce imagery: placing a specific item into new scenes while preserving its identity, which is exactly the subject-driven setting where FrameForge balances SC and PQ.
• Design and content revision: instruction-based editing with motion and spatial changes (e.g., re-posing an animal, relocating objects) where existing open-source editors often leave edits incomplete or drift.
• Contour-constrained asset generation: turning sketches or edge maps into rendered images for concept art, storyboarding, and CAD-adjacent visualization, where the 0.447 SSIM indicates tighter adherence to input geometry.
• Consumer photo tools: single-model pipelines that avoid maintaining and swapping separate task-specific adapters.

Industry relevance. The method trains only 182M parameters on top of a frozen backbone in about 6 days on two A800 GPUs, which is a practical fine-tuning budget. A single shared adapter replacing per-task adapters simplifies deployment and model maintenance, and the demonstrated transfer to HunyuanVideo1.0 suggests the approach is not tied to one specific backbone.

Future Directions

• Extending beyond three experts and fixed candidate spans. The ablation shows 5 and 7 experts help subject-driven generation but hurt canny and editing, which the authors link to sparse supervision; learning the span set rather than choosing from {17, 25, 33}, {13, 37, 61}, or {17, 69, 121} is an open question.
• Supervising or decoding the reasoning latent. CoTR uses exactly one unsupervised latent because 2 and 3 latents degrade results, suggesting the pathway is under-constrained; adding weak intermediate supervision or a decoded intermediate frame could allow richer transformation modeling.
• Closing the gap to closed-source editors. Overall GEdit G-O of 6.12 remains well below GPT-4o's 8.01 and Gemini 2.5's 7.89, even though Motion Change is competitive — improving general editing quality under the shared adapter is unresolved.
• Reconciling distributional and structural metrics. FrameForge has the best SSIM but a higher FID than ControlNet and DRA-Ctrl; the authors argue FID reflects backbone appearance statistics and can disagree with human judgment, but no human study is reported, leaving the perceptual question open.

Target Audience

Researchers and engineers working on unified or multi-task generative models, diffusion transformer adaptation, and parameter-efficient fine-tuning; practitioners adapting video foundation models to image tasks; and teams building production image editing or subject-driven generation systems who need a single shared adapter rather than per-task models. Readers without a background in diffusion transformers, RoPE, and mixture-of-experts routing will find the method sections demanding, though the motivation analysis in Figure 1 is accessible on its own.

Authors’ abstract

Unified conditional image generation requires a single model to preserve different aspects of a visual input while producing a target image that follows textual guidance. Pretrained video models provide a promising foundation by encoding visual transformations across time, yet existing approaches typically assign the condition and target to fixed temporal positions. We observe that condition-target relationships vary substantially across generation tasks, while video models naturally encode visual transformations at different temporal scales. This motivates us to model conditional generation through input-adaptive virtual temporal relations. We propose FrameForge, which captures two complementary aspects of such relations. Mixture of Temporal Experts (MoTE) models how far the target lies from the condition by dynamically selecting temporal RoPE experts with different virtual intervals. Chain of Temporal Reasoning (CoTR) models how the condition is transformed into the target through a latent intermediate state, with a reasoning-bottleneck attention mask encouraging information propagation through this pathway. Experiments on DreamBench, COCO2017, and GEdit-Bench show that FrameForge achieves superior performance in subject-driven generation, canny-to-image generation, and image editing among the compared methods.