Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing

AI summary

Overview

Research area: Computer Vision / Generative AI — specifically instruction-based image editing with diffusion transformers.

Technical level: Advanced. The paper assumes familiarity with diffusion models, DiT architectures, flow matching, LoRA adaptation, modulation/AdaLN conditioning, and diffusion-based image morphing.

Scope: The paper introduces Kontinuous Kontext, a method that adds a continuous "edit strength" scalar to a pretrained instruction-driven image editing model (Flux Kontext), trained on a synthetically generated and filtered dataset of image–instruction–strength quadruplets.

What This Paper Is About

Text instructions tell an editing model what to change but not how much, so users cannot dial an edit from "barely visible" to "fully applied." Prior continuous-control methods typically require separate training for each attribute (e.g., one module for age, another for metallicness), which does not scale. The goal of this work is a single unified model that accepts a scalar strength value alongside any edit instruction and produces smooth, monotonic edits across stylization, attributes, materials, backgrounds, and shapes.

Key Contributions

1. A continuous strength control dimension for instruction-based editing. Kontinuous Kontext augments Flux Kontext with an additional scalar input, letting users traverse smoothly from no change to a fully realized edit rather than choosing only a binary before/after.

2. A lightweight strength projector operating in the model's modulation space. An MLP maps the scalar strength (plus the pooled CLIP text embedding) to offsets on the text-token modulation parameters, rather than injecting strength as an extra text token — an approach the authors found produced abrupt transitions.

3. A synthetic data generation and filtering pipeline. Source images, LVLM-generated instructions, Flux Kontext full-strength edits, and Freemorph intermediate morphs are combined into strength-annotated trajectories, then filtered for inversion quality, trajectory smoothness, and edit strength.

4. A single model that generalizes without attribute-specific training. The method handles material, appearance, attribute, shape, background, environment, and stylization edits, and generalizes to unseen categories such as facial attributes and body shape changes.

Main Findings

• Best smoothness and instruction following against interpolation baselines (PIEBench, 540 images). Reported as δ_smooth (lower is better) / CLIP-Dir. (higher is better): Ours 0.329 / 0.241; Diffmorpher 0.371 / 0.181; Freemorph 0.365 / 0.189; WAN-Video 0.853 / 0.269.

• Better smoothness than domain-specific attribute methods. Against ConceptSliders: Ours 0.098 / 0.382 versus ConceptSliders 0.143 / 0.186 (comparison across 11 available sliders covering facial attributes, stylization, and scene edits). Against MARBLE on metallicness and glow material editing from PIEBench: Ours 0.350 / 0.101 versus MARBLE 2.577 / 0.157 — MARBLE achieved higher CLIP-Dir. but jumped abruptly to the final edit, producing a much worse smoothness score.

• Modulation-space conditioning beats text-space conditioning. Ablation results (δ_smooth / CLIP-Dir.): text-space condn 1.468 / 0.191; w/o text projector 1.092 / 0.141; w/o filtering 0.483 / 0.228; Ours 0.329 / 0.241. Adding the slider conditioning as an extra text token was the worst configuration.

• Text embeddings in the projector are necessary for calibration. Removing the pooled CLIP text embedding input produced weaker, non-smooth edits, because the projector would otherwise predict identical offsets for all edits at a given strength, causing sudden jumps (e.g., in material editing).

• Data filtering matters substantially. Filtering poor-quality and non-smooth sequences improved both smoothness and text alignment; the dataset was reduced from 110K to 64K trajectories (the appendix states ~66K edit trajectories after filtering).

• Perturbing modulation parameters already varies edit strength. A simple experiment scaling text-token modulation parameters with a scalar v ∈ (0.5, 2.0) (reported as v ∈ (0.5, 1.3) in the appendix) produced edits of varying strength while preserving image identity, motivating the projection approach.

• More linear edit trajectories than baselines. Measuring change in the source image with DINO features as strength increases, the method changes the source gradually and monotonically, achieving the highest linearity as measured by absolute Pearson correlation |r| against a linear path.

• User study wins across all criteria. Against each baseline, 20 input images were sampled for a total of 100 images evaluated, with 20 participants judging smoothness, realism, editing capability, and overall sequence quality; the method consistently outperformed all baselines (win rates reported in the paper's Figure 11).

• Stated limitations. Because it is built on Flux Kontext, the model inherits weaknesses in precise geometric manipulations such as accurate object rotation or translation, and it fails at generating consistent extrapolating edits for large transformations.

• Training configuration. A rank-4 LoRA on all attention layers plus the slider projector, trained at 512×512 resolution, on 8 NVIDIA A100 (80GB) GPUs for 110,000 iterations with effective batch size 8 and a constant learning rate of 2×10⁻⁵, taking about 72 hours. Slider conditioning is dropped with probability 0.1 (10%). Inference uses the default Euler scheduler from Flux Kontext with T = 28 steps, at generation time similar to Flux Kontext.

Methodology in Plain English

The authors start from Flux Kontext, an existing model that edits images from text instructions, and give it one extra input: a number between 0 and 1 representing how strongly to apply the edit.

The design choice that makes this work is where that number is injected. Adding it as an extra word in the text prompt failed, producing abrupt jumps between neighboring strengths. Instead, the authors exploit the fact that diffusion transformers internally compute modulation parameters (shift and scale values) that steer how strongly text tokens influence generation. A small MLP — the strength projector — takes the scalar strength and the pooled CLIP text embedding of the instruction and outputs offsets to those modulation parameters. Because the text embedding is included, the offsets are calibrated to the specific edit: the same strength value means different things for "make it winter" versus "make the car bigger."

Training data is the hard part, since real image pairs with graded strength labels do not exist. The pipeline has three steps. First, 110K images are drawn from Subject200K and an LVLM (Qwen) writes a diverse edit instruction for each, using category-specific system prompts and randomly sampled in-context examples (50–100 per category, 4 sampled per prompt); Flux Kontext then produces the full-strength edit. Second, Freemorph — a training-free diffusion morphing method built on Stable Diffusion 2.1 with a DDIM scheduler and T = 50 steps at 768×768 native resolution — generates intermediate images between source and edit by inverting both and interpolating attention features; the main text uses N = 6 for the discrete set of N+1 strengths uniformly sampled in [0,1], while the appendix states N = 5 intermediate morphs. Third, because morphing produces artifacts, incomplete objects, and inversion errors, samples are filtered using LPIPS distances: a KL-divergence threshold of 0.15 on the uniformity of the per-step deltas, a check on inversion quality, and a check that the edit actually changed the image. Endpoints are also replaced with their inversions to keep the sequence consistent. A separate 10K synthetic set of zoom-based object-size edits is also generated.

At training time, the model learns with a standard flow-matching loss over the quadruplet (source image, instruction, strength, target edit).

Why This Matters

Impact on research: The paper argues that edit intensity is naturally encoded in the modulation space of instruction-driven diffusion models, and that a lightweight learned projector into that space is enough to unlock continuous control. This reframes continuous editing from a per-attribute engineering problem (train a LoRA or adapter per concept) to a single shared control channel. It also provides a reusable data-synthesis-and-filtering recipe for training on properties that lack labeled real data.

Real-world applications:

• Photo editing and creative tools where users drag a slider to control degree of stylization, aging, season change, or material swap instead of rewriting prompts.
• E-commerce and product photography, where the strength of appearance, material, or background changes must be tuned while preserving product identity.
• Film, advertising, and game previsualization, where gradual transformation sequences (shape morphing, environmental change, lighting) are needed for reference and iteration.
• Social media and consumer AR filters that offer smooth, adjustable creative effects rather than all-or-nothing transformations.

Industry relevance: The method is built on an existing state-of-the-art editing backbone and adds only a small MLP plus LoRA weights, so generation time stays similar to the base model — an important practical property for deployment. The demonstrated generalization to unseen attributes (facial attributes, body shape) means a single shipped model can support many editable concepts without per-attribute retraining.

Future Directions

• Spatial and temporal intensity fields. The conclusion explicitly suggests that other forms of continuous control, such as spatial or temporal intensity fields, could be introduced in a similarly lightweight manner, enabling interactive tools that combine language with continuous sliders.
• Consistent extrapolating edits. Generating edits beyond full strength (for large transformations) is called out as a failure case and remains open.
• Overcoming inherited backbone weaknesses. Precise geometric operations such as accurate object rotation and translation fail because the base Flux Kontext model struggles with them; addressing this would broaden the method's applicability.
• Reducing reliance on imperfect synthetic supervision. The data pipeline depends on Freemorph, which produces artifacts, abrupt transitions, and inversion errors that must be filtered out; better intermediate-edit generation could reduce filtering loss and improve coverage.

Target Audience

Researchers and engineers working on controllable generative models, diffusion-based image editing, and diffusion transformer conditioning mechanisms. It is also relevant to practitioners building creative or commercial image-editing products who need fine-grained user control, and to readers interested in data synthesis pipelines for training generative models on properties without existing labeled data.

Authors’ abstract

Instruction-based image editing offers a powerful and intuitive way to manipulate images through natural language. Yet, relying solely on text instructions limits fine-grained control over the extent of edits. We introduce Kontinuous Kontext, an instruction-driven editing model that provides a new dimension of control over edit strength, enabling users to adjust edits gradually from no change to a fully realized result in a smooth and continuous manner. Kontinuous Kontext extends a state-of-the-art image editing model to accept an additional input, a scalar edit strength which is then paired with the edit instruction, enabling explicit control over the extent of the edit. To inject this scalar information, we train a lightweight projector network that maps the input scalar and the edit instruction to coefficients in the model's modulation space. For training our model, we synthesize a diverse dataset of image-edit-instruction-strength quadruplets using existing generative models, followed by a filtering stage to ensure quality and consistency. Kontinuous Kontext provides a unified approach for fine-grained control over edit strength for instruction driven editing from subtle to strong across diverse operations such as stylization, attribute, material, background, and shape changes, without requiring attribute-specific training.