Painting Change Map as guidance for generating painting processes

AI summary

Overview

Research area: Computer Vision — controllable video diffusion models for synthesizing the painting process (and the reverse "un-painting" process) from a finished artwork.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with diffusion/video-generation backbones, LoRA fine-tuning, and latent control signals.

Scope: The paper introduces a spatio-temporal control representation called the Painting Change Map (PCM) and shows, through benchmarks against generic spatial controls and specialized painting architectures, that fusing temporal stroke progression with static lineart gives the most controllable and accurate painting-process generation.

What This Paper Is About

Existing models can produce convincing time-lapse videos of a painting being created, but they treat the creative process as a black box: conditioners are either rigid static geometry (Canny edges, depth maps) or a single global progress scalar that says nothing about where on the canvas the artist is working. The authors argue that real control requires two things at once — where an artist acts and when those actions occur. Their goal is a fine-grained, stroke-level control map that tells a video generator exactly which parts of the canvas change at each moment, and to show that this map can itself be predicted from a finished painting and then used to reconstruct its creation.

Key Contributions

1. The Painting Change Map (PCM): a spatio-temporal representation that jointly encodes the spatial location and chronological order of changes in a painting, computed over a 1,000-timestep continuum from frame-to-frame structural differences.
2. Four RGB encoding strategies for turning the PCM into a control video a standard video diffusion model can consume: Interpolation, 24-bit Packing, History, and the hybrid Interpolation & Lineart, which performs best empirically.
3. A decoupled two-stage pipeline: a first model predicts the change map from the finished painting, and a second model uses that predicted map plus the painting to synthesize the actual color painting process.
4. A comprehensive benchmark against generic spatial controls (Canny Edges, Depth-Anything-v2, Lineart) and specialized painting architectures (Inverse Painting, ProcessPainter, PaintsUndo, Loomis Painter), covering global unpainting, localized instance-level control, and efficiency metrics.

Main Findings

• Pure change maps lose to plain edge baselines. In global unpainting with ground-truth control maps, the standalone temporal encodings (Interpolation, 24-Bit Packing, History) underperform Canny and Lineart; Interpolation, for example, reaches an MSE of 814.24 versus 592.81 for Lineart. The authors attribute this to canvas/camera drift and to the lack of fixed geometric anchors when the model is conditioned on the final painting.
• Fusing temporal change with static lineart is the winning design. Interpolation & Lineart leads every global metric: CLIP 0.926, DINOv2 0.920, FID 55.75, LPIPS 0.207, and MSE 449.68 — the paper highlights the MSE reduction from 592.81 (Lineart) to 449.68.
• Localized instance control works zero-shot. Although trained only on the global process, the method can unpaint a single region (e.g., a mountain) while holding the background fixed. Across seven instance categories it achieves the best FID (125.91), MSE (149.61) and DINOv2 (0.868), and matches Lineart on LPIPS (0.327); Canny has the highest CLIP in this setting at 0.966 versus 0.964.
• Thresholds matter, but the method is robust in a wide operating window. Low sensitivity (τ = 0.05) admits noise and artifacts, degrading LPIPS to 0.214 and MSE to 494.96; over-filtering (τ = 0.18) strips genuine strokes and slightly lowers CLIP/DINOv2. τ = 0.13 is reported as the consistently optimal setting, paired with a global motion filter θ in the 4.0–5.0% range (best grid MSE of 448.87 appears at τ = 0.18, θ = 5.0, at a CLIP of 0.924).
• Competitive with state of the art without oracle maps. With self-generated change maps, the pipeline nearly ties Loomis Painter on CLIP (0.87 vs 0.88) and LPIPS (0.45 vs 0.44) while beating it on MSE (3127.13 vs 3152.96), using 5B + 1.3B parameters against Loomis Painter's 14B.
• Large headroom under oracle conditions. Given ground-truth change maps, the method outperforms all baselines on every metric — CLIP 0.93, DINOv2 0.92, FID 55.75, LPIPS 0.21, MSE 449.68 — using only the 1.3B video model.
• Efficiency gains. The decoupled design reaches 0.62 FPS versus Loomis Painter's 0.19 (more than tripling throughput), with latency of 130.74 s versus 428.3 s, and peak memory of 25.84 GB in stage one and 17.04 GB in stage two on a single NVIDIA A100 SXM4 64GB.
• Randomized temporal sampling helps. Loomis Painter trains on the same 81 uniformly spaced frames, whereas this method samples 81-frame sequences randomly from the 1,000 available frames, which the authors credit with improved temporal robustness.

Methodology in Plain English

The pipeline starts from an existing painting video dataset. For each pair of consecutive frames, the authors subtract one frame from the next and collapse the color channels into a single per-pixel magnitude using the L2 norm; this yields a map of how strongly each pixel changed. A sensitivity threshold τ turns that magnitude into a binary "something was painted here" mask, filtering out sensor noise and slight lighting shifts. A second safeguard, the global motion filter with threshold θ, throws out frames in which more than θ percent of pixels changed, since those are almost certainly camera or canvas tremors rather than real brush strokes; θ = 5% is used in practice.

The binary masks are then accumulated over time: each pixel stores the normalized timestamp of the most recent moment it was actually modified, producing a dense tensor over 1,000 timesteps. Since video diffusion models expect an RGB control video, this tensor is projected into three channels. The chosen projection, Interpolation & Lineart, puts a black-to-white grayscale ramp keyed to the timestamp in the red channel (brighter means more recently painted) and places a Lineart edge extraction of the target frame in the green and blue channels, so the model gets both "what changed when" and a stable structural anchor.

Training uses LoRA fine-tuning of WAN 2.1 Fun (a video-to-video model built on WAN 2.1), conditioned on the control video and the finished artwork, and generates in reverse-chronological "un-painting" order in 81-frame sequences. The dataset is an acrylic subset with 132 training, 16 test, and 13 validation videos; frame count per video was raised to 1,000 by linear sampling, occlusions such as hands and tools were removed using SAM 3 instead of the original BiRefNet preprocessing, and each configuration was trained for 19 epochs (about 8 hours) on one A100 64GB at a learning rate of 1e-4. For real artworks with no source video, a two-stage inference is used: a fine-tuned Wan2.1 Image-to-Video 5B model hallucinates the change map, and the 1.3B video-to-video model turns that map into the painting sequence.

Why This Matters

The work reframes painting-process generation from a single global progress dial into a spatially explicit, stroke-level control problem, and shows that the control signal itself is learnable from a finished image. It also documents a non-obvious practical lesson: on real captured footage, temporal change alone is not enough, and static structural guidance remains essential.

Real-world applications:

• Art education and tutorial generation, where a learner can request "show me only how the sky was painted" and get a targeted demonstration.
• Artist and studio tooling that turns a finished digital or scanned painting into a shareable process video for portfolios and social media.
• Restoration and conservation documentation, where regions of an artwork can be selectively un-painted to reason about layered composition.
• Archival visualization of artworks where no process footage exists, using the predicted change map to synthesize a plausible creation sequence.

Industry relevance: the decoupled, two-stage design uses 5B + 1.3B parameters instead of a 14B monolith, runs with at most 25.84 GB of VRAM on a single data-center GPU, and more than triples throughput relative to the prior large model — a profile that matters for deployment in creative software. The general lesson that temporal control maps benefit from being fused with static structural cues also transfers to other controllable video generation tasks.

Future Directions

• Canvas stabilization. Despite global motion filtering, physical canvas movement lets older strokes drift out of alignment with current frame geometry; the authors propose adding automated canvas alignment or keypoint rectification to the data pipeline.
• What the change map alone encodes. The authors want to test whether the medium, artistic style, or even the identity of the painter can be inferred from the change map, a direction currently limited by the scarcity of sufficiently diverse and large-scale datasets.
• Closing the oracle gap. The large difference between oracle and predicted change maps (MSE 449.68 versus 3127.13; CLIP 0.93 versus 0.87) points to better change-map prediction models as the clearest route to improvement.
• Broader evaluation of localized control. Instance-level unpainting was only evaluated zero-shot on seven categories, leaving room to train for and benchmark finer, more varied localized editing scenarios.

Target Audience

Researchers and practitioners in generative video and diffusion-model control who are interested in spatial and temporal conditioning mechanisms; digital artists and creative-tool developers who want controllable painting-process synthesis; and readers studying data-driven approaches to artistic process modeling, who will find the threshold ablation and the dataset-preprocessing choices (SAM 3 occlusion removal, 1,000-frame resampling) directly reusable.

Authors’ abstract

Digital art generation has undergone a massive paradigm shift, moving from static image synthesis to dynamic video generation that can mimic the human painting process. While existing models can synthesize visually stunning time-lapses of paintings being created or "unpainted," they largely treat the creative process as a black box. Current conditioning mechanisms are either too rigid, relying on static geometric cues like Canny edges or depth maps, or too holistic, utilizing simple scalar progress variables that offer no localized control. Consequently, users lack the agency to steer how a painting evolves over time. In this work, we argue that true control over the video generation of artwork requires a joint understanding of two fundamental dimensions: where an artist acts, and when those actions take place. To achieve this, we introduce the Painting Change Map, a novel spatio-temporal control mechanism that explicitly tracks normalized structural changes across a 1,000-timestep continuum, a resolution sufficient to capture fine-grained stroke-level changes while remaining computationally tractable. By mapping localized visual differences into an encoded RGB space, our representation enables fine-grained guidance for both the forward painting and reverse unpainting processes. Extensive benchmarking demonstrates the efficacy of our approach. We outperform existing control baselines and achieve competitive or superior results compared to state-of-the-art painting methods. Furthermore, under oracle conditions, our method surpasses all baselines by a significant margin. Code is available at https://github.com/Markus-Pobitzer/painting-change-map.