LociArt: A Unified Paradigm for Free-Style Local Video Stylization

AI summary

Overview

Research area: Computer vision — video style transfer and generative video editing, specifically region-controlled (local) stylization of video.

Technical level: Advanced. The method builds on Generative Refinement Networks (GRN), binary-latent video generation, VAE latents, and diffusion-transformer-style blocks; the evaluation uses prototype-based style metrics and VLM judges.

Scope: The paper proposes LociArt, a combined dataset (LociArt-100K), model (GRNStyle), and benchmark (LociArt Benchmark) for stylizing only chosen regions of a video in a chosen artistic style while leaving everything else intact and temporally stable.

What This Paper Is About

Most video stylization tools restyle an entire scene, but artists often want to change just one object — or just the surroundings — while keeping the rest of the footage untouched. The hard part is that a mask tells a model where to edit but not what style to apply there, and mask information tends to lose either spatial precision or temporal context when pushed through a video model. The paper's goal is a single framework that learns this region-plus-style control from paired data, and an evaluation protocol that actually measures whether the style landed inside the region and stayed out of everything else.

Key Contributions

1. A unified formulation of free-style local video stylization. Local editing is posed as region-conditioned artistic transformation that must simultaneously specify where to stylize, what style to apply, preserve non-target content, and maintain temporal consistency — with an accompanying data pipeline, model, and benchmark rather than any one piece alone.

2. GRNStyle with a polar style-aware mask. Built on GRN and GRNEdit's source-evidence formulation, the model encodes each style as a fixed, equal-norm direction (a polar angle at unit radius) written into the mask's first two VAE input channels, with binary occupancy in the third. Because all style vectors have norm 1, styles differ by direction rather than by an arbitrary scalar ordering, and a linear readout recovers the region–style association directly.

3. Representation–geometry mask conditioning. Encoded mask features are anchored against the encoding of an empty mask (subtracting Φ(0)) and combined with raw-mask geometric support obtained by average pooling, size-preserving max pooling, and a 0.5 threshold. The two branches are fused through block-wise gates shared across injection blocks, coupling spatiotemporal context with spatial fidelity without adding tokens.

4. LociArt-100K and the LociArt Benchmark. An automated pipeline (object identification and removal, global stylization and filtering, matting and compositing, human verification) produces 100K paired local stylization examples across 18 style categories; the benchmark contains 80 source videos, 84 object–mask units, and 168 foreground/background stylization cases, scored with Style Prototype Similarity (SPS) plus non-target preservation and temporal consistency measures.

Main Findings

• Best regional style quality on the benchmark. GRNStyle-2B reaches SPS of 0.407 for foreground and 0.705 for background stylization, the highest in Table 1, and the best background CLIPT (0.199). The next-best SPS-fg is the fine-tuned VACE-1.3B at 0.386, and the commercial Seedance 2.5 at 0.347.

• Preservation and temporal stability are maintained. GRNStyle records PSNR-fg 25.92 dB and PSNR-bg 26.15 dB, and temporal consistency (TC) of 0.983. The paper notes that VACE's higher PSNR (32.35 fg / 29.17 bg) coincides with minimal stylization, so it does not indicate a better style–preservation trade-off.

• VLM-judged balance. Gemini 3.6 Flash scores give GRNStyle the highest Style score (4.62) alongside a high Preserve score (4.64); VACE-1.3B scores 0.05 on Style despite 5.00 on Preserve.

• Task-specific supervision matters. Fine-tuning VACE-1.3B on LociArt-100K with the same hyperparameters lifts it from SPS-fg 0.156 to 0.386 and SPS-bg 0.191 to 0.676, showing the dataset itself carries much of the gain.

• Human study agrees. With 20 participants rating a 160-case subset (16 cases per style), GRNStyle obtains the highest mean ratings for style fidelity (4.34) and background preservation (4.47) among the two video-based comparisons (CoinVE-Edit and Seedance 2.5). It beats CoinVE-Edit on both criteria for all ten styles, improves preservation over Seedance 2.5 for every style, and improves style fidelity for seven styles.

• Both conditioning components contribute. Ablation: Base 0.541 SPS / 0.180 CLIPT / 20.09 PSNR; Base+Polar 0.553 / 0.181 / 21.34; Base+RG 0.544 / 0.179 / 25.07; full GRNStyle 0.554 / 0.181 / 25.78. RG mainly improves preservation, polar mainly improves style, and together they give the best trade-off.

• Polar codes resist forgetting when styles are added. Extending a 10-style model to 15 styles using only the five new classes and no replay: old-class SPS drops 0.022 with polar codes versus 0.102 with binary masks; CLIPT drops 0.008 versus 0.019; PSNR stays stable (25.77 vs 25.75 for polar). New-class scores are slightly higher (polar: 0.590 SPS, 0.199 CLIPT, 25.74 PSNR).

• Multi-region composition emerges from single-region training. Qualitative examples show several regions stylized in one pass with the same or different learned styles, even though training used only single-region, single-style pairs; the composite field is formed by summing polar encodings of disjoint regions.

• The data pipeline's steps each matter. Qualitative comparisons show direct mask compositing leaves ghost curls or head outlines when stylized subjects extend beyond source silhouettes, and source-aligned masks clip stylized curls, hair tips, and noses; object removal alone cannot fix the mismatch, while matting follows the actual stylized contour.

Methodology in Plain English

The work has three coordinated parts.

Building supervision data. Because no dataset contains paired "local edit" examples, the authors manufacture them. They start from real videos (SA-V and Ditto-1M), use Qwen3-VL-32B to name the salient objects, and SAM3 to track them; masks are filtered to be nonempty in more than 40% of frames and to cover no more than 85% of any frame. Each source video is duplicated with the tracked object removed (via SVOR), giving clean background pixels where a stylized object might grow beyond its original outline. Both the original and object-removed videos are then stylized globally by Ditto or a fine-tuned VACE, and GPT-5.5 acts as a quality gate, scoring style adherence, content preservation, structural quality, style consistency, and temporal quality on 1–5 scales at five sampled frames (0, 20, 40, 60, 80 of 81), requiring 4, 4, 3, 4, 4. VideoMaMa estimates separate alpha mattes for the original and stylized objects so compositing follows each version's true contour. This yields two complementary targets per example — foreground stylized and background stylized — and humans check the composites before inclusion.

The model. GRNStyle works in GRN's binary-latent space: an encoder and binary quantizer turn the source video into bit codes, and the model iteratively refines partially randomized target bits, with the source bits serving as evidence of what to keep. Its distinctive move is to encode style as a direction rather than a number: each style gets a fixed angle on a unit circle, and that (cos θ, sin θ) pair is written into the mask channels only where the mask is on. Because every style vector has length 1, the model cannot accidentally learn "style 7 is bigger than style 3," and the region–style pairing exists in the input before any video encoding happens. To keep the mask sharp, the pipeline also encodes the mask through the same frozen encoder, subtracts the encoding of an all-zero mask (so "no edit" truly means zero signal), and multiplies in a pooled, thresholded version of the raw binary mask that says exactly where the latent positions are. Both signals are projected to a shared space and modulated per transformer block by a gate that combines visual evidence and the text instruction.

Evaluation. SPS compares the edited region against a prototype bank of style references in CSD feature space, so a style that never actually appears in the region scores low even if the frame looks plausible globally. It is paired with regional CLIP-T for text alignment, PSNR outside the mask for preservation, CLIP-embedding cosine similarity between adjacent frames for temporal consistency, and VLM Style/Preserve scores.

Why This Matters

Research impact. The paper reframes video stylization from a global transformation into a region-conditioned one, and argues that progress has been blocked not by model capacity alone but by missing paired supervision and by evaluation metrics that hide regional failures behind whole-video averages. Releasing a 100K-pair dataset, a defined benchmark with a style-specific metric, and an extensible style-codebook mechanism gives the subfield a shared substrate; the replay-free style-expansion experiment also raises a concrete question about how style inventories should be represented.

Real-world applications:

• Film and episodic post-production, where a single performer or prop is restyled inside an otherwise untouched shot.
• Advertising and branded content, where a product or background is re-skinned in an existing commercial without reshooting.
• Digital art and animation previsualization, where different objects in one plate receive different treatments (for example, a subject as sketch while the surroundings become watercolor).
• Social media and creator tools, where users select an object with a few clicks and apply a preset look while the rest of the clip remains photographic.

Industry relevance. The work comes from MiLM Plus at Xiaomi Inc. with academic collaborators, and it is benchmarked against a commercial system (Seedance 2.5), so the comparisons are aimed at deployment conditions rather than only academic baselines. Practical constraints are visible throughout: the pipeline enforces 81 frames at 720p, the paper includes an inference-efficiency appendix on latency and GPU memory, and it reports robustness to five mask perturbations at two severity levels and competitiveness with global stylization methods under all-one masks. The acknowledgements restrict VideoMaMa (CC BY-NC 4.0) and Ditto (CC BY-NC-SA 4.0) to academic, non-commercial use, which matters for anyone planning to build on the released data.

Future Directions

• Scale and diversify the style inventory. The released dataset covers 18 style categories and the model is trained with all 18; the replay-free experiment runs only from 10 to 15 styles. How far the polar codebook scales, and whether style relationships (related vs. distinct looks) can be exploited, is left open.

• Move beyond single-region supervision. Multi-region, multi-style results are demonstrated qualitatively rather than through a dedicated quantitative protocol, so a benchmark for simultaneous region–style combinations is an obvious next step.

• Formalize the limitations section. The paper defers limitations and future work to an appendix that is not included in the provided text, so specific failure modes and proposed remedies are not reported here.

• Tighten the evaluation loop. SPS depends on a CSD-space prototype bank and the perceptual scores come from a single VLM judge (Gemini 3.6 Flash) plus a 20-participant study on a 160-case subset; broadening judge coverage and human evaluation scale would strengthen the region-aware metrics the paper advocates.

Target Audience

Researchers and graduate students in generative video, diffusion/GRN-based image and video synthesis, and controllable editing who need paired data and region-aware metrics. It is also relevant to applied research and engineering teams in consumer hardware, film-technology, and creator-tool companies who want a deployable recipe for mask-conditioned style transfer, and to practitioners evaluating whether a style actually landed inside a selected region rather than merely somewhere in the frame. Readers should be comfortable with latent video generation, VAE conditioning, and style-transfer evaluation metrics; the paper is not an introductory treatment.

Authors’ abstract

Video stylization typically transforms entire scenes, while local artistic editing requires precise region--style association, spatial control, and temporal consistency. We introduce LociArt, a unified paradigm for free-style local video stylization that explicitly specifies where to stylize and what style to apply while preserving non-target content. At its core, GRNStyle builds on Generative Refinement Networks (GRN) and introduces a polar style-aware mask that embeds style identities as equal-norm directions within selected regions. Complementary representation--geometry conditioning combines encoded mask features with raw-mask spatial support, integrating spatiotemporal context with geometric control. To provide paired supervision, we develop an automated pipeline combining global stylization, video object removal, matting, and compositing, and construct the LociArt-100K Dataset with 100K local stylization pairs. We further establish the LociArt Benchmark, comprising 80 source videos and 84 target masks covering 168 foreground/background stylization cases, with a Style Prototype Similarity (SPS) evaluation scheme for regional style quality and complementary measures of non-target preservation and temporal consistency. Experiments show that GRNStyle outperforms existing methods in regional style quality and spatial control while maintaining strong temporal consistency. Qualitative examples further illustrate joint stylization of multiple regions with the same or different learned styles, despite training only on single-region, single-style pairs. Together, these components provide a unified foundation for learning and evaluating controllable local video stylization. Project page: https://higherhu.github.io/lociart/.