ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

AI summary

Overview

Research area: Computer vision — video generation and editing, specifically scene text editing in real-world videos, plus benchmark design and evaluation methodology.

Technical level: Advanced. The paper combines a data-construction pipeline built on segmentation, inpainting, and vision-language models with a 13-metric evaluation protocol and a fine-tuned 14B-parameter video editing model.

Scope (one sentence): ViTeX-Bench is a benchmark suite for replacing text rendered on surfaces (signs, whiteboards, labels) in real videos while preserving the rest of the scene, consisting of a dataset, a three-axis evaluation protocol, eight baseline evaluations, and an open-source reference editor.

Affiliations for all authors are Texas A&M University. The paper is arXiv:2609.40356v1 [cs.CV], dated 30 Sep 2026.

What This Paper Is About

Video generation has become realistic and controllable, but localized video editing — changing a small region while leaving everything else intact — remains comparatively underdeveloped. This paper targets video scene text editing: replacing text that appears on scene surfaces in a video (storefront signs, whiteboards, jerseys, screens, product labels) so that the requested string is rendered correctly, stays attached to the same surface as the camera or object moves, and preserves surrounding appearance, lighting, and motion. The authors' stated problem is that paired real-video data is scarce and existing general video-editing metrics do not directly measure whether the requested text remains correct over time, so they build a dataset, a task-specific evaluation protocol, and a reference editor to study the trade-offs.

Key Contributions

1. ViTeX-Dataset: 387 real-world 720p source videos screened from Panda-70M and InternVid, with per-frame text-region masks and source–target string pairs. 230 videos form a training split with reviewed, pipeline-generated paired edits (56 built with Strategy A alpha composition, 174 with Strategy B using a PISCO-based inserter); 157 videos form a permanently frozen evaluation split that withholds the paired edit.

2. A three-axis evaluation protocol with 13 metrics covering text correctness, visual and temporal quality, and edit locality. Each axis has one primary metric — SeqAcc (up), text-crop Warp (down), and DreamSim-loc (down) — with a Pareto comparison of their trade-offs rather than a weighted aggregate.

3. ViTeX-Edit-14B, an open-source reference editor built on a pretrained Wan2.1-VACE-14B backbone, adapted with a motion-aligned glyph-video conditioning stream that supplies target-character structure along the source text trajectory.

4. Calibration and robustness analyses: OCR calibration, a method-blinded human transcription study, a three-rater subjective study of 70 outputs, an independent mask re-annotation audit on 12 difficulty-stratified clips, bootstrap ranking-stability analysis, and supplementary background-motion and identity probes.

Main Findings

• No method wins on all three axes. Across eight baselines from four editing families plus the reference editor, accurate text, temporal stability, and scene preservation remain difficult to achieve together.

• Per-frame image editors lead on character accuracy. FLUX-Text reaches SeqAcc 0.528 and CharAcc 0.737; TextCtrl reaches SeqAcc 0.475 and CharAcc 0.734. FLUX-Text also shows large text-region residuals (Flicker-c = 14.81, Warp-c = 13.01), consistent with its independent per-frame edits producing flicker and glyph drift.

• Video-native editors often fail to render the requested string at all. Wan2.1-VACE-14B and Kling Video 3.0 Omni both score SeqAcc 0, reflecting unchanged or incorrectly rendered text. Kling still leads both MUSIQ measures (72.23 full-frame, 47.75 crop), showing that visual polish does not imply successful text replacement.

• ViTeX-Edit-14B achieves the highest mean CharAcc among the evaluated video-native editors, at 0.688 versus VideoPainter's 0.619 (a gain of +0.069, or 11.1% relative). VideoPainter has higher SeqAcc (0.364 vs. 0.341), so improved partial character accuracy did not yield more exact strings. The paper notes the intervals overlap and these describe observed means rather than established pairwise significance.

• The reference editor has the lowest temporal error among comparable raw outputs. ViTeX-Edit-14B records Flicker-f 3.27, Flicker-c 3.42, Warp-f 1.55, and Warp-c 1.53.

• The Source video row scores TTS 0.760 despite never making the requested edit, illustrating that the temporal text stability metric measures stability rather than correctness and must be read with SeqAcc and CharAcc.

• Composite post-processing mostly restores source pixels. For ViTeX-Edit-14B, Composite raises PSNR-loc from 29.08 to 42.95 dB and reduces DreamSim-loc from 0.024 to 0.002, while SeqAcc changes only from 0.341 to 0.345. Applied to all eight baselines, it brings PSNR-loc to approximately 43 dB and full-frame Flicker toward the source value of 3.72, indicating source-pixel restoration accounts for much of the locality gain.

• The Pareto set on the three primary metrics is FLUX-Text, TextCtrl, RS-STE, ViTeX-Edit-14B, and Wan2.1-VACE-14B. Wan2.1-VACE-14B remains non-dominated at SeqAcc 0, so membership alone does not establish editing success.

• OCR is well calibrated on readable source frames, achieving exact-match accuracy 0.851 and CharAcc 0.966 with TTS 0.760. Method-blinded human transcription agrees with the OCR-based method ranking at Spearman rho = 0.95.

• Human raters agree strongly on text and weakly on locality. Three non-author raters evaluating 70 outputs on 1–3 scales produced ordinal Krippendorff agreement of 0.87 for text, 0.80 for temporal quality, and 0.37 for locality. Mean ratings correlate with SeqAcc, text-crop Warp, and DreamSim-loc at +0.71, −0.40, and −0.53 respectively (p < 0.001). Text-crop Warp aligns more closely with temporal ratings than full-frame Warp (−0.40 vs. −0.20), supporting its choice as the temporal primary.

• Rankings are stable within the sampled domain. Bootstrap resampling of the 152 source-detectable clips (1,000 replicates, seed 2064) yields mean Kendall tau = 0.936 against the full SeqAcc ranking and retains the leading method in 95% of size-152 replicates.

• Non-Latin performance is much weaker. In the non-Latin slice, AnyText2 leads with SeqAcc 0.168 and CharAcc 0.295; all other editors have CharAcc below 0.19.

• Four diagnostic failure modes. Wan2.1-VACE-14B returns the masked region essentially as the source; FLUX-Text yields legible but temporally inconsistent glyphs; Kling produces high per-frame quality with incorrect text; TextCtrl+AnyV2V drifts in both text and surrounding scene structure.

Methodology in Plain English

Building the data. The authors searched Panda-70M and InternVid with keyword queries and manually screened videos for editability and absence of sensitive content. For each retained video they assembled four assets: a dilated text-region mask annotated with a semi-automatic GUI built on the SAM 3 video segmentation model (an annotator marks keypoints on the first frame, SAM 3 propagates the mask, and morphological dilation adds margin); a source–target string pair proposed by Qwen3-VL-32B-Instruct, which reads the source string from the first-frame mask crop and suggests a similar-length replacement that an annotator then audits; a clean background video produced by removal-1.3B, a fine-tuned version of Wan2.1-VACE-1.3B that removes glyphs along with their cast shadows and highlights in the spirit of ROSE; and a first-frame target-text patch created by rewriting the first frame with the image editor Gemini 3 Pro Image (Nano Banana Pro), masked by a second SAM 3 pass. Two construction strategies handle motion: Strategy A alpha-composites the target patch onto each frame of the clean background and applies only to static text regions, while Strategy B uses a PISCO inserter fine-tuned on an auxiliary scene-text insertion set with amodal-completion supervision to handle dynamic videos. Dynamic videos use Strategy B exclusively; static videos run both and keep the higher-quality output.

Defining the task. Each instance is a tuple of source video V, per-frame binary mask M, source string, and target string. The videos are 120 frames at 720×1280 and 24 fps. A method outputs an edited video that should render the target string inside the mask while leaving the rest unchanged.

Measuring success. Text correctness is measured by running the PP-OCRv5 recognizer on the dilated-mask crop of each source and predicted frame. Because source text is not always readable, correctness is scored only on frames where source-frame OCR matches the source string at least halfway (Sim ≥ 0.5), and string comparison uses a substring edit distance that leaves unmatched prefixes and suffixes free, so a correct target embedded in a longer OCR string is not penalized. From this come SeqAcc (exact substring match), CharAcc (partial credit), and TTS (whether the decoded string is stable across adjacent detectable frames). Visual quality is scored at two scopes, the full frame and a static text-crop bounding box enclosing the union of all masks, using Flicker (adjacent-frame MAE), Warp (MAE after backward-warping by RAFT forward flow computed on the source video, so motion is compensated), and the no-reference MUSIQ quality estimator. Edit locality is measured by reconstructing a locality-only prediction that keeps predicted pixels outside the mask and substitutes source pixels inside, then averaging PSNR, SSIM, LPIPS, and DreamSim against the source. The three primaries are SeqAcc, text-crop Warp, and DreamSim-loc, compared through a Pareto set rather than a weighted score.

Building the reference editor. ViTeX-Edit-14B starts from Wan2.1-VACE-14B, which provides a text-encoding stream and a Video Condition Unit for the source video and mask. A third stream is added: a target-text glyph video, built by rendering the target string as a white-on-black glyph image in a typeface chosen to match the source font, detecting the source-text quadrilateral in the first frame, tracking it across frames, and projectively warping the glyph image with the resulting per-frame homographies so it follows the source text's position, scale, and perspective. A frozen Wan VAE encodes this glyph video to a latent, a stride-(1,2,2) patch embedding flattens it into tokens, and 64 learnable queries perform cross-attention pooling into a fixed-length glyph token bundle. Every VACE block queries that bundle through an added condition cross-attention layer whose output projection is zero-initialized, preserving backbone behavior at initialization. The authors report that they first tried using the glyph video at inference without fine-tuning and routing it through the existing VCU during fine-tuning, and both gave poor character correctness in qualitative pilots. During fine-tuning the main DiT trunk, the uMT5-XXL text encoder, and the Wan VAE are frozen; only the VACE branch, the glyph encoder, and the condition cross-attention layers are updated. Training follows a two-stage Flow-Matching supervised fine-tuning curriculum over 576 GPU-hours on 8× H100 80GB, and inference runs a single 50-step pass.

Composite control. Composite is a deterministic, training-free post-processing wrapper applicable to any editor. It matches the predicted region to the source through annulus-based LAB color transfer and blends the region onto the source with a 4-pixel feathered boundary. The same wrapper is applied to all eight baselines and to ViTeX-Edit-14B, with its effects reported separately from raw model performance.

Baselines. Eight baselines span four families: per-frame image editing (AnyText2, TextCtrl, FLUX-Text, RS-STE, whose per-frame outputs are concatenated into a video); first-frame editing and propagation (TextCtrl edited on the first frame and propagated by AnyV2V using the I2VGen-XL backbone); mask-conditioned video inpainting (Wan2.1-VACE-14B and VideoPainter, the latter with a CogVideoX 1.0 backbone requiring spatial resizing and linear-blend temporal upsampling, which alters adjacent-frame residuals so its Flicker and Warp scores are excluded from temporal rankings); and instruction-guided video editing (Kling Video 3.0 Omni with a fixed editing-instruction template).

Why This Matters

Impact on research. The paper makes character-level correctness over time an explicit, first-class measurement instead of relying on general perceptual or preservation scores. By releasing a frozen evaluation split, a 13-metric protocol with calibration analyses, and an open reference editor, it gives the video-editing community a reproducible way to separate "the video looks good" from "the requested text was actually rendered and stayed rendered." The Pareto framing also formalizes the observation that correctness, temporal stability, and locality trade off against each other, so single-number leaderboard rankings would be misleading.

Real-world applications:

• Advertising and signage localization — swapping pricing, language, or branding on storefront signs, billboards, and in-venue displays in existing footage without reshooting.
• Film and broadcast post-production — correcting or replacing on-screen text such as jerseys, screens, product labels, and whiteboards in already-shot material.
• Sports and event footage — updating sponsor or team text on jerseys and boards across a clip while keeping the underlying motion and lighting intact.
• Multilingual content adaptation — translating on-surface text in video catalogs for regional distribution.

Industry relevance. The benchmark's findings are directly relevant to teams choosing between per-frame image editors, propagation-based editors, and video-native diffusion editors. The results show that per-frame editors currently win on character accuracy but fail on temporal consistency, that mask-conditioned video models can score well on perceptual quality while never rendering the requested string, and that a simple deterministic post-processing wrapper recovers much of the locality gap for any editor. Those are practical engineering signals for anyone building a text-editing product on top of video diffusion models.

Future Directions

• Extending coverage. The dataset focuses on localized, readable, predominantly Latin-script text. Dense layouts, curved surfaces, severe occlusion, extreme motion, and non-Latin scripts remain sparsely covered; the weak non-Latin results (highest CharAcc 0.295, all other editors below 0.19) point to a clear gap.

• Controlled ablation of the glyph branch. The authors state that component ablations for ViTeX-Edit-14B remain future work, and that two simpler conditioning alternatives were rejected only on the basis of qualitative pilots.

• Training-scale and data-scale studies. The reference editor is trained on 230 paired videos; how performance scales with more paired data is an open question, and the paper notes that broader coverage requires additional data.

• Stronger provenance and independent auditing. Per-record target-string rejection/resampling counts and first-frame editing retry rates were not logged in the initial release, and other annotation stages besides masks lack independent agreement studies. Improving locality measurement is also open, since the human study found only 0.37 agreement on that axis.

Target Audience

Researchers and practitioners working on video editing, video diffusion models, and scene text editing will get the most from this paper, particularly those who need an evaluation protocol for character-level correctness over time rather than a general perceptual score. It also suits benchmark designers interested in three-axis evaluation, Pareto comparison, and calibration methodology, as well as applied teams in post-production, advertising, and localization evaluating whether current video editors can reliably replace on-screen text. Readers need a working familiarity with diffusion-based video models, OCR, and standard image quality metrics such as PSNR, SSIM, LPIPS, and DreamSim to follow the experimental sections.

Authors’ abstract

Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits that must preserve the original scene dynamics. Video scene text editing replaces text on scene surfaces, such as storefront signs, whiteboards, and product labels, while preserving the surrounding content, motion, and camera dynamics. Although scene text editing is well studied for images, video scene text editing that achieves high visual quality, temporal consistency, and edit locality remains underexplored. Existing resources offer limited paired real-video data, and general video-editing metrics do not directly measure whether the requested text remains correct over time. We introduce ViTeX-Bench, a benchmark suite comprising ViTeX-Dataset and a three-axis evaluation protocol. The dataset contains 387 real-world 720p videos with text-region masks and editing instructions: 230 provide reviewed, pipeline-generated paired edits for training, and 157 form a frozen evaluation split. The protocol evaluates text correctness, visual and temporal quality, and edit locality through 13 metrics, with one primary metric per axis and a Pareto comparison of their trade-offs. OCR calibration, human evaluation, and annotation-sensitivity analyses support the interpretation of these scores. Across eight baselines from four editing families, accurate text, temporal stability, and scene preservation remain difficult to achieve together. We also release ViTeX-Edit-14B, an open-source reference editor fine-tuned on the paired training split with motion-aligned glyph-video conditioning. It achieves CharAcc 0.688, the highest mean among the evaluated video-native editors, and the lowest comparable text-crop Warp among raw editor outputs. ViTeX-Bench provides a reproducible foundation for studying these trade-offs in video scene text editing.