BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

AI summary

Overview

Research area: Computer vision — evaluation benchmarks for video object removal and video editing, spanning generative video models, vision-language model (VLM) judges, and physical/causal reasoning.

Technical level: Intermediate. The paper is readable without deep mathematics, but understanding the benchmarking results requires familiarity with standard video reconstruction metrics (PSNR, SSIM, LPIPS, FVD) and with mask-based versus instruction-driven video editing paradigms.

Scope (one sentence): The paper introduces BeyondMasks, a 180-sequence paired benchmark of synthetic and real-world videos with clean background references, and CORE, a VLM-based scoring protocol that measures whether object removal eliminates not just the object but the physical effects it induced.

What This Paper Is About

Most video object removal benchmarks score only how well a model reconstructs the pixels inside a mask, treating the task as local inpainting. In real scenes, however, removing an object is a causal intervention: the object also casts shadows, appears in reflections, changes illumination, scatters light through translucent material, and leaves dynamic traces such as footprints or dust, and those effects often extend well beyond the mask. The paper's goal is to build a benchmark with temporally aligned clean references so that researchers can systematically test whether a removal method recovers the scene as if the object had never been present.

Key Contributions

1. A causally grounded formulation of video object removal. The authors formalize removal as a counterfactual rendering problem: given a frame produced by a rendering process V_t = R(S_t, O), removal should approximate the interventional frame V_t^bg = R(S_t, ∅), not merely hallucinate plausible pixels inside a mask.

2. BeyondMasks, a paired benchmark of 180 video sequences. The dataset contains 90 synthetic scenes (generated with Veo3's video editing capabilities) and 90 real-world tripod-captured scenes, each with a temporally aligned clean background reference, per-frame object masks, and validated text prompts. The authors state this more than doubles the scale of prior paired video object removal benchmarks (Table 1 lists HQVI at size 10 and ROSE-Bench at size 60, versus 180 here).

3. CORE (Causal Object Removal Evaluation), a structured VLM-based metric. CORE uses Gemini 3.1 with a fixed comparative prompt over three temporally aligned inputs — the object-present video, the clean background, and the edited output — and decomposes scoring into ObjectScore (CORE-OS) for object disappearance and AfterEffectScore (CORE-AES) for elimination of induced physical traces.

4. A benchmark of contemporary removal and editing methods exposing systematic failures. The evaluation shows that methods achieving high masked-region fidelity frequently leave behind shadows, reflections, illumination changes, and dynamic perturbations.

Main Findings

• Pixel metrics diverge from causal correctness. DiffuEraser attains the highest PSNR (25.1880), SSIM (0.8769), and lowest LPIPS (0.1124) in Table 2, yet its CORE-AES (2.625) is lower than several competing methods, indicating incomplete removal of object-induced effects. The paper text states that ROSE achieves the highest CORE-OS despite only moderate reconstruction metrics; Table 2 lists EffectErase with the highest CORE-OS (4.015) and CORE-AES (3.162) in the reported table.

• Text-guided methods score lowest on CORE. Lucy Edit (T) records the lowest PSNR (18.5884), highest FVD (480.08), and the lowest CORE-OS (1.443) and CORE-AES (1.551).

• CORE aligns closely with human judgment. In a user study with 8 annotators evaluating 5 methods across 20 videos, Pearson correlation between CORE and mean human ratings was 0.615 for ObjectScore and 0.733 for AfterEffectScore, against human inter-rater agreement of 0.693 and 0.785 for the same dimensions.

• Difficulty varies sharply by after-effect type. Averaged across the top-performing six models per category (Table 3), translucent effects show the highest CORE-AES (2.701) and reflections the second highest (2.446), while steam/scattering is lowest (2.032) and causal physical effects second lowest (2.141); shadows reach 2.520 and light source effects 2.248. Reflection has the best PSNR (23.9842) and SSIM (0.8558), while light source effects have the worst PSNR (22.4233) and translucent the worst LPIPS (0.2162).

• After-effect removal is harder than object removal. Across the qualitative examples, most methods erase the primary object successfully, but secondary phenomena that are spatially diffuse and extend beyond the mask persist.

• Supervision modality trades off failure modes. Mask-based methods remove the object well thanks to explicit spatial supervision but often leave environmental residue; text-guided methods are more flexible but inconsistently remove fine physical traces and may hallucinate new content.

• Mask-based pipelines discard useful context. Many mask-based methods zero out the masked region before inference, deleting evidence such as background partially visible through translucent objects, which forces hallucination instead of recovery.

• Category distribution is natural, not balanced. Table 4 reports 127 shadow, 61 reflection, 46 translucent, 37 causal physical effect, 38 light source, 20 steam/scattering, and 6 no-explicit-after-effect sequences; categories are not mutually exclusive and counts do not sum to 180.

Methodology in Plain English

The authors treat object removal as an intervention experiment. For each scene they obtain two versions: one with a target object present and one clean background of the same scene, aligned frame by frame so the only difference is the object and everything the object causally causes.

• Synthetic half (90 scenes): They first generate a background-only video with Veo3, then insert the target object into that same scene through controlled editing. Because the object-present video is derived from the clean one, temporal alignment is preserved and the object's insertion naturally produces coherent shadows, illumination changes, reflections, translucency, and dynamic traces. Every pair was manually reviewed for physical plausibility and temporal coherence.
• Real-world half (90 scenes): Using a tripod-stabilized setup, they record a short clip with the object present and immediately capture a second clip after removing it, keeping camera position and settings fixed so geometry, lighting, and composition stay constant.
• Masks: Generated semi-automatically with SAM2 for interactive segmentation plus CoTracker 3 for temporal propagation, with annotators refining the first frame using positive and negative point prompts, periodically correcting propagated masks, and cross-reviewing each sequence. Critically, masks cover only the target object and deliberately exclude shadows, reflections, and other after-effects, so models must infer secondary effects rather than receive supervision for them.
• Evaluation (CORE): Rather than comparing pixels, CORE feeds the object-present video, the clean reference, and the edited output to Gemini 3.1 under a fixed structured prompt with metadata about the target object and relevant after-effect categories. The model identifies object traces in the input, infers the expected appearance from the clean reference, compares the edited output against it under a naturalness criterion, and classifies residual errors as object-related or after-effect-related, producing CORE-OS and CORE-AES. The rubric rewards semantically coherent background reconstruction even with pixel differences, and penalizes residual object evidence, persistent after-effects, and hallucinated foreground content.
• Reference baselines: Nine recent methods are described as benchmarked across mask-based, text-guided, and hybrid paradigms — ProPainter, DiffuEraser, Generative Omnimatte, OmnimatteZero, VACE, Lucy Edit, CoCoCo, MiniMax, and ROSE — evaluated with public implementations at default configurations on the native resolution of BeyondMasks, using PSNR, SSIM, LPIPS against paired ground truth, FVD for temporal quality, and CORE-OS/CORE-AES. Table 2 additionally reports EffectErase, so the table lists ten methods while the text describes nine. Additional supplementary analyses (dataset composition, method reviews, image-metric comparisons, synthetic-versus-real breakdowns, CORE robustness, and prompt templates) are referenced but their numbers are not included in the available content.

Why This Matters

Impact on research: The paper argues that a whole class of evaluation practice — scoring masked-region reconstruction and treating high PSNR/SSIM/LPIPS as success — systematically hides a failure mode. By publishing a paired benchmark with clean references and a validated VLM judge, it gives the field a way to measure causal consistency, and it reframes the task from local inpainting to intervention-consistent scene recovery.

Real-world applications:

• Film and video post-production, where removing a prop, rig, or crew member must also remove the shadow, reflection, and light spill it created, not just its pixels.
• AR and VR scene editing, where a deleted virtual or real object must not leave a mismatched shadow or mirror image that breaks the illusion.
• Street-view and mapping imagery, where cars and pedestrians must be removed along with their reflections on wet asphalt, glass storefronts, and vehicle paint.
• Surveillance and privacy redaction, where removing a person must also remove shadow, reflection, and footprint traces that could re-identify the scene or reveal the redaction.
• Content creation and social media tools, where instruction-driven "remove this" edits currently produce inconsistent physical results in consumer workflows.

Industry relevance: The finding that text-guided models score lowest on both CORE-OS and CORE-AES, and that mask-based pipelines lose useful context by zeroing the mask before inference, is directly actionable for teams building generative video foundation models and instruction-based editors. CORE offers a scalable automated evaluation signal, which is attractive for model development loops where human review of every generated clip is impractical.

Future Directions

• Physics-aware removal architectures. The paper's context-loss analysis suggests future systems should incorporate context-aware reasoning that exploits partially visible background through translucent objects and temporal redundancy across frames, rather than discarding masked pixels before inference.
• Better handling of volumetric and dynamic after-effects. Steam/scattering (CORE-AES 2.032) and causal physical effects such as footprints, water ripples, and deformation (CORE-AES 2.141) are the weakest categories; the paper frames these as unresolved for current methods.
• Extending the causal formulation beyond removal. The intervention framing (recovering R(S_t, ∅)) generalizes to other counterfactual edits, such as object insertion or relocation, which are not explored here.
• Strengthening and broadening VLM-based evaluation. CORE's correlation with humans (0.615 and 0.733) approaches but does not reach the human inter-rater ceiling (0.693 and 0.785); improving judge reliability, scaling evaluation, and testing robustness across VLM evaluators remain open. Whether the benchmark's size of 180 sequences is enough to capture the full diversity of real-world interactions is also an implicit question.

Target Audience

Researchers and engineers working on video generation, video editing, and video object removal; benchmark and evaluation-metric designers, especially those building VLM-as-judge protocols; practitioners in film post-production, AR/VR, mapping, and privacy redaction who need removal that is physically consistent rather than only locally plausible; and students entering generative video research who want a clear example of how evaluation design shapes what a field optimizes for.

Authors’ abstract

Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.