Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models

AI summary

Overview

Research area: Computer vision, specifically multimodal video understanding and video scene segmentation using vision-language models (VLMs).

Technical level: Intermediate. The paper assumes familiarity with fine-tuning large multimodal models, detection metrics (F1, AP), and shot-based video representation, but explains its design choices clearly.

Scope: The paper introduces Scene-VLM, a fine-tuned vision-language model framework that segments long-form video into scenes by jointly reasoning over frames, subtitles, and metadata, while also producing confidence scores and natural-language rationales for each predicted boundary.

What This Paper Is About

Video scene segmentation means finding the narrative boundaries in a long video — the points where location, time, characters, or story theme change — so a movie or episode can be chopped into coherent scenes. Prior methods use encoder-based classifiers that look mostly at visuals, judge each shot on its own within a local window, and give no explanation for their decisions. This paper asks whether a fine-tuned vision-language model can do better by reading frames, dialogue, and metadata together and predicting boundaries sequentially, like following a story.

Key Contributions

1. A fine-tuned VLM framework for scene segmentation. Scene-VLM uses a structured multimodal shot representation (frames, subtitles, optional metadata such as actor identities) and replaces point-wise classification with sequential prediction, where each shot's verdict is causally conditioned on earlier verdicts.
2. A context–focus window design. A larger context window surrounds a smaller focus window where predictions are emitted, so every predicted shot has temporal evidence from both before and after it. In practice, 20 shots of context and a central focus window of 10.
3. A confidence scoring scheme for VLMs. Because VLMs emit text tokens rather than classification scores, the authors derive a per-shot confidence from the softmax logits of the "Yes" and "No" tokens, normalized as P(Yes)/(P(Yes)+P(No)), enabling controllable precision–recall trade-offs that were previously limited to encoder-based methods.
4. Rationale generation with minimal supervision. The model can be aligned, using a small set of human-annotated explanations, to produce coherent natural-language justifications for its boundary decisions, which the authors state is the first demonstration of rationale generation for this task.

Main Findings

• State of the art on MovieNet-318: Scene-VLM reaches 62.1 F1 and 66.8 AP, versus 55.3 F1 / 58.6 AP for MEGA and 48.4 F1 / 60.8 AP for TranS4mer. The paper reports gains of +6.8 F1 and +8.2 AP over MEGA, and +13.7 F1 and +6.0 AP over TranS4mer (the abstract phrases this as +6 AP and +13.7 F1 over the previous leading method).
• Zero-shot generalization to BBC Planet Earth: Trained on MovieNet-318 and evaluated zero-shot, Scene-VLM reaches 45.8 AP, beating TranS4mer (43.6 AP) by +2.2 AP. Prior work does not report F1 on BBC, so only AP is compared.
• Vision dominates, but text and metadata add complementary signal: Removing visual frames collapses F1 from 62.1 to 32.0. Removing the visual shot-ID markers costs 1.3 F1, while removing subtitles or actor IDs each costs roughly 1 point (61.1 and 61.3 F1 respectively). Visual-only reaches 58.6 F1, whereas subtitle-only (31.5 F1) and actor-only (24.8 F1) degrade sharply.
• Attention is distributed differently by measure: Summed attention is dominated by visual tokens and by previously generated output tokens. After normalizing each modality by its token count, subtitle and actor tokens receive attention comparable to visual tokens.
• The model attends forward: For predictions at shots 11 and 15 (averaged over 30 context samples sharing the same positive transition index), the prediction token attends more strongly to subsequent shots than to preceding ones, which the authors interpret as the model trusting its own earlier predictions and looking ahead for confirmation. Attention also peaks at the first and last input shots.
• Context margins prevent edge degradation: Without the focus mechanism, per-position F1 collapses at sequence boundaries. With it, performance stays consistent across positions, with no outliers beyond 3 standard deviations of the mean (band shown at 0.565 and 0.655).
• Sequential prediction and longer context both help: Reducing the focus window to a single shot degrades performance at every context size (e.g., 20&10 gives 62.1 F1 vs. 20&1 at 60.1), and larger context windows improve results (20&10 is 62.1 F1 vs. 10&10 at 58.4 and 5&5 at 55.8).
• More frames help modestly: F1 moves from 61.8 at K=1 frame per shot to 61.9 at K=2 and 62.1 at K=3, suggesting a single representative frame suffices for most shots.
• Larger models perform better: 1.5B parameters gives 55.9 F1 / 58.7 AP, 3B gives 59.6 F1 / 62.8 AP (+3.7 F1, +4.1 AP), and 7B gives 62.1 F1 / 66.8 AP (+2.5 F1, +4.0 AP over 3B).
• Video chaptering generalization under matched backbones: With both methods using a Qwen2.5-VL-7B backbone, Scene-VLM scores 32.2 F1, 63.9 tIoU, 10.6 SODA, and 52.2 CIDEr, versus 28.4 F1, 59.5 tIoU, 10.1 SODA, and 45.5 CIDEr for the adapted Chapter-LLaMA. Chapter-LLaMA with its original LLaMA 3.1-8B backbone remains highest in absolute terms (42.6 F1, 70.6 tIoU, 16.4 SODA, 82.4 CIDEr).
• Minimal supervision fixes rationale generation: Naive prompting produced 22/30 parsing failures and 14/30 hallucinations on 30 sampled transitions. After training on 35 human-annotated explanation examples, Scene-VLM + Explain produced 0/30 parsing failures and 0/30 hallucinations.

Methodology in Plain English

The researchers take a pretrained Qwen2.5-VL-7B model and fine-tune it on scene segmentation data. Each video is split into shots, and each shot is turned into a multimodal representation: 3 sampled frames (resized to 147 × 63 pixels, with a small numerical shot-ID tag overlaid in the top-left corner), the corresponding dialogue from Whisper transcripts or provided subtitles, and optional metadata such as actor identities. These are packed into an XML-style prompt averaging 7939 words.

The model receives 20 consecutive shots as context but is only asked to answer for the central 10. It outputs lines in the format "shot_id: Yes/No", where "Yes" means that shot ends a scene. Because outputs are generated left to right, each verdict depends on the verdicts before it. Training uses next-token prediction loss with LoRA (rank 8, alpha 16) for 4 epochs on roughly 29k MovieNet samples across 8 A100 40 GB GPUs.

For confidence, instead of adding a classifier head, the authors read the probability of the "Yes" and "No" tokens at each verdict position and normalize them into a score. For explanations, they first tried simply prompting the trained model to justify itself, which failed; they then curated 35 examples pairing boundaries with short human-written rationales and did an additional fine-tuning stage.

Evaluation covers MovieNet-318 (318 movies split 190/64/64 for train/validation/test, roughly 1000 shots each), zero-shot BBC Planet Earth (10 episodes, averaging 50 minutes, with 670 scenes and 4.9K shots total), and a reproducible VidChapters-7M subset of 1000 training samples and 300 evaluation videos.

Why This Matters

Research impact. This is presented as the first application of VLMs to video scene segmentation, shifting the field from encoder-based classifiers toward multimodal generative reasoning. It also shows that a VLM can expose calibrated confidence scores and generate explanations, two capabilities the paper's own capability comparison (Table 1) attributes to no prior method in either camp — classical methods lack explainability and sequential prediction, while Chapter-LLaMA lacks confidence scores and cinematic scene segmentation.

Real-world applications:

• Structured summarization and content organization: slicing long films and episodes into navigable scenes for catalogs and playback interfaces.
• Semantic retrieval: letting users search within long-form video by scene rather than by raw timestamp.
• Contextual advertising: placing ads at narrative boundaries so breaks land where they make sense.
• Editorial review workflows: the paper motivates rationales specifically for human editors who must judge whether a proposed boundary matches narrative intent.

Industry relevance. The work comes from Amazon Prime Video with academic collaborators at Ben-Gurion University and Tel-Aviv University, and the chaptering experiment targets web video chapter titles of the kind platforms display on uploads. The authors note the model can be aligned to generate rationales with only 35 annotated examples, which matters for deployment where explanation-labeled data is scarce.

Future Directions

• Looser output formats. The authors flag the structured "Yes/No" output as a limitation, since it trades generative flexibility for reliable confidence extraction, and suggest exploring more flexible schemes.
• Reinforcement learning for reasoning. They propose using RL to integrate explicit reasoning into the prediction process, aiming to improve accuracy and interpretability together.
• Scaling and efficiency. Model-size gains remained substantial even at 7B parameters, suggesting further scaling may help; the number of frames per shot is also a lever on token count versus accuracy.
• Cross-domain robustness. Zero-shot performance on documentary footage trails in-domain cinematic results, leaving open how well the approach transfers to domains with less visual structure than film — the paper attributes Chapter-LLaMA's weakness on cinematic content to films' stronger visual structure, implying domain-dependent behavior in both directions.

Target Audience

Researchers and engineers working on video understanding, multimodal large models, and long-form content analysis will get the most from this paper, particularly those interested in fine-tuning VLMs for structured temporal tasks. It is also relevant to practitioners in streaming and media platforms who need scene boundaries, chaptering, calibrated confidence thresholds, or human-readable justifications for automated editorial decisions. Readers should be comfortable with detection metrics and the basics of VLM fine-tuning.

Authors’ abstract

Segmenting long-form videos into semantically coherent scenes is a fundamental task in large-scale video understanding. Existing encoder-based methods are limited by visual-centric biases, classify each shot in isolation without leveraging sequential dependencies, and lack both narrative understanding and explainability. In this paper, we present Scene-VLM, the first fine-tuned vision-language model (VLM) framework for video scene segmentation. Scene-VLM jointly processes visual and textual cues including frames, transcriptions, and optional metadata to enable multimodal reasoning across consecutive shots. The model generates predictions sequentially with causal dependencies among shots and introduces a context-focus window mechanism to ensure sufficient temporal context for each shot-level decision. In addition, we propose a scheme to extract confidence scores from the token-level logits of the VLM, enabling controllable precision-recall trade-offs that were previously limited to encoder-based methods. Furthermore, we demonstrate that our model can be aligned to generate coherent natural-language rationales for its boundary decisions through minimal targeted supervision. Our approach achieves state-of-the-art performance on standard scene segmentation benchmarks. On MovieNet, for example, Scene-VLM yields significant improvements of +6 AP and +13.7 F1 over the previous leading method.