Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning

AI summary

Overview

Research area: Computer vision and multimodal vision-language learning, specifically weakly-supervised dense video captioning (WSDVC) — the task of finding and describing multiple events in untrimmed videos when only ordered event captions (no timestamps) are available for training.

Technical level: Advanced. The paper assumes familiarity with Gaussian temporal masks, CLIP feature spaces, transformer decoders, contrastive/captioning losses, and the WSDVC evaluation protocol.

Scope: The paper proposes a single framework (SBS) that replaces rigid, text-only LLM-synthesized "transition captions" with visually grounded, selectively applied and adaptively localized transition supervision, and validates it on two benchmarks.

Paper details: arXiv:2609.04183v1 [cs.CV], 03 Sep 2026, CC BY 4.0, from Hanyang University, South Korea.

What This Paper Is About

In weakly-supervised dense video captioning, a model is given only an ordered list of captions per video and must figure out where each event starts and ends. Recent work (SAIL) tries to help by asking an LLM to invent a caption for every gap between two neighboring events, but these captions are written from text alone (so they can hallucinate content that is not in the video) and are always placed at the fixed midpoint of the gap with a fixed width. SBS instead asks a Vision-Language Model to first look at the frames, decide whether a gap actually contains a real transition, and then place and size the transition region based on where the video's visual content actually changes.

Key Contributions

1. Reformulating transition augmentation as discovery rather than synthesis. The paper reframes inter-event supervision in WSDVC from text-only caption generation into visually grounded transition event discovery, where the decision to inject supervision is driven by what the video shows.

2. Repurposing a VLM as a transition-search tool. Rather than using a VLM only as a caption generator, SBS generates frame-level narratives with BLIP-2, measures cosine dissimilarity between adjacent frame-caption embeddings, and feeds that signal into an adaptive gate that opens only when a gap shows a salient semantic change.

3. Adaptive inter-event masks. For gaps that pass the gate, SBS blends the temporal midpoint between neighboring event centers with the detected semantic change point (using interpolation coefficient alpha) and selects the mask width from a candidate set that maximizes cross-modal alignment between the masked video features and the transition caption embedding.

4. A gated attraction loss with similarity filtering. The training objective adds a gated attraction loss whose magnitude is scaled by the gate confidence, and only includes inter-event pairs whose alignment score exceeds a threshold, so noisy or hallucinated captions do not harm learning.

Main Findings

• State of the art on ActivityNet Captions. SBS reaches CIDEr 36.87 and F1 58.18, surpassing the previous weakly-supervised state of the art SAIL (CIDEr 35.38, F1 57.00), with SODA_c 6.49, METEOR 8.87, ROUGE-L 15.60, BLEU-4 2.47, R@Avg 56.13, and P@Avg 60.38. The authors attribute the localization gains to improvements in both recall (56.13) and precision (60.38).

• State of the art on YouCook2. SBS scores SODA_c 4.24, METEOR 3.99, CIDEr 16.28, ROUGE-L 5.80, BLEU@N 3.25, R@Avg 22.39, P@Avg 21.95, and F1 22.17, above SAIL (4.08, 3.63, 14.61, 5.42, 2.94, 20.76, 21.13, 20.94) and ILCACM (3.60, 3.41, 13.49, 4.75, 2.59, 17.76, 18.01, 17.88).

• Competitive with fully supervised baselines. On ActivityNet, SBS's CIDEr of 36.87 exceeds fully supervised CM2 (33.01), E2DVC (33.63), CACMI (33.80), and ROS-DVC (35.04), and its F1 of 58.18 exceeds all four (55.21, 56.14, 57.10, 55.50 respectively).

• Beats much larger MLLM-based systems. Against fully supervised MLLM methods on ActivityNet — TimeChat (7B, SODA_c 4.7, CIDEr 19.0, F1 36.9), VTG-LLM (7B, 5.1, 20.7, 34.8), TRACE (7B, 6.0, 25.9, 39.3), and TimeExpert (5.9B, 6.5, 28.4, 40.5) — SBS uses a 133M-parameter backbone without full supervision and achieves 6.49, 36.87, and 58.18.

• Each component adds measurable value. In the ablation, the base configuration scores SODA_c 6.34, ROUGE-L 15.26, CIDEr 35.03, F1 56.86. Adding VLM captions gives 6.29, 15.45, 35.73, 57.73; adding the narrative-aware gate gives 6.45, 15.43, 36.61, 57.67; adding adaptive masks instead gives 6.48, 15.50, 36.51, 57.80; and combining all three gives the best results, 6.49, 15.60, 36.87, 58.18.

• The gain comes from visual grounding, not from a particular VLM. Every VLM tested beats LLM-based SAIL — InternVL3-1B (CIDEr 36.71, F1 57.98), Qwen2.5-VL-3B (36.82, 57.62), xGen-MM-4B (36.45, 57.86), SmolVLM2-2.2B (36.50, 57.94), and BLIP-2-2.7B (36.87, 58.18, the best).

• Text captions are a better transition signal than raw visual features or random gating. Gate criteria comparison: Random (6.26, 15.32, 35.87, 56.88), Raw Video (6.39, 15.32, 36.34, 57.42), Caption (6.49, 15.60, 36.87, 58.18).

• Inter-event representations align better with their captions. Average cosine similarity between pooled inter-event visual features and caption features rises from 0.1460 (SAIL) to 0.2699 (SBS), with 0.2624 for the version without similarity filtering.

• The gate substantially outperforms always-injecting a transition. On a manually verified set of 95 inter-event gaps (36 with a transition, 59 without), drawn from 200 randomly sampled gaps in ActivityNet validation videos, SBS's gate achieves Recall 74.99, Precision 64.29, F1 69.23, versus SAIL's always-inject (Recall 100, Precision 37.89, F1 54.96) and random gating (37.90, 49.48, 42.92).

• Overhead is negligible. Training times are 1H 42M 31S (ILCACM), 1H 49M 50S (SAIL), and 1H 52M 53S (SBS); inference times are 7M 16S, 7M 35S, and 7M 51S; GPU usage is 33.08 GiB, 33.11 GiB, and 33.13 GiB. Captions are extracted in advance and mask width selection is a dot product.

• The best interpolation coefficient is 0.5. SAIL's fixed midpoint (alpha = 0) yields the lowest scores, and intermediate values of alpha give stable gains, with alpha = 0.5 best.

Methodology in Plain English

The starting point is an existing reconstruction-based WSDVC pipeline (ILCACM/SAIL) where each predicted event is represented by a soft Gaussian mask over video frames, and the model learns by reconstructing captions from masked features and aligning masked video features with text in CLIP space.

SBS adds two things on top of that.

Deciding whether a gap deserves a transition. The authors run BLIP-2 over the frames of the video to produce a sentence for each frame — a "narrative flow." Each sentence is embedded with a CLIP text encoder, and for every gap between two consecutive predicted event centers, the method computes the cosine dissimilarity between neighboring frame captions. A high dissimilarity suggests the narrative changed, which the authors take as a proxy for a real event transition, citing cognitive science work on humans segmenting activity at points of perceptual change. Instead of a global threshold, each gap gets its own adaptive threshold: the mean dissimilarity in that gap plus beta times the gap's standard deviation. A sigmoid gate converts the gap's maximum dissimilarity minus this threshold into a confidence score between 0 and 1, and the gate opens when the score is at least 0.5. Gaps shorter than 4 frames are skipped entirely. The authors argue captions give a more semantically abstract signal than raw pixel features, which are easily confused by camera motion and lighting.

Deciding where and how wide the transition is. For an accepted gap, the method finds the frame with the largest dissimilarity — the semantic change point. It then blends the midpoint between the two neighboring event centers with this change point using a coefficient alpha (0.5 works best in the reported experiments), so that the transition is anchored near the actual change while still being prevented from collapsing onto either neighboring event. The caption generated at that frame becomes the transition's description. To set the width, it tries each candidate width from {0.2, 0.4, 0.6}, builds the corresponding Gaussian mask, average-pools the masked video features, and keeps the width whose cosine similarity with the transition caption embedding is highest.

Training and filtering. The total loss combines a captioning loss, a contrastive loss, and the new attraction loss. The attraction loss is multiplied by the gate confidence and computed only over intervals where the gate opened and the alignment score exceeds a threshold theta of 0.2, so unreliable caption-region pairs are excluded.

Experimental setup. Features come from CLIP ViT-L/14, the caption decoder is Distilled-GPT2, and training uses AdamW with a learning rate of 1e-4 on a single NVIDIA A6000 GPU. ActivityNet Captions is trained for 10 epochs in both stages; YouCook2 uses 4 and 25 epochs for the captioning and localization stages. Hyperparameters are alpha = 0.5, beta = 2, theta = 0.2, candidate widths {0.2, 0.4, 0.6}, and lambda_attr = 0.4. Frames are sampled at 32 per video for ActivityNet and 100 for YouCook2, and the number of event queries is 22 and 18 respectively. Evaluation uses SODA_c, METEOR, CIDEr, ROUGE-L, and BLEU-N for captioning and mean Average Precision, mean Average Recall, and F1 for localization, averaged over IoU thresholds of 0.3, 0.5, 0.7, and 0.9.

Why This Matters

Research impact. The paper challenges a common shortcut in weakly-supervised video-language work: using an LLM to hallucinate training signal from text alone. It shows that grounding that signal in what a VLM actually sees — and applying it selectively rather than uniformly — produces better results on both captioning and localization, and it introduces a human-verified protocol for evaluating transition detection in gaps that have no ground-truth annotations.

Real-world applications:

• Automatic indexing and chaptering of long video archives, where a system must both find event boundaries and label them without frame-level annotation budgets.
• Video search and retrieval over user-generated or surveillance footage, where matching a query to the right time span is as important as matching the content.
• Assistive technologies that describe ongoing video to visually impaired users, requiring correct segmentation of continuous activity into describable events.
• Training-data curation pipelines, where grounded VLM narratives can replace or filter noisy LLM-generated pseudo-labels.

Industry relevance. The method adds only about 3 minutes of training time and 16 seconds of inference time over the baseline, and roughly 0.05 GiB more GPU memory, so the accuracy gains do not require a step change in compute. Beating 7B-parameter fully supervised MLLM systems with a 133M-parameter weakly supervised model is directly relevant to deployment settings where annotation cost and inference budget are the binding constraints.

Future Directions

• Reducing dependence on VLM caption quality. The authors state in their limitations section that when the VLM produces generic, repetitive, or inaccurate descriptions — particularly in domains underrepresented in its pre-training — the caption-space dissimilarity signal becomes unreliable, causing the gate to either miss genuine transitions or open on spurious ones.

• Extending the manual transition-validation protocol. The human-verified evaluation set contained only 95 gaps from ActivityNet validation videos; scaling this to more gaps, more videos, and to YouCook2 would allow stronger conclusions about gate behavior.

• Exploring signals beyond caption-embedding dissimilarity. The paper compares random gating, raw CLIP visual features, and caption-based gating, and the caption signal wins; whether other semantically abstract representations improve further is left open.

• Testing on more diverse and longer video domains. Both benchmarks are relatively short by modern standards (120 seconds average for ActivityNet, 320 seconds for YouCook2), so generalization to longer or non-cooking, non-generic domains remains an open question.

Target Audience

Researchers and graduate students working on video understanding, dense video captioning, temporal grounding, and weakly-supervised multimodal learning will benefit most, especially those familiar with the ILCACM/SAIL line of work. Practitioners building video indexing, search, or description systems who cannot afford dense temporal annotation will also find the accuracy-versus-cost analysis and the negligible-overhead comparison useful. Readers without background in CLIP feature spaces, Gaussian temporal masks, or the WSDVC evaluation protocol will need to consult the cited prior work first.

Authors’ abstract

Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.