SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning

AI summary

Overview

• Research area: Computer Vision / video-language understanding, specifically Weakly-Supervised Dense Video Captioning (WSDVC).
• Technical level: Advanced (assumes familiarity with CLIP-style cross-modal alignment, masked attention, DETR-style event query decoders, and captioning metrics).
• Scope: The paper proposes SAIL, a method that makes Gaussian event masks semantically aligned with their event captions and uses LLM-generated synthetic captions as auxiliary supervision, evaluated on ActivityNet Captions and YouCook2.

What This Paper Is About

Dense video captioning normally requires expensive frame-accurate annotations marking where each event starts and ends. Weakly-supervised versions avoid that, training only on captions, but the current state-of-the-art method (ILCACM) builds its event masks purely by making them non-overlapping, so the masks end up uniform and unrelated to what the captions actually describe. SAIL fixes this by steering masks toward video regions that are cross-modally similar to their own caption, and by using an LLM to invent plausible "in-between" captions that fill the temporal gaps left by sparse annotations.

Key Contributions

1. Similarity-aware mask guidance. A margin ranking objective (L_sim) that maximizes cosine similarity between average-pooled masked video features and their own caption embedding while minimizing similarity to other event captions in the same video, forcing masks to cover semantically relevant regions rather than merely distinct ones.
2. LLM-based caption augmentation. A "Video Context Inference Expert" prompt fed to Qwen3-8B generates N_s − 1 transitional captions per video from consecutive ground-truth caption pairs, densifying supervision for sparse datasets.
3. Inter-mask auxiliary mechanism. Rather than injecting synthetic captions into the main contrastive loss as hard negatives, the method creates "inter-masks" centered at the midpoint between adjacent predicted event centers with a fixed width w^inter, and aligns them with the synthetic caption embeddings via a separate loss L_aug.
4. State-of-the-art validation. Experiments on ActivityNet Captions and YouCook2 show improvements in both captioning and localization, with the weakly-supervised model surpassing several fully-supervised baselines on many metrics.

Main Findings

• Masking baseline is suspiciously strong: A fixed, non-trainable mask baseline performs comparably to ILCACM on both captioning and localization, which the authors interpret as evidence that ILCACM only learns to spread masks over distinct temporal regions without capturing semantic event relevance.
• ActivityNet Captions results (SAIL vs. ILCACM): SODA_c 6.29 vs. 6.08, METEOR 8.63 vs. 8.48, CIDEr 35.38 vs. 33.42, ROUGE-L 15.29 vs. 14.77, BLEU-1 15.86 vs. 15.36, BLEU-2 8.30 vs. 8.12, BLEU-3 4.27 vs. 4.17, BLEU-4 2.30 vs. 2.26.
• ActivityNet Captions localization (SAIL vs. ILCACM with CLIP): R@Avg 54.39 vs. 53.72, P@Avg 59.87 vs. 58.92, F1 57.00 vs. 56.20. SAIL's F1 of 57.00 exceeds the fully-supervised CM² (55.21) and E²DVC (56.14) figures reported in the same table.
• YouCook2 results (SAIL vs. ILCACM): SODA_c 4.08 vs. 3.60, METEOR 3.63 vs. 3.41, CIDEr 14.61 vs. 13.49, ROUGE-L 5.42 vs. 4.75, BLEU@N 2.94 vs. 2.59, R@AVG 20.76 vs. 17.76, P@AVG 21.13 vs. 18.01, F1 20.94 vs. 17.88. The authors note the ILCACM localization numbers on YouCook2 are their own re-implementation because the original paper does not report them.
• Ablation of components (ActivityNet): Baseline 6.08 SODA_c / 8.48 METEOR / 33.42 CIDEr / 56.20 F1; similarity-aware mask only 6.27 / 8.58 / 35.18 / 56.89; synthetic captions + inter-mask only 6.29 / 8.52 / 34.92 / 56.79; both 6.29 / 8.63 / 35.38 / 57.00. The paper states the similarity-aware mask alone improves CIDEr by +1.76 and SODA_c by +0.21.
• Synthetic caption scaling: Performance improves monotonically with the fraction of synthetic captions used: 0% baseline 6.08 / 33.42 / 14.77 / 56.20; 25% 6.08 / 33.76 / 15.02 / 56.38; 50% 6.27 / 34.19 / 15.00 / 56.39; 75% 6.23 / 34.66 / 15.09 / 56.91; 100% 6.29 / 35.38 / 15.29 / 57.00.
• Synthetic caption usage strategy: Using synthetic captions as hard negatives ("Synth + HN") gives SODA_c 6.24, METEOR 8.44, CIDEr 33.84, ROUGE-L 15.10, BLEU-4 2.02, already beating ILCACM. The inter-mask auxiliary approach ("Synth + Inter") is best at 6.29 / 8.63 / 35.38 / 15.29 / 2.30, confirming that soft guidance beats hard constraints.
• Mask type robustness: Gaussian masks perform best (SAIL CIDEr 35.38, SODA_c 6.29). Hard binary masks are much weaker (SAIL 4.09 SODA_c, 6.53 METEOR, 17.54 CIDEr, 11.07 ROUGE-L, 35.81 R@Avg, 51.00 P@Avg, 42.00 F1), and Cauchy masks sit in between (SAIL 6.18 / 8.66 / 33.85 / 15.06 / 54.36 / 59.11 / 56.63). SAIL improves over the baseline for every mask type tested.
• Mask width diversity: Mean standard deviation of mask widths across training videos is 0.3535 for SAIL versus 0.3489 for ILCACM, with min/max of 0.2549/0.3914 versus 0.2690/0.3900. For one tracked event, ILCACM's width barely moves (0.3530 → 0.3530 → 0.3525 → 0.3507 → 0.3523) while SAIL's decreases (0.3529 → 0.3489 → 0.3467 → 0.3398 → 0.3281), tracking a shorter actual event duration.
• Annotation sparsity is a real problem: Reducing ground-truth caption density from 100% to 25% consistently degrades both CIDEr and SODA_c on the baseline, and the paper cites an ActivityNet example of a 235-second video with only three annotated events.
• Computational cost is negligible: Training time 1H 41M 59S for SAIL versus 1H 38M 27S for ILCACM; inference 7M 01S versus 7M 11S; GPU memory 33.11 GiB versus 33.06 GiB. LLM caption augmentation is a one-time text-only preprocessing step taking 0.28 hours for ActivityNet.
• Hyperparameters: Δ = 0.1, w^inter = 0.6, α_aug = 0.25, fixed across all experiments; robust performance is reported across a range of w^inter and α_aug values.

Methodology in Plain English

The starting point is a system where each event query predicts a Gaussian-shaped soft mask over the video timeline, defined by a center and a width. That mask multiplies the video features to make a "positive" view of one event, while the inverse mask makes a "negative" view of everything else; captions are generated for both and constrained to reconstruct the full video description. This is the ILCACM baseline.

SAIL adds two things. First, it takes the pooled masked features and pushes them, via a margin ranking loss, to be cosine-similar to their own caption and dissimilar to the other captions in the same video. Because the mask parameters are differentiable, this gradient flows back into the mask geometry: masks that cover the wrong temporal region get penalized. Second, because datasets are sparse (ActivityNet averages 3.7 events across videos averaging 120 seconds, and the paper shows a 235-second video with only three annotations), the method asks Qwen3-8B to write one plausible transitional caption between each adjacent pair of real captions, producing N_s − 1 synthetic captions per video. These synthetic captions are not used as hard negatives; instead, new inter-masks are centered at the average of adjacent predicted event centers with fixed width w^inter = 0.6, and a soft loss pulls those inter-mask features toward the synthetic caption embeddings. The total objective is L_pos + L_neg + L_sim + α_aug·L_aug, with α_aug = 0.25. Training uses CLIP ViT-L/14 features, Distilled-GPT2 as the caption decoder, and AdamW.

Why This Matters

Impact on research: The paper challenges the assumption that weak supervision only needs to separate events temporally; it shows that injecting semantic alignment into the mask construction itself changes the learned geometry, and that LLM-generated captions can substitute for some temporal boundary supervision. The fact that a weakly-supervised model exceeds fully-supervised CM² and E²DVC figures on several ActivityNet metrics suggests the boundary between the two settings is narrower than usually assumed.

Real-world applications:

• Automatic indexing and chaptering of long instructional or lecture videos from caption text alone.
• Searching untrimmed user-generated video libraries without frame-level annotation budgets.
• Generating timestamped event descriptions for sports, cooking, or how-to content at scale.
• Assistive or accessibility tooling that produces navigable event summaries of long videos.

Industry relevance: The method adds only a dot-product operation over the baseline, with training time, inference time, and GPU memory within noise of ILCACM (33.11 GiB vs. 33.06 GiB), and the LLM step is a one-time 0.28-hour text-only preprocessing pass. That makes it attractive for production pipelines already built around masked-captioning baselines.

Future Directions

• Extending the LLM-augmentation idea to other multimodal tasks where data scarcity limits learning, as the authors state in their conclusion.
• Determining how well the approach scales to datasets with even sparser annotations than the 25% density tested, and where synthetic caption quality degrades.
• Addressing the remaining gap on localization metrics, where SAIL's ActivityNet F1 of 57.00 is still well below the fully-supervised SDVC (56.56), PDVC (56.71), and E²DVC (56.14) recall-precision balance on the same benchmark.
• Understanding when LLM-generated transitional captions introduce harmful hallucinations; the paper reports the hard-negative variant degraded BLEU-4 from 2.30 to 2.02, but does not report a systematic error analysis of synthetic caption accuracy.

Target Audience

Researchers and graduate students working on video-language models, dense video captioning, and weak supervision, plus practitioners building video indexing or captioning systems who need temporal localization without paying for boundary annotations. A reader should already be comfortable with CLIP-style contrastive alignment, Gaussian attention masks, and standard captioning metrics (METEOR, CIDEr, ROUGE-L, BLEU-N, SODA_c), since the paper introduces them without background exposition.

Authors’ abstract

Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.