AdSum: Two-stream Audio-visual Summarization for Automated Video Advertisement Clipping

AI summary

Overview

Research area: Computer vision and multimodal machine learning, applied to video summarization for the advertising industry.

Technical level: Intermediate. The paper assumes familiarity with deep learning building blocks (convolutions, attention, fusion strategies, loss functions), but its central idea — selecting shots from a 30-second ad to build a 15-second ad — is easy to follow without that background.

Scope: The paper introduces a new advertising-specific video summarization task, releases a paired 30-second/15-second ad dataset (AdSum204), and proposes a two-stream visual-audio model that predicts which shots advertisers kept when cutting a long ad into a short one.

What This Paper Is About

Advertisers often need multiple versions of the same ad at different lengths for one campaign, and today they do this by manually picking and re-editing shots from a long ad to make a shorter one. The authors reframe this manual labor as a machine learning problem: given all the shots in a 30-second ad, predict which subset of shots was used to produce the firm's 15-second version of that same ad. Their key argument is that advertising summarization differs from existing video summarization because the result must preserve a coherent storyline, and because audio carries essential information (product names, dates, calls to action) that visual-only methods ignore.

Key Contributions

1. A new task framing. The paper is the first to frame video ad clipping as a shot selection problem tailored to advertising, rather than as highlight detection as in sports or surveillance summarization.
2. AdSum204, a new dataset. 102 pairs of 30-second and 15-second video ads from real advertising campaigns — 204 video ads total — with shot boundaries and shot-to-shot mappings between the long and short versions. The paper states the dataset is larger than or comparable to existing public video summarization datasets.
3. A two-stream audio-visual model (AdSum). Separate visual and audio streams predict frame importance, using 3D CNN clip-level features in the visual stream and Wav2Vec2-BERT clip-level features in the audio stream, fused either before attention (early fusion) or after scoring (late fusion).
4. State-of-the-art results on AdSum204. AdSum with early fusion outperforms the benchmarked methods CA-SUM and CSTA on Average Precision, AUROC, Spearman, and Kendall metrics, and the paper shows the audio-only stream alone outperforms the visual-only stream.

Main Findings

• Early fusion wins. AdSum with early fusion achieved the best scores across all four metrics: AP 0.783, AUROC 0.665, Spearman σ 0.273, and Kendall τ 0.224, compared to late fusion at 0.764, 0.649, 0.245, and 0.201. The authors attribute this to fusing visual and audio embeddings before attention, which lets attention capture cross-modal interplay.
• Pre-trained general summarization models transfer poorly. CA-SUM pre-trained on TVSum scored AP 0.659 and AUROC 0.504, while on SumMe it scored AP 0.649 and AUROC 0.468. CSTA pre-trained on TVSum reached AP 0.651 and AUROC 0.477; on SumMe, AP 0.622 and AUROC 0.432. The authors note these models were not trained on video ads.
• Training on ad data helps substantially. CSTA trained on AdSum204 reached AP 0.773, AUROC 0.646, σ 0.241, τ 0.199, a large improvement over the same model pre-trained on TVSum or SumMe.
• Binary cross-entropy beats mean squared error. Replacing the MSE loss used in prior work with BCE improved AP from 0.773 to 0.777, AUROC from 0.646 to 0.647, and Spearman from 0.241 to 0.242 (Kendall stayed at 0.199).
• A half window size of 3 beats 5. CSTA with 3HWS plus BCE scored AP 0.781, AUROC 0.652, σ 0.251, τ 0.207, whereas 5HWS plus BCE scored AP 0.779, AUROC 0.650, σ 0.247, τ 0.204. The authors read this as evidence that 3D-CNN clip-level embeddings capture temporal information effectively.
• Audio outperforms vision. The audio-only stream (AdSum-A) beat the visual-only stream (AdSum-V) on all four metrics: AP 0.780 versus 0.775, AUROC 0.664 versus 0.654, σ 0.270 versus 0.256, and τ 0.224 versus 0.210.
• The two streams specialize by position in the ad. When shots were split into 0–50% and 50–100% of each video, the audio model performed better on the first half while the visual model performed better on the latter half, which the authors link to ads front-loading attention-grabbing audio and back-loading product visuals.
• A worked example. In a McDonald's ad titled "McHappy Day 2023" (referred to as "video 7"), the audio model selected ending shots better, which the authors connect to the closing narrative.
• Dataset statistics. The 30-second ads average 18 shots (minimum 8, maximum 38); the body text reports the 15-second ads average 8 shots (minimum 3, maximum 18), while the Figure 4 caption lists 10 shots on average. Average shot duration is reported as 1.67 seconds for 30-second ads and 1.52 seconds for 15-second ads in the text (the Figure 4 caption states 1.53 seconds for the 15-second ads). Shot durations range from 0.19 to 13.56 seconds in the 30-second ads and from 0.23 to 10.44 seconds in the 15-second ads.
• Frame rates. Of the 102 pairs, 74 are 23.98 FPS, 18 are 25 FPS, 9 are 29.97 FPS, and one pair is 30 FPS. All 30-second videos were standardized to 23.98 FPS.
• Not reported. Although the paper states that Precision, Recall, and F1 were computed from true positives, false positives, and false negatives, their numeric values are not reported in the provided content.

Methodology in Plain English

Building the dataset. The authors started with the top 50 fast-food chains by sales (per QSR Magazine) and the top 10 most popular soft drink brands in the USA, collected their YouTube channels via the YouTube Data API, and searched for ads that existed in both 30-second and 15-second versions from the same campaign. This produced 135 candidate pairs, of which 127 were English-speaking. Because the ground truth is the advertiser's own short ad — not a human annotator's opinion — the authors only had to work out which shots the advertiser reused.

Annotating the pairs. They ran TransNetV2 to detect shot boundaries in both ads, experimenting with probability thresholds of 0.1, 0.3, and 0.5. Then they computed SIFT features on the first, middle, and last frames of every shot and matched short-ad shots to long-ad shots by highest average similarity. All mappings were manually reviewed for 100% annotation accuracy. 33 pairs were dropped because shots were missing from the 30-second ad, leaving 102 valid pairs.

Predicting which frames matter. The pipeline has three steps: generate shots, predict frame importance, and select shots. Videos were sampled every 12th frame to match the 2 FPS convention of prior work, and frames from the same shot were grouped into clips using half-window sizes of 3 and 5.

• Visual stream: a pre-trained Swin3D video Swin Transformer turns each clip into a 1024-dimensional feature, chosen because it does not require a fixed input length. The resulting feature map is treated like an image, and GoogleNet produces an attention map, merged with the features via a Hadamard product. A linear classifier with a sigmoid outputs the clip's importance score.
• Audio stream: the same architecture, but Wav2Vec2-BERT embeds the temporally synchronized audio clip into a 1024-dimensional vector.
• Fusion: early fusion mixes the visual and audio feature maps with a weighting coefficient β before attention; late fusion mixes the two streams' final importance scores with a coefficient α. Both α and β were set to 0.5.
• Loss: binary cross-entropy, chosen because the ground truth labels are binary (selected or not selected), rather than the MSE loss used in prior summarization work.

Making the short ad. Frame importance scores are averaged within each shot, shots are ranked, and the top-ranked shots are added until the cumulative duration reaches the target.

Evaluation. Five-fold cross-validation split by ad pair ID, with Adam optimization, 50 epochs, batch size 1, and learning rate 0.001. Comparisons were made against CA-SUM (unsupervised) and CSTA (supervised) under their original implementations.

Why This Matters

Impact on research. The paper opens a new application area for video summarization and provides evidence that domain matters: models pre-trained on TVSum and SumMe transfer poorly to ads, and an audio-only model beats a visual-only model in this setting, which contradicts the intuition that visual features suffice for summarization.

Real-world applications:

• Automatically producing 15-second social media cuts from 30-second TV ads for the same campaign.
• Generating campaign variants at scale without re-editing by hand, reducing the cost and time the paper describes as labor-intensive and time-consuming.
• Supporting brands managing many simultaneous ad versions across platforms, since the method selects shots rather than re-shooting content.
• Guiding creative decisions by showing which shots advertisers systematically keep, and where audio versus visual content carries the selection signal.

Industry relevance. The authors focus on food and beverage because of the sector's heavy advertising investment and its market size, which the paper projects growing from $6.2 trillion in 2024 to $9.8 trillion by 2032 globally. The paper notes that industry leaders have called for solutions to the manual clipping problem, and the dataset, code, and annotations are released publicly for follow-on work.

Future Directions

• Scale the dataset. The authors note AdSum204 is larger than or comparable to existing public video summarization datasets but should be expanded with more video pairs and a broader range of industries beyond food and beverage.
• Deepen cross-modal interaction. The authors suggest exploring cross-attention mechanisms so visual and audio streams interact more richly than the current fusion strategies allow.
• Support other durations. While 30-second and 15-second ads dominate TV and most social media, the authors propose applying the framework to generate ad versions of other lengths without losing generalizability.
• Improve the model further. The paper acknowledges the two-stream model still has room for improvement even though it outperforms the benchmarks.

Target Audience

Researchers working on video summarization, multimodal learning, or audio-visual representation learning who want a new domain and benchmark to test against; advertising and marketing analytics practitioners interested in automating creative versioning; and industry data scientists building tools to adapt long-form ad content into shorter formats. Readers with a general deep learning background will find the paper accessible, while those without it can still follow the task definition, dataset description, and reported comparisons.

Authors’ abstract

Advertisers commonly need multiple versions of the same advertisement (ad) at varying durations for a single campaign. The traditional approach involves manually selecting and re-editing shots from longer video ads to create shorter versions, which is labor-intensive and time-consuming. In this paper, we introduce a framework for automated video ad clipping using video summarization techniques. We are the first to frame video clipping as a shot selection problem, tailored specifically for advertising. Unlike existing general video summarization methods that primarily focus on visual content, our approach emphasizes the critical role of audio in advertising. To achieve this, we develop a two-stream audio-visual fusion model that predicts the importance of video frames, where importance is defined as the likelihood of a frame being selected in the firm-produced short ad. To address the lack of ad-specific datasets, we present AdSum204, a novel dataset comprising 102 pairs of 30-second and 15-second ads from real advertising campaigns. Extensive experiments demonstrate that our model outperforms state-of-the-art methods across various metrics, including Average Precision, Area Under Curve, Spearman, and Kendall. The dataset and code are available at https://github.com/ostadabbas/AdSum204.