DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning

AI summary

Overview

Research area: Computer vision / vision-language models for instructional video understanding, specifically multi-scene video captioning for educational content.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, captioning metrics (BLEU, METEOR, CIDEr, BERTScore), and prompting techniques such as chain-of-thought, but the pipeline itself is described in accessible stages.

Scope in one sentence: The paper introduces DynaStride, a four-stage pipeline that samples frames from instructional video scenes, generates action-object subcaptions with a vision-language model, filters redundant windows with an adaptive stride algorithm, and aggregates the survivors into one coherent scene-level instructional caption, evaluated on 210 YouCookII cooking videos against GPT-4o and VideoLLaMA-3.

What This Paper Is About

Instructional videos are organized into scenes, each showing a distinct step of a task, and good scene-level captions make those steps easier to learn, search, and index. Existing automatic captioning methods tend to fail in one of two ways: they either describe every frame and produce dense, repetitive text, or they produce short captions that omit actions, objects, and temporal order. DynaStride targets this gap by generating concise, temporally coherent captions for each scene without requiring the system to detect scene boundaries itself, using YouCookII's existing scene annotations and human-written reference captions.

Key Contributions

1. A four-stage captioning pipeline (DynaStride) that combines adaptive frame sampling and windowing, multimodal chain-of-thought (MMCoT) subcaption generation, a dynamic stride window selection algorithm, and subcaption aggregation into a single scene-level instruction. The pipeline requires no manual scene segmentation.

2. A dynamic stride window selection algorithm that embeds each window's subcaption with all-MiniLM-L6-v2, compares it to the last retained window via cosine similarity, and either skips the window (increasing stride) or keeps it (resetting stride). This adaptively balances temporal context against redundancy.

3. An MMCoT subcaption formulation that prompts Qwen2.5-VL-7B-Instruct over horizontally concatenated frame windows to output exactly two parts, an action description and a list of objects, separated by a "|" delimiter, which the downstream aggregator consumes.

4. Empirical evaluation and ablations on YouCookII against GPT-4o and VideoLLaMA-3, plus ablations over three aggregator models (Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, Qwen3-4B-Instruct-2507) and three frame sampling settings (5, 20, and 40 frames), with code released on GitHub.

Main Findings

• DynaStride beats VideoLLaMA-3 on every reported metric. On the YouCookII validation subset, DynaStride scored BLEU-4 4.18 (0.07), METEOR 24.31 (0.10), CIDEr 0.56 (0.00), BERTScore Precision 0.25 (0.00), Recall 0.26 (0.00), F1 0.27 (0.00), and SBERT Similarity 0.61 (0.00), versus VideoLLaMA-3's 4.10 (0.32), 22.71 (0.63), 0.49 (0.02), 0.19 (0.01), 0.22 (0.00), 0.21 (0.01), and 0.58 (0.00).

• Against GPT-4o the picture is mixed. DynaStride scored higher on CIDEr (0.56 vs 0.48), BERTScore Precision (0.25 vs 0.19), BERTScore F1 (0.27 vs 0.23), and SBERT Similarity (0.61 vs 0.60), but lower on BLEU-4 (4.18 vs 4.73), METEOR (24.31 vs 28.47), and BERTScore Recall (0.26 vs 0.29). The paper frames the gains as evidence that its captions are semantically closer to the ground-truth references.

• Aggregator choice strongly affects N-gram quality. Qwen3-4B-Instruct produced the best overall scores and the lowest standard deviation across all metrics (for example METEOR 24.31 (0.10) and CIDEr 0.56 (0.00)), while Phi scored 2.78 (0.24) BLEU-4 / 22.8 (0.38) METEOR / 0.37 (0.02) CIDEr and Mistral scored 3.36 (0.08) / 19.49 (0.12) / 0.51 (0.01). Phi's standard deviations were reported as more than three times higher than Qwen3's, indicating less consistent generations.

• More frames per window improved DynaStride the most. At 5 frames DynaStride scored CIDEr 0.52 and SBERT 0.60; at 20 frames 0.58 and 0.61; at 40 frames it reached its best values across all metrics: BLEU-4 4.91 (0.03), METEOR 26.36 (0.18), CIDEr 0.61 (0.00), BERTScore F1 0.27 (0.00), and SBERT Similarity 0.63 (0.00).

• Temporal metrics were mixed. In the temporal analysis, VideoLLaMA-3 had the highest DTW alignment (3.15 (0.02)) and the highest contradiction score (0.06 (0.00)) versus DynaStride's 3.01 (0.00) and 0.05 (0.00), and GPT-4o's 2.86 (0.14) and 0.03 (0.00). DynaStride recorded NSP=True 1 (0.00), NSP=Shuffled 0 (0.00), and NSP=Delta 1 (0.00); GPT-4o and VideoLLaMA-3 recorded NSP=Delta 0 (0.00).

• Qwen3 also led the aggregator temporal comparison, with Align_DTW 3.01 (0.00), NSP=True 1 (0.00), NSP=Shuffled 0 (0.00), NSP=Delta 1 (0.00), versus Phi (3.14 (0.03), 1, 1, 0.00) and Mistral (3.11 (0.01), 0.99, 0.99, 0.00).

Methodology in Plain English

The pipeline runs in four steps on each scene of an instructional video.

First, frame sampling and windowing. A scene is treated as an ordered sequence of frames. Because consecutive frames look nearly identical, the pipeline keeps every M-th frame and then groups consecutive survivors into sliding windows of K frames, so each window covers a short stretch of the action rather than a single instant.

Second, MMCoT subcaption generation. The frames in a window are glued together side by side into one wide image, and Qwen2.5-VL-7B-Instruct is asked to reason internally about what happens across the sequence, then output only two things: the action being performed and the objects involved, separated by a "|" character and wrapped in <CONCLUSION> tags. Keeping the reasoning hidden and the output short reduces object hallucinations.

Third, dynamic stride window selection. Each subcaption is turned into an embedding with all-MiniLM-L6-v2. The system compares the new window's embedding to that of the most recently kept window using cosine similarity. If similarity is at or above a threshold, the window is skipped as redundant and the stride is multiplied by a scaling factor (capped at a maximum). If the window is novel, its caption is kept and the stride resets to its base value. The reported settings are window size K = 10, base stride 10, maximum stride 3 times the base, scaling factor 1.5, and similarity threshold 0.5.

Fourth, aggregation. The retained subcaptions are concatenated in chronological order and passed to Qwen3-4B-Instruct, which is prompted to emit a single concise imperative sentence wrapped in <ANSWER> tags. The result is the final scene-level instructional caption. All models are used off the shelf, without fine-tuning, for reproducibility.

Evaluation used 210 videos uniformly sampled from the YouCookII validation set (the official test captions were not released), run with three seeds, reporting mean and standard deviation. Experiments used 30× RTX A6000 GPUs and 800 GB of storage. Frames were resized to 384×384 pixels during preprocessing.

Why This Matters

Impact on research. The paper contributes a reproducible, modular alternative to end-to-end video captioning for instructional content, with released code and a clear ablation structure. It also adds evidence that the choice of aggregation model, and the number of frames placed into each window, can matter as much as the choice of vision-language backbone. The temporal-metric results (where VideoLLaMA-3 led on DTW alignment) suggest that improvements in semantic and N-gram quality do not automatically translate into better temporal alignment, an issue future captioning work will have to address.

Real-world applications:

• Accessibility: high-quality scene-level captions make instructional videos usable for visually impaired or differently abled learners, as the paper notes.
• Content indexing and search: concise step-level captions let educators and platforms index, summarize, and retrieve specific procedures inside long videos.
• Personalized and adaptive learning: step descriptions can feed recommendation or adaptive-learning systems that sequence content for individual learners.
• Scalable educational content production: automatic captioning reduces the manual cost of annotating large instructional video libraries.

Industry relevance. The pipeline is built entirely from freely available pretrained models (Qwen2.5-VL-7B-Instruct, Qwen3-4B-Instruct-2507, Mistral-7B-Instruct-v0.2, Phi-3-mini-4k-instruct, all-MiniLM-L6-v2) and adopts adaptive sampling explicitly to cut computational overhead, which matters for platforms processing large video catalogs. Educational technology, video hosting, e-learning, and technical-training vendors are the most direct beneficiaries.

Future Directions

1. Extend to raw, unsegmented video. DynaStride relies on YouCookII's scene boundary annotations; the authors propose robust scene boundary detection through temporal action detection models or weakly supervised segmentation to remove that dependency.

2. Make sampling saliency-aware. Adaptive frame sampling informed by visual saliency, action dynamics, or instructional importance could improve coverage while preserving efficiency, and could help recover the subtle or rapid actions the current design may skip.

3. Improve aggregation for procedural coherence. Better caption aggregation that explicitly models procedural coherence and temporal alignment could raise instructional clarity, since redundancy is not fully eliminated during aggregation today.

4. Human evaluation and domain expansion. The paper calls for human evaluations and testing across diverse instructional domains, alongside mechanisms for domain adaptation or active feedback, since standard N-gram and semantic metrics do not capture instructional relevance, temporal accuracy, or educational usefulness.

Target Audience

This paper is most useful to researchers and engineers working on video-language models, video captioning, and multimodal reasoning pipelines, especially those focused on educational or procedural content. It also suits practitioners in educational technology and video platforms who need practical, reproducible captioning systems built from open models, and students or newcomers with intermediate background in vision-language modeling who want a clearly staged example of combining chain-of-thought prompting with retrieval-style redundancy filtering. Readers looking for a fine-tuned end-to-end architecture, large-scale human evaluation, or results beyond the cooking domain will find those absent, and the authors acknowledge as much in their limitations section.

Authors’ abstract

Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.