Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos

AI summary

Overview

Research area: Computer Vision, specifically multi-label fine-grained video classification with multimodal (visual + text) learning, applied to educational content detection for young children.

Technical level: Advanced — the paper builds on supervised contrastive learning, class prototype estimation, and multimodal transformer fusion, though the motivation and results are accessible to a general audience.

Scope: The paper introduces a prototype-based contrastive learning framework plus a new expert-annotated dataset (APPROVE) for detecting and categorizing fine-grained educational content in online videos.

What This Paper Is About

Children watch large amounts of online video, and only some of it is genuinely educational. Automatically identifying which videos teach literacy and math skills (and exactly what they teach — for example "letter names" versus "letter sounds") is hard because videos contain multiple overlapping educational topics whose visuals look alike, and because the distinction often lives in the audio, not the frame.

The authors pose this as a fine-grained multi-label video classification problem: a single video can carry several labels, and those labels can be visually near-identical. They propose learning a representative "class prototype" for each educational code and using contrastive learning to pull each video toward the prototypes of its true labels and push it away from the rest, while fusing visual frames with automatically transcribed speech.

Key Contributions

1. APPROVE dataset — a fine-grained, multi-label dataset of educational YouTube videos, containing 193 hours of expert-annotated content across 19 classes (7 literacy codes, 11 math codes, and a background class), with roughly 3 labels per video on average. It is annotated by trained education researchers following a standard validation protocol, and includes visually similar background videos that contain no educational content.

2. Class-prototype-based multi-label contrastive learning — a loss that treats class prototypes as anchors rather than pairing samples against each other, which sidesteps the problem that supervised contrastive learning cannot cleanly define "positive" and "negative" pairs when labels partially overlap.

3. A multimodal transformer network (MTN) — image encoder, text encoder, and a cross-modal fusion encoder that combine video frames with ASR-transcribed speech, trained jointly with the prototype loss in an end-to-end two-stage scheme.

4. Broad empirical validation — the approach outperforms strong baselines (BCE, Focal loss, Asymmetric loss, SupCon) on APPROVE, a YouTube-8M subset, and COIN.

Main Findings

• APPROVE results (video + text): The proposed method reaches 88.4 AUPR, 90.7 LRAP, and 85.5 R@80, versus the strongest baselines at 86.1 AUPR (Focal), 89.2 LRAP (Asymmetric loss), and 82.4 R@80 (Asymmetric loss). The paper reports gains of 3.1% in R@80 and 2.3% in AUPR over the strongest baselines.

• Math is easier than literacy: On the Math subset the method scores 88.4 AUPR / 93.2 LRAP / 83.2 R@80; on the Literacy subset it scores 73.6 / 84.7 / 54.7. The paper attributes this to high inter-class similarity in literacy, naming follow_words, letters_in_words, and sounds_in_words as the three hardest classes.

• Text alone beats video alone: On APPROVE, using only text gives 82.5 AUPR / 87.4 LRAP / 75.4 R@80, while using only video gives just 46.7 / 57.9 / 19.6. Combining both gives the best result, confirming that audio cues carry much of the educational signal.

• YouTube-8M subset (YT-46K): The method reaches 70.9 AUPR, 74.9 LRAP, and 49.1 R@80, beating Focal loss (69.7 / 72.7 / 44.6). The paper notes the gain from text is smaller here because YT-8M was collected primarily for visual classification.

• COIN: With Top-1 accuracy as the metric, the method scores 57.5, compared with 56.1 for Focal loss, 54.9 for BCE, 53.7 for cross-entropy, and 54.7 for SupCon.

• Prototype handling matters: In the prototype ablation (APPROVE R@80 / COIN Top-1), Random initialization gives 84.1 / 56.6, Orthogonal gives 84.8 / 57.0, Learned gives 85.5 / 57.5, and Hierarchical gives 86.0 / 57.8.

• Fusion depth saturates: 1 layer gives 84.9 / 57.1, 2 layers gives 85.5 / 57.5, and 4 layers gives 85.3 / 57.6 on APPROVE / COIN.

• Encoder choices differ by dataset: ViT-B/32 works best on APPROVE (85.5) while ViT-B/16 works best on COIN (57.8); ResNet50 trails on both (84.8 / 55.2). For text, t5-small (87.3 / 57.9) outperforms DistilBERT (85.5 / 57.5).

• Robustness: With 10% of video frames missing, APPROVE drops from 85.5 to 80.1; with 10% of text missing it drops to 75.8; with 30% of text missing it drops to 68.9. Run-to-run variance is reported as 85.5 ± 0.5 (APPROVE) and 57.5 ± 0.8 (COIN), comparable to the baselines.

• CLIP initialization helps: Initializing backbones with CLIP raises APPROVE to 86.7 from 85.5 and COIN to 63.5 from 57.5; the paper notes the gain is larger on COIN because CLIP models may not have been exposed to educational videos.

Methodology in Plain English

The authors start from supervised contrastive learning, which normally works by pulling same-class samples together and pushing different-class samples apart. That logic breaks when a video has several labels, because two videos can share some labels but not others — they are neither fully "positive" nor fully "negative" pairs.

Their fix is to learn one prototype (a representative embedding) per class. For any video, the prototypes matching its true labels are positives and all other prototypes are negatives. Training minimizes the distance between the video's representation and its true-label prototypes and maximizes distance to the rest, with negative sampling to handle class imbalance. Prototypes are estimated by solving a least-squares-style relation between the matrix of video embeddings and the label matrix — effectively an average that also accounts for label co-occurrence — and are updated over training with an exponential moving average to prevent collapse.

For the video representation itself, the model uses three encoders: a visual encoder over video frames, a text encoder over ASR transcripts, and a fusion encoder applying multi-head self-attention within and across modalities. Speech is first isolated from the audio track (removing music and background) before transcription. Training proceeds in two stages — first aligning each modality separately with fixed prototypes, then training the whole multimodal network end-to-end while continuing to refine the prototypes.

At inference, the model skips the usual step of discarding the projection head and training a linear classifier. Instead it predicts directly from the cosine similarity between the test video's representation and each learned prototype, normalized so that the similarity range maps to a confidence range.

Why This Matters

Impact on research: The paper highlights a gap in standard video benchmarks — datasets like HMDB, UCF, and Kinetics are single-label and not fine-grained, while multi-label sets like YouTube-8M are machine-labeled rather than expert-labeled. APPROVE (193 hours, expert-annotated, multimodal, fine-grained, multi-label) fills that gap and provides a testbed for the harder problem where classes are visually near-identical and the distinguishing signal may be auditory.

Real-world applications:

• Parental and educator content filtering — automatically surfacing videos that actually teach literacy or math rather than merely looking educational.
• Curriculum alignment tools — mapping a library of videos to specific Common Core codes such as "letter sounds" or "comparing groups."
• Recommendation systems for early childhood — prioritizing high-quality educational content in feeds aimed at young children.
• Media production analytics — helping educational content developers understand what topics their catalogs actually cover.

Industry relevance: Streaming platforms, children's media companies, and educational technology providers all need scalable content understanding. Because the model relies on automatically transcribed audio and frames rather than manual metadata, it can be applied to large existing video libraries. The efficiency of the approach — it reuses pretrained encoders and notes that text alone already gets 75.4 R@80 — suggests a lighter-weight deployment path is viable.

Future Directions

• Extending beyond literacy and math: The paper covers two content classes chosen from the Common Core Standards; other kindergarten subject areas are not addressed.
• Handling modalities that are actually missing: The robustness study simulates missing frames and text at rates up to 30%, but real videos may lack usable audio or speech entirely; the approach's behavior in that regime is not reported.
• Closing the literacy gap: Literacy scores (73.6 AUPR / 54.7 R@80) lag well behind math (88.4 / 83.2), and the paper identifies the confusable code pairs but does not propose a targeted remedy.
• Improving prototype construction: Hierarchical prototypes scored highest on both datasets tested (86.0 on APPROVE, 57.8 on COIN) but were not adopted as the default, leaving room to explore richer prototype structures.

Target Audience

Researchers in computer vision and multimodal machine learning working on multi-label or fine-grained video understanding will find the contrastive learning formulation and the prototype estimation procedure most relevant. Education researchers, child-development specialists, and policy analysts interested in measuring the educational quality of online video will find the dataset and its annotation protocol valuable. Applied engineers building content moderation, recommendation, or catalog-tagging systems for video platforms — particularly in the children's media space — will benefit from the practical baseline comparisons and the robustness analysis. Some familiarity with contrastive learning and transformer architectures is helpful for the methodology section, though the problem framing and results are readable without it.

Authors’ abstract

The recent growth in the consumption of online media by children during early childhood necessitates data-driven tools enabling educators to filter out appropriate educational content for young learners. This paper presents an approach for detecting educational content in online videos. We focus on two widely used educational content classes: literacy and math. For each class, we choose prominent codes (sub-classes) based on the Common Core Standards. For example, literacy codes include `letter names', `letter sounds', and math codes include `counting', `sorting'. We pose this as a fine-grained multilabel classification problem as videos can contain multiple types of educational content and the content classes can get visually similar (e.g., `letter names' vs `letter sounds'). We propose a novel class prototypes based supervised contrastive learning approach that can handle fine-grained samples associated with multiple labels. We learn a class prototype for each class and a loss function is employed to minimize the distances between a class prototype and the samples from the class. Similarly, distances between a class prototype and the samples from other classes are maximized. As the alignment between visual and audio cues are crucial for effective comprehension, we consider a multimodal transformer network to capture the interaction between visual and audio cues in videos while learning the embedding for videos. For evaluation, we present a dataset, APPROVE, employing educational videos from YouTube labeled with fine-grained education classes by education researchers. APPROVE consists of 193 hours of expert-annotated videos with 19 classes. The proposed approach outperforms strong baselines on APPROVE and other benchmarks such as Youtube-8M, and COIN. The dataset is available at https://github.com/rohit-gupta/MMContrast/tree/main/APPROVE