HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition

AI summary

Overview

Research area: Affective computing / computer vision — facial affect analysis and multimodal ambivalence-hesitancy recognition, evaluated in the 11th Affective Behavior Analysis in-the-wild (ABAW) Challenge.

Technical level: Advanced (assumes familiarity with valence-arousal regression, action units, and multimodal fusion, though the paper's core ideas are describable in plain terms).

Scope: One sentence: the paper describes a two-challenge ABAW-11 submission that keeps all pretrained face, audio, and text extractors frozen and recovers accuracy through lightweight heads plus systematic prediction calibration and late multimodal fusion.

What This Paper Is About

Top ABAW entries typically win by fine-tuning very large vision transformers or adding self-supervised pretraining, which makes them heavy and hard to deploy. This paper asks how far you can get with the opposite strategy: freeze strong but lightweight pretrained extractors (MT-EmotiDDAMFN, MT-EmotiEffNet-B0, HuBERT-large, RoBERTa-GoEmotions), train only small task-specific heads, and then invest effort in post-processing and fusion. The goal is to show that calibration — temporal smoothing, per-class expression bias, per-AU threshold tuning, AffectNet blending, and weighted backbone blending — can match substantially heavier end-to-end pipelines while remaining efficient and privacy-friendly.

Key Contributions

1. Frozen lightweight backbones stay competitive. The authors demonstrate that frozen affect-recognition extractors plus lightweight task-specific heads remain highly competitive for both multi-task facial affect analysis and multimodal ambivalence recognition.
2. Calibration beats backbone scale. They show that validation-time calibration (temporal smoothing, expression calibration, AU threshold optimization, backbone fusion) produces larger gains than swapping in substantially heavier architectures.
3. A unified lightweight framework across two different ABAW challenges, achieving state-of-the-art validation and public-test performance without any end-to-end fine-tuning.
4. Released code. The complete PyTorch implementation is released to support reproducible affective computing research (https://github.com/bakinalexey/abaw-11-mtl-bah-recognition).

Main Findings

• MTL validation score of 1.56. On the s-Aff-Wild2 validation split, the ensemble reaches P_MTL = 1.56 (P_VA = 0.56, P_EXPR = 0.46, P_AU = 0.54), improving on the ABAW-7 validation ensemble (1.49, same split protocol) and far above the official ABAW-11 ConvNeXt baseline (0.45).

• Ablation shows where the gains come from. Frame-level heads give 1.31 (MT-EmotiDDAMFN) and 1.27 (MT-EmotiEffNet-B0); post-processing raises these to 1.53 and 1.49; weighted backbone blending reaches 1.56.

• Each post-processing step has a distinct task effect. For MT-EmotiDDAMFN, the incremental path is 1.31 → 1.39 (VA smoothing) → 1.44 (EXPR smoothing) → 1.49 (EXPR bias) → 1.50 (AffectNet blending) → 1.53 (AU threshold tuning). Smoothing helps VA and EXPR by up to +0.08 and +0.11 for DDAMFN, but does not help AU detection, which benefits almost exclusively from threshold tuning.

• Best fusion weights. The best MTL configuration is w_VA = 1.0, w_EXPR = 0.4, w_AU = 0.5, meaning DDAMFN dominates VA estimation while EXPR and AU predictions are mixed almost equally. The final ensemble uses a blend threshold τ_blend = 0.88 (0.85 for MT-EmotiDDAMFN, 0.90 for MT-EmotiEffNet-B0 in single-backbone ablations).

• Frame-level A/H performance rose from ABAW-8 to ABAW-11. Frame-level Weighted F1 on the validation set rises from 0.74 in ABAW-8 to 0.79. Text is the strongest single modality (Weighted F1 0.77, Macro F1 0.59) versus face (0.71 / 0.52) and audio (0.67 / 0.53), and late fusion (0.79 Weighted F1) beats early fusion (0.76).

• Video-level A/H Macro F1 of 0.731 on the public test set, against an official zero-shot Video-LLaVA baseline of 0.283. For comparison, prior reported BAH scores in Table 4 are: ABAW-10 baseline 0.343, Lenovo PCIE 0.675, LEYA 0.714, Fennec 0.715, and VisPBF 0.727.

• The best public configuration is max aggregation with a hard global-text gate (late fusion weights (0.20, 0.45, 0.35), smoothing δ = 40, gate threshold τ_global = 0.41, τ = 0.35). A transcript-only global classifier reaches a comparable Macro F1 (0.734) with higher AP (0.87 vs. 0.82), confirming that spoken language is a very strong cue.

• Video-level validation ablations. Majority class 0.28 Macro F1 / 0.60 AP, RandomForest on concatenated features 0.67 / 0.82, early fusion MLP 0.68 / 0.85, late fusion with mean aggregation 0.72 / 0.85 — all below the late-fusion configurations on the public split.

• No fine-tuning of heavy backbones in either task, which the authors frame as evidence that systematic prediction calibration and lightweight multimodal fusion can rival substantially heavier end-to-end approaches.

Methodology in Plain English

Multi-task learning (facial frames). Each frame of s-Aff-Wild2 must be labelled three ways at once: valence and arousal values in [-1, 1], one of eight basic expressions (Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise, Other), and twelve binary action units. The team uses two frozen extractors: MT-EmotiDDAMFN, which outputs a 522-dimensional descriptor (512-dim embedding + 10 AffectNet scores), and MT-EmotiEffNet-B0, which outputs a 1290-dimensional descriptor (1280-dim embedding + 10 scores). Both were pretrained on VGGFace2 and fine-tuned on AffectNet. Only the heads are trained: a linear softmax EXPR head, a VA head that consumes just the ten AffectNet scores with tanh activations, and an AU head with one 128-unit ReLU hidden layer and 12 sigmoid outputs. Then five post-processing steps run inside each video: Gaussian temporal smoothing with separate parameters for VA and EXPR (AU predictions are not smoothed), a per-class expression logit bias found by coordinate search, per-AU threshold tuning, weighted blending of the two backbones, and AffectNet blending that overrides the fused expression label when the AffectNet confidence exceeds a threshold.

Ambivalence/hesitancy (video-level). The BAH dataset release used here contains 1,427 videos (10.6 hours, 300 participants), with splits of 778 training, 124 validation, and 525 public test videos. Three frozen encoders produce features: MT-EmotiEffNet-B0 for 1280-dimensional face embeddings, HuBERT-large for 1024-dimensional audio hidden states, and RoBERTa-base trained on GoEmotions for 768-dimensional text embeddings. Audio and text are linearly interpolated onto the visual frame timeline. One small MLP per modality (64 ReLU units, one logistic output, positive-class weighting N_neg/N_pos) predicts a frame-level probability, and those probabilities are combined by late fusion with weights summing to one. Frame scores are optionally smoothed, then aggregated per video by mean or max. A separate logistic-regression global text classifier, trained on pooled transcript embeddings from the 902 training plus validation videos, acts as a hard gate that forces a negative prediction when the transcript alone does not suggest A/H. Fusion weights, smoothing width, aggregation rule, and thresholds were chosen by grid search on the 525 public test videos.

Data details worth noting. Of 142,333 s-Aff-Wild2 training frames, only 103,917 have valence/arousal values, 90,645 have expression labels, and 103,316 have AU annotations. The validation set has 26,876 faces with complete VA and AU labels but only 15,440 expression labels. Final MTL post-processing was applied without further adaptation to 51,159 official test frames. The A/H validation ablation covers 79,538 labelled frames, and the primary A/H submission was applied to the private test set of 152 videos.

Why This Matters

Impact on research. The paper argues that a large share of reported MTL error on in-the-wild faces comes from temporal inconsistency and calibration rather than from insufficient backbone capacity. If that holds, challenge leaderboards may be rewarding model scale where cheaper, better-calibrated pipelines would do. It also reinforces a recurring finding in hesitancy recognition: transcript content is the dominant cue, and audiovisual signals mainly resolve borderline cases. The authors intend to update their comparison tables with final test-set results after the competition concludes.

Real-world applications (as the paper frames them):

• Mobile affect recognition, where deploying frozen lightweight extractors on-device avoids transmitting raw audiovisual data.
• Digital health and mental health monitoring, described as a privacy-sensitive setting where only compact frame-level scores or pooled embeddings need to leave the device.
• Human-computer interaction and socially aware AI that must run without server-side compute.
• A two-stage deployment pattern for hesitancy detection: a fast video-level text screen, with multimodal fusion reserved for ambiguous cases.

Industry relevance. The "compute-on-device, fuse-locally" design directly targets deployment cost and data-privacy constraints. The authors note that this framework is particularly suitable for privacy-sensitive applications where transmitting raw audiovisual data is undesirable — relevant to consumer devices, telehealth platforms, and any product subject to data-residency or consent restrictions.

Future Directions

• Closing the gap between transcript-only and multimodal decisions. A transcript-only global classifier already reaches 0.734 Macro F1 with higher AP (0.87 vs. 0.82) than the full pipeline, so the open question is how audiovisual fusion can add value beyond borderline cases rather than duplicating what text already provides.
• Verifying against final test results. The authors state they will update their comparison table once the competition concludes; the private test set used for the primary submission contains 152 videos.
• Transferring the calibration recipe to other tasks and datasets. The paper's central claim — that calibration beats backbone scale — is demonstrated on s-Aff-Wild2 and BAH and could be tested elsewhere.
• Reconciling task-specific calibration requirements. Because VA needs temporal smoothing, expressions need category-level bias, and AUs need thresholds, a principled way to choose post-processing per task remains unresolved.

Target Audience

Researchers and engineers working on affective computing, facial expression and action unit analysis, and multimodal video understanding, especially those balancing accuracy against deployment constraints on mobile or privacy-sensitive platforms. It will also interest practitioners building hesitancy or engagement detection from interview-style video, competition participants tracking ABAW benchmark progress, and anyone looking for an argument and a recipe for freezing large encoders instead of fine-tuning them end to end. Readers who want the mathematical detail of the fusion and gating equations will find them in Section 3; readers focused on results can start with Tables 1 through 6.

Authors’ abstract

This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition. For multi-task learning with simultaneous prediction of valence, arousal, facial expressions, and action units on s-Aff-Wild2 dataset, we use frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0, with separate heads and systematic post-processing: temporal Gaussian smoothing, per-class expression bias, AffectNet blending, per-AU threshold tuning, and weighted backbone fusion. On the official validation set, our ensemble significantly exceeds the performance of the ConvNeXt baseline. For ambivalence/hesitancy video recognition on the expanded BAH dataset, we extend the audiovisual pipeline to video-level Macro F1 by late fusion of face, HuBERT audio, and RoBERTa text classifiers, temporal aggregation, and a global-text gate. Frame-level Weighted F1 on validation set rises from 0.74 in ABAW-8 to 0.79, while the best public-test video-level Macro F1 reaches 0.73. In both tasks, competitive performance is achieved without fine-tuning heavy backbones. These results indicate that systematic prediction calibration and lightweight multimodal fusion can rival substantially heavier end-to-end approaches while offering improved efficiency and deployment flexibility.