Point-Supervised Facial Expression Spotting with Gaussian-Based Instance-Adaptive Intensity Modeling

AI summary

Overview

• Research area: Computer vision, specifically facial expression analysis — facial expression spotting (FES) in untrimmed videos, covering macro-expressions (MaEs) and micro-expressions (MEs), under a point-supervised (weakly supervised) setting.
• Technical level: Advanced. The paper assumes familiarity with temporal action localization, optical flow features, contrastive learning, pseudo-labeling, and Gaussian distribution modeling.
• Scope: A two-branch, point-supervised framework that replaces hard pseudo-labels with Gaussian-based soft pseudo-labels and decouples expression-intensity estimation from apex-based classification, evaluated on three facial expression datasets (arXiv:2511.16952v3, by Yicheng Deng, Hideaki Hayashi, and Hajime Nagahara of The University of Osaka).

What This Paper Is About

Facial expression spotting means finding where expressions start and end in long, unedited videos and labeling each one as either a macro-expression or a micro-expression. Existing methods need expensive frame-level annotations marking onset, apex, and offset frames. This paper asks whether the task can be learned from just a single timestamp per expression instance, and it addresses two problems that arise when existing point-supervised action localization methods are reused: micro-expressions get overshadowed because their intensity is lower than macro-expressions, and hard pseudo-labels cannot describe the smoothly varying intensity of an expression.

Key Contributions

1. Diagnosis of two limitations in applying point-supervised temporal action localization (P-TAL) to point-supervised facial expression spotting (P-FES): general two-branch frameworks fuse branch outputs during training and inference, causing micro-expressions to be overshadowed, and hard pseudo-labeling makes neutral frames and low-intensity expression frames ambiguous.
2. A redesigned decoupled framework in which the two branches are optimized separately during training and operate independently during inference, plus a new class-aware apex classification branch that distinguishes MaEs from MEs using only pseudo-apex frames.
3. A Gaussian-based instance-adaptive Intensity Modeling (GIM) module that detects a pseudo-apex frame around each point label, estimates the instance duration, and constructs an individual Gaussian distribution per instance to assign soft pseudo-labels as continuous intensity supervision for a regression-based class-agnostic expression intensity branch.
4. An Intensity-Aware Contrastive (IAC) loss that contrasts neutral frames with expression frames of various intensities, weighting the contrast by intensity difference rather than relying solely on class labels; the paper also proposes evaluating apex frame detection accuracy in the FES task.

Main Findings

• Micro-expression spotting improves over the authors' previous conference version: on SAMM-LV the ME F1 rises from 0.2033 (Deng et al. [14]) to 0.2417; on CAS(ME)² it rises from 0.0588 to 0.1370; on CAS(ME)³ it rises from 0.0708 to 0.0958.
• Overall F1 also improves over the previous conference version: SAMM-LV overall goes from 0.3587 to 0.3705; CAS(ME)² overall goes from 0.4000 to 0.4023; CAS(ME)³ overall goes from 0.2273 to 0.2335.
• Macro-expression F1 is slightly lower than the previous version: on SAMM-LV, MaE is 0.4176 versus 0.4189; on CAS(ME)², MaE is 0.4339 versus 0.4395; on CAS(ME)³, MaE is 0.2438 versus 0.2396.
• Fully-supervised methods remain stronger in many settings: SpotFormer reports SAMM-LV 0.4447 MaE / 0.4281 ME / 0.4401 overall, CAS(ME)² 0.5061 / 0.2817 / 0.4841, and CAS(ME)³ 0.2664 / 0.2037 / 0.2559, all higher than the proposed point-supervised framework.
• Point-supervised baselines reproduced for P-FES are weaker than the proposed method: LAC scores 0.3223 overall on SAMM-LV and 0.3598 on CAS(ME)²; HR-Pro scores 0.2895 and 0.3261; TSP-Net scores 0.2703 on SAMM-LV and 0.3358 on CAS(ME)², and on CAS(ME)³ it reaches only 0.0029 ME F1.
• The Gaussian prior is structural, not a rigid symmetry assumption: soft pseudo-labels are computed from feature distance to the pseudo-apex frame rather than temporal distance, so intensity trajectories can be asymmetric per instance. The paper states that supporting analysis appears in its Section IV-D2 (Table V) and Section IV-F (Fig. 5), but those tables and figures are not included in the truncated content provided.
• Detailed ablation results, hyperparameter values, and qualitative results are not reported in the available content, because the text is truncated partway through the dataset and evaluation-metric section.

Methodology in Plain English

The researchers start by splitting a video into overlapping snippets of 17 frames, padding the beginning and end with 8 repetitions of the first and last frames so the original first and last frames can serve as snippet centers. Within each snippet they compute MDMO optical flow, and SpotFormer embeds those optical flow features into a sequence of feature vectors.

Those features feed two separate branches. The class-agnostic expression intensity branch is trained as a regression model rather than a binary classifier. For each point label, the model looks in a small window around the label and picks the frame with the highest predicted intensity as the pseudo-apex. It estimates a rough duration by counting nearby frames whose intensity exceeds a threshold, expands that duration by a coefficient, and then builds a Gaussian for that instance: the mean is the feature of the pseudo-apex frame, and the variance comes from the average squared feature distance between frames in the proposal and that apex. The Gaussian value becomes a soft label between 0 and 1, so frames near the apex get high targets and low-intensity edges get small targets instead of all being labeled 1.

The class-aware apex classification branch marks only pseudo-apex frames (with soft labels of 1) and their neighbors as positives, and pseudo-neutral frames plus high-intensity non-positive frames as negatives, trained with Focal loss to handle class imbalance.

Training proceeds easy-to-hard over three stages: first hard labels on a narrow window around each point label, then a small Gaussian centered on the labeled frame, then the full GIM module. Supporting losses include an L1 norm on the intensity scores, a reward term to push reliable expression frames higher, a temporal smoothness term, and an Intensity-Aware Contrastive loss. The IAC loss mines pseudo-neutral frames using the top-k lowest intensity scores, restricts contrasts to reliable pseudo-labeled frames with intensity above 0.5, and weights each pair by intensity difference — pulling same-class frames together more when their intensities are close, and pushing different-class frames apart more when their intensities differ.

At inference, the intensity scores are turned into proposals with a multi-threshold strategy, each proposal scored with an outer-inner-contrastive (OIC) score. The expression class is decided solely from the apex scores: a proposal is MaE if it falls within MaE duration and its pseudo-apex MaE score exceeds 0.5, and ME under the analogous condition. Class-wise non-maximum suppression removes redundant proposals.

Why This Matters

• Impact on research: It shows that point supervision, requiring just one timestamp per expression, is a viable alternative to fully-supervised FES, and it identifies concrete failure modes (branch fusion suppressing micro-expressions, hard labels blurring intensity) that other weakly supervised facial expression and action localization work can act on.
• Real-world applications:
  • Lie detection and deception analysis, where micro-expressions are considered involuntary cues.
  • Psychological counseling and clinical interviewing, where subtle facial responses carry diagnostic signal.
  • Human–computer interaction and social robotics, where systems need to notice fleeting expressions in real time.
  • Virtual reality and emotion-aware interfaces that adapt to a user's spontaneous expressions.
• Industry relevance: The reduced annotation cost matters for any organization collecting large volumes of facial video, since only one timestamp per expression is needed rather than precise onset/apex/offset boundaries. The authors released code at https://github.com/KinopioIsAllIn/GIM. The framework is also described as backbone-agnostic, so it can be paired with appearance-based or multimodal feature extractors.

Future Directions

• Closing the gap to fully-supervised methods: SpotFormer's fully-supervised F1 scores remain higher on all three datasets, so improving point-supervised accuracy — especially macro-expression accuracy, which the redesign slightly reduced — is an open direction.
• Better micro-expression recall: ME F1 remains low in absolute terms (0.2417 on SAMM-LV, 0.1370 on CAS(ME)², 0.0958 on CAS(ME)³), leaving substantial room for improved micro-expression sensitivity.
• Apex frame detection as a new evaluation axis: the paper proposes evaluating apex detection accuracy as a step toward downstream expression recognition, but the reported results for that evaluation are not included in the available content.
• Extension to other backbones and modalities: the authors state the GIM module imposes no constraints on how features are extracted, leaving integration with appearance-based, multimodal, and other spatiotemporal models as an untested direction.

Target Audience

Researchers and graduate students working on facial expression analysis, micro-expression spotting, and weakly supervised temporal action localization, as well as practitioners who need to detect expressions in untrimmed video without paying for frame-level boundary annotations. Readers should be comfortable with deep learning pipelines, optical flow features, contrastive losses, and Gaussian distribution modeling, since the paper is written at an advanced technical level.

Authors’ abstract

Automatic facial expression spotting, which aims to identify facial expression instances in untrimmed videos, is crucial for facial expression analysis. Existing methods primarily focus on fully-supervised learning and rely on costly, time-consuming temporal boundary annotations. In this paper, we investigate point-supervised facial expression spotting (P-FES), where only a single timestamp annotation per instance is required for training. We propose a unique two-branch framework for P-FES. First, to mitigate the limitation of hard pseudo-labeling, which often confuses neutral and expression frames with various intensities, we propose a Gaussian-based instance-adaptive intensity modeling (GIM) module to model instance-level expression intensity distribution for soft pseudo-labeling. By detecting the pseudo-apex frame around each point label, estimating the duration, and constructing an instance-level Gaussian distribution, GIM assigns soft pseudo-labels to expression frames for more reliable intensity supervision. The GIM module is incorporated into our framework to optimize the class-agnostic expression intensity branch. Second, we design a class-aware apex classification branch that distinguishes macro- and micro-expressions solely based on their pseudo-apex frames. During inference, the two branches work independently: the class-agnostic expression intensity branch generates expression proposals, while the class-aware apex-classification branch is responsible for macro- and micro-expression classification. Furthermore, we introduce an intensity-aware contrastive loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames with various intensities. Extensive experiments on the SAMM-LV, CAS(ME)$^2$, and CAS(ME)$^3$ datasets demonstrate the effectiveness of our proposed framework.