FEA-SLT: A Gloss-Free End-to-End Framework for Facial-Expression-Aware Sign Language Translation

AI summary

Overview

Research area: Computer vision and cross-modal machine translation, specifically gloss-free Sign Language Translation (SLT) with explicit facial-expression modeling.

Technical level: Advanced. The paper builds on multi-stream Vision Transformer encoders, cross-attention-based feature modulation (FiLM-style), bidirectional contrastive alignment, and parameter-efficient fine-tuning (LoRA) of large language models.

Scope: The paper introduces FEA-SLT, a gloss-free end-to-end framework that adds a domain-transferred facial-expression stream and a bidirectional facial-manual fusion module to an existing spatial-motion SLT backbone, and evaluates it on PHOENIX14T and CSL-Daily.

What This Paper Is About

Sign language conveys meaning through two channels: manual signals (hand shapes and movements) and non-manual signals such as facial expressions, mouthing, and head positioning. Existing gloss-free SLT systems are heavily manual-centric and treat facial expressions as auxiliary visual context, which causes semantic errors when two different concepts share nearly identical hand movements — the paper's example is Chinese Sign Language "REVENGE" versus "RECIPROCATE." The goal is to explicitly extract expression-sensitive facial dynamics and fuse them with manual features so that translation accuracy improves, particularly for facial-sensitive utterances.

Key Contributions

1. A decoupled multi-path architecture that processes manual and facial channels through dedicated encoders, using domain-transferred facial representations (a ViT fine-tuned on FER2013) to capture expression-sensitive dynamics for grammatical disambiguation.
2. A Facial-Expression-Aware Fusion (FEAF) module whose Facial-Expression-Aware Modulation (FEAM) component explicitly models prosodic coordination between manual and facial channels via bidirectional modulation, using a Modulator that predicts scaling and gating parameters.
3. State-of-the-art BLEU results among gloss-free methods on PHOENIX14T and CSL-Daily, plus the best BLEURT scores on both benchmarks, with the paper reporting 60.8 on PHOENIX14T and 57.5 on CSL-Daily.
4. Ablation studies and targeted subset analyses showing gains from domain-transferred facial features and bidirectional fusion, particularly for utterances involving affective or interrogative semantics, with a diagnostic comparison on a fixed 84-instance facial-sensitive subset.

Main Findings

• PHOENIX14T BLEU results: FEA-SLT reaches BLEU-1 52.59, BLEU-2 39.72, BLEU-3 31.67, BLEU-4 26.38, and ROUGE-L 48.63. It surpasses SpaMo by +2.79 in BLEU-1 and outperforms MMSLT by +1.60, +0.88, and +0.65 in BLEU-2, BLEU-3, and BLEU-4 respectively. On ROUGE-L it trails MLSLT and Sign2GPT while improving over MMSLT and SpaMo.
• CSL-Daily BLEU results: FEA-SLT reaches BLEU-1 50.83, BLEU-2 37.82, BLEU-3 29.03, BLEU-4 22.94, and ROUGE-L 50.40. It outperforms MMSLT by +1.83 in BLEU-4 and +1.48 in ROUGE-L, and exceeds the weakly supervised VAP by +2.09 in BLEU-4.
• BLEURT semantic adequacy: FEA-SLT attains 60.8 on PHOENIX14T and 57.5 on CSL-Daily, the best on both benchmarks, surpassing the strongest prior baseline on each — the reproduced SpaMo and SONAR-SLT — by +1.9 and +1.4 points respectively.
• Facial pathway ablation (PHOENIX14T): The controlled MS-only baseline achieves 24.76 BLEU-4 and 46.75 ROUGE-L. Adding the facial-expression stream improves BLEU-4 by 0.54 and ROUGE-L by 0.58. Adding bidirectional FEAM further improves BLEU-4 by 1.08 and ROUGE-L by 1.30. The full facial pathway contributes +1.62 BLEU-4 and +1.88 ROUGE-L over the controlled backbone, reported as statistically significant at p < 0.01.
• Bidirectionality matters: Bidirectional mutual modulation outperforms the stronger single-direction variant by 0.63 BLEU-4 and 0.72 ROUGE-L.
• Facial pathway complements alignment: Multimodal alignment alone improves BLEU-4 by 1.68. Without MA, the facial pathway yields a total gain of 1.67 BLEU-4; with MA, the facial pathway yields 1.62 BLEU-4. The paper describes these as comparable gains, indicating complementary benefits beyond global visual-text alignment.
• Encoder choice matters: The FER-adapted ViT-B/16 reaches 26.38 BLEU-4 and 48.63 ROUGE-L, surpassing its unadapted counterpart by +0.80 BLEU-4 and +0.55 ROUGE-L. DINOv2-ViT-B/16 has comparable parameters (86M) but performs worse (25.22 BLEU-4, 47.83 ROUGE-L); ResNet-50 (26M) reaches 25.68 BLEU-4 and 48.40 ROUGE-L.
• Interrogative punctuation: On a punctuation-sequence evaluation of interrogative constructions, FEA-SLT outperforms the baseline across all reported metrics, with an absolute F1 improvement of 3.69.
• Qualitative cases: On CSL-Daily, SpaMo translates a fear-related utterance as "Daytime illuminates me," FEA-SLT without the facial stream produces "Night is very quiet," and full FEA-SLT recovers "It is dark, I am afraid." In a second case, SpaMo inserts an unrelated "famous post office," while FEA-SLT recovers the interrogative structure but adds an erroneous "Premier's."

Methodology in Plain English

The researchers start from a strong spatial-motion SLT backbone (SpaMo) so they can isolate the effect of adding facial information.

1. Split the video into three streams. Spatial features capture static configurations like hand shapes and body postures, using a multi-scale strategy that processes each frame at a global 224² resolution and a local 448² resolution, with the high-resolution input split into four patches. A ViT extracts the [CLS] token from each view (one global, four local), and the global token is concatenated with the average of the four local tokens. Motion features are extracted from overlapping clips by a pretrained video encoder using a sliding window of width w and stride s_m. Facial features come from a separate path.
2. Transfer facial knowledge from emotion recognition. Rather than training a facial encoder on sign language facial annotations (which do not exist at scale), they use a ViT pretrained on the FER2013 facial expression dataset. The argument is that expression recognition forces sensitivity to the same muscle-movement patterns that serve grammatical functions in sign language. Frames are uniformly downsampled at interval s_e, faces are detected with RetinaFace to extract regions of interest, and feature-level interpolation bridges detection gaps.
3. Fuse the streams bidirectionally. All features are mapped to a unified dimension d. A Modulator uses cross-attention to align the conditioning stream to the target stream's temporal resolution, then two parallel heads predict channel-wise affine shifts and a frame-level gate, applying a FiLM-style transformation. FEAM applies this in both directions: facial context modulates the spatial and motion streams, while spatial and motion context refine the facial stream. The two reverse-modulated facial features are combined with a learnable scalar weight that is passed through a sigmoid. Each modality gets a learnable class embedding before temporal concatenation.
4. Smooth and adapt to the language model. A 1D Temporal Convolutional Network followed by a GELU-activated MLP projects the fused sequence into the LLM's embedding space.
5. Train in one stage. Two objectives are jointly optimized: a bidirectional contrastive alignment loss between pooled multimodal features and pooled target text embeddings, and a cross-entropy generation loss with label smoothing using a task-specific prompt. All pretrained encoder weights stay frozen; only the lightweight head layers, the FEAF module, and LoRA adapters for the LLM are trained.

Why This Matters

Impact on research. The work argues that progress in gloss-free SLT has been manual-centric, and shows that a facial stream adapted from facial expression recognition adds measurable gains under a strong contemporary backbone. It provides a controlled ablation isolating the facial pathway from global visual-text alignment, and it raises the question of how to distinguish affective facial appearance from grammatical facial markers — which the paper itself flags as unresolved.

Real-world applications:

• Assistive translation tools that render sign language video into spoken-language text for Deaf-to-hearing communication.
• Public-service and customer-service interaction, where a signing user needs real-time text output.
• Educational tools for sign language learners, where facial grammar is often the hardest component to master.
• Broadcast and captioning contexts, since PHOENIX14T consists of German Sign Language weather forecasts.

Industry relevance. The framework is built on frozen off-the-shelf encoders plus LoRA fine-tuning, which keeps trainable parameters low and fits resource-constrained deployment patterns. The transfer of a facial expression recognition model into a translation task illustrates a broader pattern: reusing a pretrained domain-specific model rather than relying on a general-purpose vision backbone.

Future Directions

• Broaden beyond facial dynamics. The current framework covers facial dynamics only; mouthing (including proper-noun mouth patterns), head movements, and body posture remain outside its scope.
• Disentangle affective from grammatical facial signals. FER-transferred representations do not separate the two. The authors propose component-level masking or occlusion to measure the contribution of specific facial regions.
• Strengthen statistical and generalization analysis. All experiments use fixed seed 0, and the bootstrap analysis measures significance over fixed-model outputs rather than training variance. Multi-seed runs and signer-stratified evaluation are called for.
• Extend to more sign languages and involve the community. Evaluation covers only DGS and CSL; the authors name American Sign Language (ASL) and British Sign Language (BSL) as future targets and recommend involving Deaf and Hard of Hearing community members. Robustness to self-occlusion, illumination variation, and extreme head poses also remains an open problem, with failure statistics given in the appendix.

Target Audience

Researchers working on sign language translation, gloss-free and end-to-end multimodal translation, and non-manual signal modeling. Also relevant to practitioners building accessibility and assistive communication systems, and to engineers interested in transferring pretrained facial expression recognition models into downstream multimodal pipelines. Readers need familiarity with Vision Transformers, cross-attention, contrastive learning, and LLM fine-tuning to follow the methodology sections in detail.

Authors’ abstract

Sign Language Translation (SLT) is a challenging cross-modal task requiring joint modeling of manual articulations and non-manual signals. Existing gloss-free SLT methods effectively capture gestural dynamics but often underutilize facial expressions, which play crucial grammatical and disambiguating roles. This limitation can cause semantic degradation when distinct concepts share similar manual configurations. To address this issue, we propose FEA-SLT (**F**acial-**E**xpression-**A**ware **S**ign **L**anguage **T**ranslation), a gloss-free end-to-end framework that uses facial dynamics to provide complementary cues to manual signals. FEA-SLT employs a domain-transferred facial encoder to extract expression-sensitive representations and integrates them with manual features through a linguistically motivated *Facial-Expression-Aware Fusion* (FEAF) module. FEAF captures reciprocal dependencies between manual and facial channels via bidirectional modulation, enhancing syntactic fidelity. Experiments on PHOENIX14T and CSL-Daily show that FEA-SLT achieves state-of-the-art BLEU performance among gloss-free methods, while targeted analyses support improved translation of facial-sensitive utterances. Code is available at https://github.com/TuGuobin/FEA-SLT.