Saudi Sign Language Translation Using T5

AI summary

Overview

• Research area: Sign language translation (SLT), multimodal computer vision, and natural language processing — specifically applying the T5 text-to-text transformer architecture to Saudi Sign Language (SSL).
• Technical level: Intermediate. Readers benefit from familiarity with transformer encoder–decoder models, pose estimation, and translation metrics (BLEU, ROUGE-L, BLEURT), though the paper is written clearly enough for graduate students new to the subfield.
• Scope: The paper introduces a processing pipeline and a new SSL dataset with three evaluation protocols, then shows that pre-training on American Sign Language (YouTubeASL) before fine-tuning on SSL roughly triples BLEU-4 scores over training from scratch.

What This Paper Is About

Saudi Sign Language has very little recorded training data compared with larger sign languages like American Sign Language, which limits the quality of automatic translation systems. The authors ask whether knowledge learned from a large, unrelated sign language (ASL) can be transferred to a small one (SSL), and they build a pose-based T5 translation system plus a purpose-designed SSL dataset with three test splits to answer this question quantitatively.

Key Contributions

1. A tailored preprocessing pipeline for sign language video. The pipeline centers the signer, crops to a linguistically motivated "signing space," extracts 104 pose keypoints (body, face, hands) with MediaPipe, normalizes them locally and globally, and halves the frame rate to reduce sequence length.
2. A new Saudi Sign Language dataset with three evaluation protocols. The corpus covers 2,000 unique sentences across everyday communication and specialized domains (banking, law, education, healthcare, emergency services, transportation), recorded by 18 signers, and is split so that signer identity and sentence novelty are varied independently.
3. Demonstration of cross-lingual transfer between sign languages. Pre-training on YouTubeASL and fine-tuning on SSL produces roughly a 3x improvement in BLEU-4 on the hardest test sets compared with training from scratch.
4. A systematic comparison of T5 variants and label languages. Three architectures (T5-base, T5v1.1-base, mT5-base) are evaluated, and mT5 is fine-tuned on both the original Arabic transcripts and machine-translated English transcripts, revealing a consistent advantage for English labels.

Main Findings

• Pre-training on ASL delivers large gains. For the T5-base model on Test 1 (unseen signers, unseen sentences), BLEU-4 rises from 2.73 without pre-training to 7.48 with YouTubeASL pre-training; on Test 2 it rises from 2.01 to 5.72. Gains appear consistently across BLEU-1 through BLEU-4, BLEURT, and ROUGE-L.
• Seen sentences are far easier than unseen sentences. Test 3, where sentences were seen in training but signers were not, yields BLEU-4 scores above 90 with pre-training, versus single-digit or low-double-digit scores on the two tests involving unseen sentences. This confirms that lexical memorization is easy but generalizing to new sentence content is the real bottleneck.
• Signer appearance matters less than sentence novelty. Performance on Test 2 (seen signers, unseen sentences) is close to Test 1 (unseen signers, unseen sentences), suggesting the pose-based representation largely removes dependence on signer identity for these models.
• English transcripts beat Arabic transcripts for mT5. mT5 fine-tuned on Arabic text performs substantially worse than mT5 fine-tuned on translated English text (for example, Test 1 BLEU-4 of 0.81 versus 5.66 with pre-training). The authors attribute this to weaker Arabic representation in mT5's mC4 pre-training corpus.
• Architecture ranking shifts with pre-training. Without pre-training, T5v1.1-base is generally strongest; after YouTubeASL pre-training, plain T5-base leads on most metrics.
• Pose-only input is competitive; deep visual features were not. Preliminary experiments with DINO features did not yield satisfactory results, which the authors attribute to the small size of the SSL dataset.
• Domain-specific challenges persist. Female signers' faces are deliberately covered (masks or veils), removing lip and facial cues that carry grammatical information in sign language, and the corpus skews male (13 of 18 signers).

Methodology in Plain English

The researchers convert sign language video into sequences of body keypoints rather than raw pixels, which removes background clutter and signer appearance and reduces the data the model must process.

The pipeline works in stages. A lightweight YOLO detector locates people; videos with multiple people are discarded to avoid confusing the signer with bystanders. A "signing space" box is defined around the shoulders at four times the shoulder width, expanded when keypoints fall outside it, and stabilized across the whole video by taking the median box coordinates. MediaPipe then extracts precise body, hand, and face keypoints. Leg keypoints are dropped, the 478-point face mesh is reduced to 37 informative points, and both hands keep all 21 points each — 104 keypoints total. Hand and face points are normalized in their own local boxes, while body points are normalized relative to the signing space. Frames are cropped, padded to square, resized, and half of them discarded to shorten sequences.

For the model, each 208-dimensional keypoint vector is projected by a single learnable linear layer into the T5 encoder, replacing the usual text token embeddings. The decoder generates text. Three variants are used: T5-base and T5v1.1-base (English only) and mT5-base (multilingual, tested with both Arabic and English targets).

Training proceeds in two stages. First, each model is pre-trained for 200,000 steps on YouTubeASL, a large in-the-wild ASL dataset, using Adafactor on AMD MI250x hardware. Second, models are fine-tuned for 100 epochs on the SSL corpus using AdamW on 8 NVIDIA A100-80GB GPUs. The comparison of interest is between fine-tuning from the original T5 weights and fine-tuning from the YouTubeASL checkpoints, yielding eight total fine-tuning experiments. Evaluation uses BLEU, ROUGE-L, and BLEURT (BLEURT is skipped for Arabic-trained models because it was not trained on Arabic).

Why This Matters

Impact on research. The paper provides direct evidence that sign language representations transfer across languages, which supports a practical strategy for under-resourced sign languages: pre-train on a large corpus from a different sign language rather than waiting for a large native corpus to be collected. The three-way test split design also offers a cleaner way to disentangle signer generalization from sentence generalization than many earlier benchmarks, and the released code and dataset give the community a reproducible baseline.

Real-world applications.

• Healthcare settings where qualified SSL interpreters are scarce and communication failures have direct safety consequences.
• Banking, government agencies, and legal services that serve Deaf clients and are increasingly expected to provide accessible communication.
• Education, where Deaf students benefit from instruction delivered in their native language rather than through intermediaries.
• Emergency services and public safety, where rapid, accurate translation of time-critical messages is essential.

Industry relevance. Accessibility technology is a growing commercial segment, and the finding that a modest dataset plus cross-lingual pre-training can produce a usable system lowers the barrier to entry for companies targeting regional sign languages. The paper's Saudi institutional authorship (HUMAIN, SDAIA, University of West Bohemia) signals active interest in deploying such systems within national digital-service infrastructure, and the pose-only design is attractive for privacy, since raw video never has to be stored or transmitted in recognizable form.

Future Directions

• Explore additional input modalities. The authors plan to test appearance-based features such as DINO or MAE representations, but note that these failed in preliminary runs, possibly because the dataset is too small for deep visual features to pay off. Determining the dataset size at which they become useful is an open question.
• Improve the preprocessing pipeline. Currently, any frame containing more than one person is discarded, wasting training data. Extending the pipeline to track and isolate the signer among multiple people would increase usable data, and the authors flag normalization choice as a critical but underexplored design decision.
• Address the Arabic-versus-English label gap. mT5 performed far better with English transcripts, which suggests that better Arabic modeling, Arabic-aware evaluation metrics, or more balanced multilingual pre-training could raise performance on the original language.
• Handle occluded faces and signer imbalance. Face coverings remove non-manual grammatical markers, and the signer pool skews male. Methods that compensate for missing facial cues, and corpora that better balance signer demographics, are natural next steps.

Target Audience

This paper is most valuable to researchers working on sign language translation, low-resource machine translation, and cross-lingual transfer, as well as to engineers building accessibility products for Arabic-speaking Deaf communities. It will also interest graduate students looking for a well-structured example of how to design dataset splits that isolate specific generalization challenges, and institution-level readers evaluating whether pose-based translation pipelines are mature enough for real deployment.

Authors’ abstract

This paper explores the application of T5 models for Saudi Sign Language (SSL) translation using a novel dataset. The SSL dataset includes three challenging testing protocols, enabling comprehensive evaluation across different scenarios. Additionally, it captures unique SSL characteristics, such as face coverings, which pose challenges for sign recognition and translation. In our experiments, we investigate the impact of pre-training on American Sign Language (ASL) data by comparing T5 models pre-trained on the YouTubeASL dataset with models trained directly on the SSL dataset. Experimental results demonstrate that pre-training on YouTubeASL significantly improves models' performance (roughly $3\times$ in BLEU-4), indicating cross-linguistic transferability in sign language models. Our findings highlight the benefits of leveraging large-scale ASL data to improve SSL translation and provide insights into the development of more effective sign language translation systems. Our code is publicly available at our GitHub repository.