Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

AI summary

Overview

Research area: Sign Language Recognition (SLR) combined with cross-lingual transfer learning and video domain adaptation. The work applies the Temporal Attentive Adversarial Adaptation Network (TA3N), containing the Temporal Relation Network (TRN) module, to transfer knowledge between different sign languages.

Technical level: Advanced. The paper assumes familiarity with deep video classification, adversarial domain adaptation, and multi-scale temporal modeling, although its central question (which temporal timescale is best to align) is stated plainly.

Scope in one sentence: The paper measures how well a model trained on Argentine Sign Language (LSA) or Chinese Sign Language (CSL) can be adapted to American Sign Language (ASL), and which multi-scale temporal relation setting (3, 5, 7, 10 or 15) and input modality (RGB vs. Optical Flow) produce the best target-domain accuracy.

What This Paper Is About

Sign language datasets are scarce: the abstract notes resources are severely lacking for the over 100 distinct sign languages, and the introduction states that SLR research tends to focus on only a few of the 135 sign languages used worldwide. Low-resource datasets like ASL tend to overfit, so the authors transfer knowledge from higher-resource sign languages (LSA and CSL) into ASL using domain adaptation rather than generic ImageNet-style pre-training.

The goal is to find out whether domain adaptation beats neural-network-based transfer learning for sign language to sign language transfer, and specifically which multi-scale temporal relation scale and which video input modality work best.

Key Contributions

1. Multi-scale temporal alignment between different sign languages. The authors implemented the TRN-based multi-scale temporal alignment of TA3N to perform domain adaptation between distinct sign languages (LSA to ASL and CSL to ASL). They state that no prior studies had applied domain adaptation between different sign languages.

2. Isolating the effect of temporal timescale. They compared 3-, 5-, 7-, 10- and 15-multiscale TRNs, treating 3, 5 and 7 as shorter-term and 10 and 15 as longer-term, to determine which scale improves target recognition most.

3. Two learning settings. Domain adaptation was run in full-scale transfer learning (80:20 target training-to-test split) and few-shot transfer learning (20:80 split, roughly 2 target training samples per class), with the paper noting that to its knowledge no prior SLR research had explored few-shot transfer learning.

4. Modality comparison. Every experiment was repeated with RGB and Optical Flow inputs, using the Gunner Farneback optical flow algorithm, and the two were compared for each configuration.

Main Findings

• Domain adaptation beat neural-network-based pre-training. The abstract reports that Domain Adaptation performed better than neural network-based transfer learning, particularly for improving ASL recognition. The detailed comparison is in Section 4.3, which is not included in the provided content.

• Shorter-term temporal alignment was better overall. Aligning shorter-term temporal relations (3, 5 and 7) scored higher, on average, than longer-term ones in terms of improvement over strong baselines, both in full-scale and few-shot domain adaptation, and this held for both LSA-to-ASL and CSL-to-ASL in all modalities used.

• Full-scale LSA to ASL, RGB: the 5-multiscale TRN gave the highest domain-adapted accuracy at 12.93%, a 59.62% increase over the baseline, the largest improvement in that group.

• Full-scale LSA to ASL, Optical Flow: the 3-multiscale TRN reached the maximum accuracy of 10.08%, a 15.15% improvement over its baseline, while the 7-multiscale TRN produced the highest improvement at approximately 92.017% from its baseline and the second-highest accuracy at 9.958%; the authors conclude 7-multiscale is best for this group.

• Full-scale CSL to ASL: the 7-multiscale TRN gave the highest accuracies in both RGB (11.86%, a 24.19% improvement over baseline) and Optical Flow (10.08%, a 10.31% improvement). For Optical Flow specifically they judged the 3-multiscale TRN most effective, since it improved nearly 20% over its baseline with a close-second accuracy of 10.01% versus 7-multiscale's 10.08%.

• Negative transfer occurred. In full-scale CSL to ASL Optical Flow, the 15-multiscale TRN produced a domain-adapted accuracy of 8.618%, lower than its baseline of 10.112%. In few-shot CSL to ASL, the 15-multiscale TRN likewise produced domain-adapted models below baseline (8.796% vs 8.95%).

• High improvement percentages did not always mean the best model. For CSL to ASL full-scale RGB, the 15-multiscale TRN improved its baseline the most (57.38%) but yielded the second-lowest domain-adapted accuracy (9.906%); similarly, in few-shot CSL to ASL RGB, the 5-multiscale TRN improved its baseline the most (19.32%) yet its model had the second-lowest accuracy (8.572%).

• Few-shot LSA to ASL, RGB: the 3-multiscale TRN achieved the maximum accuracy of 6.85% against a baseline of 6.4%, a 7.03% difference.

• Few-shot LSA to ASL, Optical Flow: the 10-multiscale TRN achieved the maximum accuracy of 7.34%, a 26.48% increase over its baseline, the largest improvement in that group and the one case where a longer-term scale was selected as best.

• Few-shot CSL to ASL: the 7-multiscale TRN produced the maximum accuracies for both modalities, 9.266% in RGB (3.41% over the 8.96% baseline) and 7.948% in Optical Flow (4.55% over the 7.602% baseline). For Optical Flow the 3-multiscale TRN was judged most effective, with the second-highest accuracy of 7.788% and a 12.54% improvement.

• Full-scale learning gave higher accuracies than few-shot learning. The paper states that conducting full-scale learning resulted in domain-adapted models with higher accuracy than the few-shot setting.

• RGB generally outperformed Optical Flow. The abstract concludes RGB outperforms Optical Flow in the majority of cases; the sentence supporting this in the results is truncated in the provided content.

• Absolute accuracies are low, and the authors explain why. They attribute the modest numbers to the difficulty of the task: both one-handed and two-handed dynamic word-level signs, ASL subsets drawn from diverse online sources with varying lighting, backgrounds, aspect and camera distance, and only about 1.3 repetitions per sign per subject in the WLASL subset used, versus 5 repetitions in LSA64 and CSL.

Methodology in Plain English

The pipeline has two steps: pre-processing, then domain adaptation.

Pre-processing. Videos are converted into RGB frames and Optical Flow frames. Optical Flow tracks pixel motion between frames and was computed with the Gunner Farneback algorithm, which the authors chose because it is invariant to appearance and therefore easier to generalize for low-resource data. A maximum of 200 frames per sample was extracted, since most samples contain fewer than 100 frames.

Data. The source domains are LSA64 for Argentine Sign Language and the CSL Isolated Chinese Sign Language dataset for Chinese Sign Language; the target is WLASL300, a 300-class subset of Word-Level American Sign Language. Classes were chosen where labels overlap across source and target: 26 mutual labels between LSA and ASL, and 23 between CSL and ASL. Training and testing video counts are 1,040/260 for LSA, 4,600/1,150 for CSL, 284/71 for the ASL subset used with LSA, and 257/64 for the ASL subset used with CSL.

Model. The TA3N architecture first encodes frames with a CNN, refines them in a Spatial Module, then feeds them to the TRN module, which fuses features of N sparsely and evenly sampled, chronologically ordered frames into N-frame temporal relations and accumulates them into a multi-scale TRN (T2 + T3 + ... + TN). Two networks follow: a Temporal Classification Network for class prediction and a Temporal Adversarial Discriminator for domain prediction. The discriminator assigns higher attention weights to temporal relations that are more domain-discriminative. Training minimizes prediction loss while maximizing domain prediction loss, plus an Attentive Entropy Loss (the product of domain entropy and class entropy) that pushes the model to focus on videos with high domain discrepancy.

What was tuned. TA3N's default multi-scale TRN is 5, and it was originally benchmarked on the large-scale action datasets HMDB51 and UCF101 with a ResNet-101 backbone. Because the default implementation performed poorly on sign language, the authors swept N over 3, 5, 7, 10 and 15. Parameter counts ranged from 2,582,844 (3-multiscale, LSA to ASL) up to 18,323,796 (15-multiscale, LSA to ASL), and from 2,580,534 up to 18,321,486 for CSL to ASL. They also switched TA3N from its default unsupervised mode to a supervised one by setting the use_target parameter to sV, so classification loss is computed on the concatenation of labeled source and target data.

Evaluation. All experiments used randomized 5-fold cross-validation, training for up to 100 epochs with a batch size of 20. They report mean accuracy and standard deviation for each fold set. Two settings were used: full-scale (80:20 target training-to-test) and few-shot (20:80, about 2 training samples per class).

Why This Matters

Impact on research. The paper opens a direction — domain adaptation between different sign languages — that it says had no prior studies. It also challenges the default assumption that pre-training large networks on ImageNet-like data is the best transfer route for SLR, arguing that sign-language-to-sign-language transfer captures high-level visual features that general non-sign-language datasets cannot provide. Its finding that short-term temporal alignment beats long-term alignment, and that a smaller N than TA3N's default of 5 is often better for low-resource SLR, is a concrete, reusable design guideline. The negative-transfer cases (15-multiscale TRN) are equally useful evidence about when adaptation hurts.

Real-world applications:

• Assistive translation tools that convert sign language video into spoken-language text for people who do not sign.
• Recognition systems for low-resource sign languages where no large labeled dataset exists, built by adapting from a higher-resource sign language.
• Video platforms and educational services that could auto-index or caption signed content.
• Rapid prototyping of gesture interfaces using only RGB or Optical Flow from ordinary cameras, since the paper deliberately avoids equipment such as specialized cameras or Leap Motion sensors.

Industry relevance. The method relies only on video frames, so it needs no extra hardware. That makes it relevant to consumer camera and mobile pipelines, accessibility features in communication apps and video conferencing, and content platforms that want searchable signed video. The WHO projection cited in the paper — that by 2050 one in ten people will have some degree of hearing loss — frames the scale of the accessibility market.

Future Directions

• Generalizing beyond ASL. The authors state the methodology can be applied universally to any sign language recognition task; testing other source–target pairs, including sign languages from other families, is the natural next step.
• Reconciling the timescale result with the exceptions. Shorter-term alignment won on average, but 7-multiscale was best for CSL to ASL and 10-multiscale was best for few-shot LSA to ASL Optical Flow. What determines the right scale is still open.
• Avoiding negative transfer. The 15-multiscale TRN degraded accuracy below baseline in two CSL-to-ASL settings; detecting and preventing such cases automatically is unresolved.
• Closing the few-shot gap. Full-scale learning outperformed few-shot learning in these experiments, so improving few-shot transfer for very small target sets remains a target.
• Open items in the provided content: the detailed comparison against neural-network-based transfer learning (Section 4.3), any Section 5, and the full conclusions (Section 6) are truncated, so their specific numbers are not reported here.

Target Audience

Researchers and graduate students in sign language recognition, video understanding and transfer learning; practitioners building accessibility or gesture-based products who need to know whether to invest in domain adaptation rather than ImageNet pre-training; and readers interested in low-resource video classification, since the paper's core lesson — align short-term temporal relations, keep the multi-scale setting small, and prefer RGB — is stated in terms that transfer beyond sign language.

Authors’ abstract

Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.