Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

AI summary

Overview

Research area: Sign language processing, cross-lingual transfer learning, and sign language phonology.

Technical level: Intermediate. The core ideas are intuitive (reuse a feature inventory shared between languages), but the evaluation protocol involves phonological feature decomposition, constrained decoding, and distance-based matching that require some familiarity with sign language linguistics or multi-label classification.

One-sentence scope: This paper builds and evaluates the first zero-shot pipeline that recognizes Catalan Sign Language handshapes using models trained only on American Sign Language, by predicting shared phonological features instead of language-specific labels.

What This Paper Is About

Most sign language recognition research concentrates on a handful of high-resource languages like American Sign Language, because training these systems requires expensive video annotations that only fluent Deaf signers can produce. Low-resource sign languages such as Catalan Sign Language (LSC) have grammar documentation but almost no annotated video data, so they are locked out of modern recognition methods.

The authors ask whether a model trained purely on ASL can recognize LSC handshapes without ever seeing a single labeled LSC video. Their bet is that handshapes in both languages decompose into the same small set of articulatory features, so a model that predicts features rather than words can transfer across the language boundary.

Key Contributions

1. First zero-shot cross-lingual handshape recognition framework. The authors transfer from ASL to LSC without any target-language video training labels, evaluating three architecture families (a static MLP, SL-GCN, and the multimodal transformer SHuBERT) trained on two ASL corpora (PopSign and Sem-Lex).

2. A formal phonological alignment scheme. They map the 37 LSC handshapes from Navarrete-González's SignHub finger configuration taxonomy into the ASL-LEX 2.0 phonological feature space across five features: selected fingers, finger flexion, spread, thumb position, and thumb contact. This includes reconciling structural mismatches, such as SH-LSC treating the thumb as an ordinary selected finger while ASL-LEX handles it as a separate "thumb position" parameter.

3. A distance-based decoding protocol with an expected-accuracy metric. Because the same feature vector can correspond to more than one handshape (and predictions can land between valid configurations), the model returns a set of nearest candidate handshapes rather than a single label. Performance is scored as the probability of a correct pick under uniform selection from that candidate set.

4. Empirical demonstration that transfer works, but only after format harmonization. The headline result is 80.0% phonological feature accuracy and 54.5% expected handshape accuracy — achieved only after aligning the LSC benchmark's recording geometry and frame rate with the ASL training data, a step the authors show is responsible for most of the improvement.

Main Findings

• Zero-shot transfer is viable but fragile. All three architectures score far lower on LSC than in-domain on ASL. Under uncropped, full-frame-rate conditions, the best model (Sem-Lex MLP) reaches 62.6% feature accuracy and 27.3% expected handshape accuracy. After corpus-matched harmonization, the same model reaches 80.0% and 54.5%.

• Recording-format mismatch, not language difference, dominates the error. Spatial cropping alone yields roughly +10 to +11 points on both metrics across models and crops. Frame-rate subsampling contributes at most about 0.7 points on average, meaning nearly all of the "cross-lingual" gap was actually a capture-geometry gap.

• Harmonization is genuinely zero-shot. The crop is computed from each clip's own pose sequence plus constants taken from the source corpus — no labels, no fitting on the LSC test set. The authors also show that tuning the crop and frame rate on the LSC benchmark yields at most 2.4 (features) / 2.7 (handshape) points beyond the corpus-matched setting, so the gains are not an artifact of target-side tuning.

• The simplest model wins zero-shot. Despite being by far the weakest in-domain (48.8% expected handshape accuracy on Sem-Lex versus 78.4% for SL-GCN), the MLP is the strongest zero-shot model on LSC. The authors attribute this to high-capacity spatio-temporal models overfitting source-corpus biases and pose artifacts.

• Conversely, high-capacity models overfit the source domain. SL-GCN achieves 96.5% in-domain handshape accuracy on PopSign but only 8.3% zero-shot on LSC — a collapse that suggests it learned dataset-specific dynamics rather than transferable hand geometry.

• Sem-Lex transfers better than PopSign. Training on Sem-Lex (webcam, 41 Deaf signers) outperforms PopSign (smartphone, 47 learners) for the MLP and SHuBERT on LSC, though SL-GCN is indistinguishable across the two corpora. This likely reflects differences in recording modality and signer population.

• Reproducing Sem-Lex posed a hidden obstacle. Training SL-GCN on the officially released Sem-Lex pose files gives 57.1% gloss top-1, 9.5 points below the published 66.6%. Re-extracting poses from the raw videos with a unified MediaPipe pipeline raises this to 72.2%, exceeding the original baseline. The authors flag this as a likely property of the released pose data.

• SHuBERT can be pruned aggressively. Restricting SHuBERT to hand pose streams only matches the all-channel reference within noise, while a body-only variant collapses to predicting the majority class. Self-supervised pretraining, by contrast, matters a great deal: removing it costs 7.4 points of in-domain feature accuracy on Sem-Lex and 14.2 points of handshape accuracy.

• Auxiliary supervision is neutral for transfer. Retraining with only the five handshape-related heads instead of all 16 phonological heads produces no systematic zero-shot benefit or harm. Auxiliary heads mildly regularize deep models in-domain but neither help nor hurt cross-lingual transfer.

• The feature mapping is structurally lossy. Two pairs of LSC handshapes (7-flat_open vs. index+thumb-flat_open, and the closed variants) share identical five-feature vectors because the encoding ignores unselected finger flexion. Across 37 handshape classes there are only 35 unique canonical feature vectors, capping an ideal predictor at 94.6% expected accuracy.

Methodology in Plain English

The approach rests on a linguistic observation: handshape is not an atomic category but a bundle of smaller articulatory parameters, and those parameters are the same across signed languages. So instead of teaching a model to output ASL glosses (which are useless for LSC) or LSC handshape labels (which don't exist), the authors teach it to output the five phonological features.

Training. Three model families are trained on ASL data to predict phonological features from video or pose. The MLP sees a single representative hand frame as 63 raw 3D landmarks. SL-GCN sees full pose sequences as spatio-temporal graphs and is pretrained on isolated sign recognition before fine-tuning on features. SHuBERT is a self-supervised multimodal transformer pretrained on YouTube-ASL that fuses pixels and pose, fine-tuned here on phonological prediction.

Alignment. A rule-based mapping assigns the five ASL-LEX phonological features to each of the 37 LSC handshapes. Label conventions are harmonized where the two annotation schemes disagree — for instance, SH-LSC "Extended" becomes ASL-LEX "FullyOpen," and the thumb is pulled out of the selected-finger set and re-encoded as thumb position, matching ASL-LEX convention.

Inference. The model predicts each feature independently, but ASL has more classes than LSC for some features, so output logits are masked to valid LSC categories before taking the argmax. This still allows joint predictions that don't correspond to any real LSC handshape, since the heads are independent.

Decoding. Each candidate LSC handshape is compared against the prediction using a composite distance. Selected fingers use Hamming distance over the four non-thumb digits; flexion uses a graded scale where a fully-open-to-fully-closed jump costs double an adjacent step; spread, thumb position, and thumb contact are simple match/mismatch. The handshapes achieving the minimum distance form a candidate set, and the score is 1 divided by the size of that set if the true handshape is in it, and 0 otherwise.

Harmonization. Because the LSC benchmark is 1080p studio footage at 50 fps while the ASL corpora are webcam or smartphone clips near 30 fps, raw comparison confounds language transfer with capture geometry. The authors crop the LSC clips to a bounding box centered on the median shoulder position, scaled to match the training corpus's framing, and subsample frames to 25 fps. Crucially, this uses only the clip's own pose sequence and constants from the source corpus.

Why This Matters

Impact on research. This work reframes the low-resource sign language problem. Rather than collecting expensive target-language annotations, future systems could reuse a shared phonological inventory as a transfer substrate — an approach that scales to any language whose handshape inventory has been described linguistically. It also isolates an easily overlooked confound: much of what looks like cross-lingual degradation in prior work may be recording-format mismatch, and the finding that a shallow MLP beats large transformers zero-shot is a meaningful caution against assuming that bigger models transfer better.

Real-world applications:

• Sign language learning tools for low-resource sign languages that give feedback on handshape formation without requiring per-language annotated training video.
• Assistive recognition prototypes for Deaf communities in languages that lack commercial ASL-derived products, bootstrapped from a published handshape inventory.
• Lexicographic and documentation support, where predicted phonological features can help linguists annotate new sign language corpora faster by pre-filling handshape labels for expert correction.
• Fingerspelling and isolated handshape drills in educational software, where the target is a specific hand configuration rather than a full sign.

Industry relevance. Companies building sign language products have concentrated entirely on ASL because that is where the data is. A viable zero-shot transfer path changes the economics: a single ASL-trained model with a target-language feature mapping could cover multiple sign languages without new video collection. The result that SHuBERT can be pruned to hand-only streams without loss also matters operationally, cutting both compute and annotation requirements for deployment.

Future Directions

• Extend to multiple signers and recording conditions. The LSC benchmark has one Deaf signer and two videos per handshape across 37 classes. Signer-independent generalization and robustness to uncontrolled recording conditions cannot be established from this data, and the authors explicitly name this as a limitation.

• Test typologically unrelated language pairs. ASL and LSC both belong to the Francosign family, so their structural overlap may partly reflect shared lineage rather than universal phonological structure. Transferring between unrelated families would be a much stronger test of the universality hypothesis.

• Enrich the feature space. The five-feature encoding ignores unselected finger flexion, which causes structural ties between distinct handshapes, and it also ignores non-dominant hand configuration and handshape transitions over time. Adding these dimensions would expand transfer across the full manual phonological spectrum.

• Break decoding ties deterministically. Distance ties are currently resolved by uniform random guessing, which caps the metric and makes it a diagnostic rather than a true accuracy score. Per-feature model confidences could resolve distance ties; structural ties would need additional feature information such as unselected finger flexion.

Target Audience

This paper is most useful to researchers in sign language processing and multimodal machine learning who work on low-resource transfer, and to computational linguists interested in whether phonological representations are genuinely language-agnostic. It also speaks to practitioners building sign language applications who need to decide whether an ASL-centric model can be extended to another signed language. Readers with a background in sign language phonology or cross-lingual NLP will get the most out of the technical details; those without that background can still follow the core argument, since the methodology is explained in fairly accessible terms.

Authors’ abstract

Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features -- selected fingers, flexion, spread, thumb position, and thumb contact -- shared across both languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign language technologies to low-resource languages without any target-language video training labels.