POSESTITCH-SLT: Linguistically Inspired Pose-Stitching for End-to-End Sign Language Translation

AI summary

Overview

Research area: Natural Language Processing, specifically sign language translation (SLT) — translating sign language video into spoken-language text.

Technical level: Intermediate. The paper assumes familiarity with encoder-decoder Transformers, BLEU evaluation, and pretraining pipelines, but the central idea (stitching word-level pose clips into sentences) is explained in plain terms.

Scope in one sentence: The paper proposes PoseStitch-SLT, a pretraining scheme that synthesizes sentence-level sign pose data by stitching together word-level pose clips using linguistic templates, and shows this improves gloss-free, pose-based sign language translation on two benchmarks.

What This Paper Is About

Sign language translation is hard mainly because there are very few large, sentence-aligned sign language datasets — far fewer than in spoken-language NLP. The authors' goal is to improve translation without relying on expensive gloss annotations (hand-written sign labels) or raw signer-identifiable video. Their idea is to manufacture large amounts of synthetic training data: take word-level sign datasets that already exist, take grammatical sentence templates, and "stitch" the corresponding word poses into full sentences so a standard Transformer can be pretrained on them before learning from real data.

Key Contributions

1. A linguistically grounded synthetic data and pretraining strategy called PoseStitch-SLT, which builds sentence-level pose sequences by stitching word-level pose clips according to grammatical templates, for pose-based, gloss-free sign language translation.
2. State-of-the-art results on two benchmarks in a gloss-free, pose-only setting using a standard Transformer: BLEU-4 improvements from 1.97 to 4.56 on How2Sign (ASL) and from 0.55 to 3.43 on iSign (ISL) on the test sets.
3. Two families of synthetic datasets released as part of the work: BLiMP-based (BLiMP-ISL with 22,219,407 sentences and BLiMP-ASL with 2,880,008 sentences) and BPCC-based (BPCC-ISL with 1,640,469 sentences and BPCC-ASL with 1,173,536 sentences).
4. Public release of dataset and code at https://github.com/Exploration-Lab/PoseStich-SLT.

Main Findings

• Large BLEU-4 gains over the prior state of the art on How2Sign (ASL): BLEU-4 improves from 1.97 to 4.56 on the test set and from 2.27 to 5.04 on the dev set, comparing against the GloFE baseline as reported in Table 1.
• Large BLEU-4 gains on iSign (ISL): BLEU-4 improves from 0.55 to 3.43 on the test set and from 0.61 to 3.54 on the dev set, again against GloFE.
• Best overall scores (denoted "Best (Ours)"): On How2Sign, BLEU-1/2/3/4 of 27.35 / 13.56 / 7.92 / 5.04 on dev and 26.74 / 12.90 / 7.26 / 4.56 on test. On iSign, 17.31 / 8.09 / 5.02 / 3.54 on dev and 17.67 / 8.20 / 5.00 / 3.43 on test.
• Same Word Order (SWO) generally beats Random Word Order (RWO): On How2Sign, the SWO + BLiMP setting gives the best test BLEU-4 (4.56) while SWO + BPCC gives the best dev BLEU-4 (5.04). On iSign, SWO + BPCC is best on both dev (3.54) and test (3.43).
• Synthetic data is essential: A baseline with the same architecture and hyperparameters but no pose-stitched synthetic data performs significantly worse — for example, test BLEU-4 of 2.62 on How2Sign and 0.64 on iSign.
• Strong in-domain generalization to unseen stitched sentences: On two synthetic evaluation sets not seen during training, the model reaches BLEU-4 scores of 97 and 47. The authors attribute this to the very large dataset size paired with a small vocabulary, which makes generalized representations easier to learn.
• Semantic similarity to the target domain correlates with translation quality: Using SBERT (all-MiniLM-L6-v2) embeddings, the paper reports that more similar synthetic/target sentences correspond to better translation, and the model performs worse on the least similar sentences — evidence that it learns context rather than surface patterns.
• Qualitative behavior: Predicted translations consistently capture the main semantic content and align better with references than GloFE, but phrase repetition and hallucinations still occur, which the authors attribute to noise in the synthetic data.
• ROUGE-L and SacreBLEU scores are also reported in the appendix (Tables 12 and 13); the main text states improvements are consistent across both datasets.

Methodology in Plain English

The pipeline works in four stages.

1. Find a shared vocabulary. The authors take BLiMP, a benchmark of English grammatical templates covering 67 paradigms across 12 categories, and intersect its vocabulary with two word-level sign datasets: CISLR for Indian Sign Language (ISL, 4,764 unique words) and WLASL for American Sign Language (ASL, 2,000 unique words). The overlaps are 504 words for CISLR and 508 words for WLASL.
2. Generate sentences. Filling BLiMP templates with the shared vocabulary produces 22,219,407 sentences for BLiMP-ISL and 2,880,008 for BLiMP-ASL. Because template vocabulary coverage is narrow, the authors add sentences from the BPCC corpus (230 million English bitext pairs), keeping only sentences with more than 90% word match to the CISLR or WLASL vocabulary. After merging short sentences to match the target length distributions, this yields BPCC-ISL (1,640,469 sentences) and BPCC-ASL (1,173,536 sentences).
3. Stitch poses. For every word in a generated sentence, the corresponding word-level sign video is retrieved and converted into 2D keypoints using Mediapipe, which outputs 576 keypoints (21 left hand, 21 right hand, 33 body pose, 468 face). The authors hand-pick 76 of these — all hand points, 11 body points, and 23 facial points — giving a 152-dimensional vector per frame. Keypoints below a 0.8 confidence threshold are filled from the nearest neighboring frame. Word-level segments are then concatenated into a continuous stream with boundary-aware temporal smoothing, and frame rates are matched to the target datasets.
4. Train with a blended curriculum. A standard Transformer encoder-decoder (BERT encoder, GPT2 decoder, via HuggingFace) is trained on the synthetic pose-sentence pairs, then linearly annealed toward the real data: at step 0 it samples 0% real data, rising to 85% real data (iSign or How2Sign) at 60,000 steps. The authors use this blended progressive curriculum instead of separate pretraining and fine-tuning phases, which they say avoids catastrophic forgetting. For iSign, encoder and decoder each have 4 layers, hidden size 512, 8 attention heads; the BPE tokenizer has a vocabulary of 15,000, the optimizer is ADAMW with learning rate 3e-4, batch size 16, and dropout 0.1.

Two stitching variants are compared: Same Word Order (SWO), where poses follow English word order, and Random Word Order (RWO), where poses are randomly permuted to inject syntactic noise. The authors adopt English word order as a proxy for sign language order because reliable syntactic resources for ASL and ISL grammar are not available.

Why This Matters

Impact on research. The paper is notable for deliberately holding the model architecture fixed — a vanilla Transformer encoder-decoder — so that any gains can be attributed to the data synthesis and curriculum rather than to architectural tricks. This positions synthetic, template-driven pretraining as an alternative axis of progress for low-resource sign language translation, alongside the more common approach of adding gloss supervision or specialized architectures. It also keeps the setting privacy-preserving: pose keypoints only, no raw video, no gloss annotations.

Real-world applications (from the framing of the work and its setting):

• Assistive translation tools for deaf and hard-of-hearing users, though the authors explicitly state the system still lacks the capability to function as a fully realized interpreter in real-life scenarios.
• Generating parallel sign-language data for ASL and ISL in settings where human annotation is too costly, since the method needs no proprietary data or gloss labels.
• Curriculum or pretraining pipelines for other low-resource multimodal translation tasks, since the framework is described as modular and extensible as new sign language datasets appear.
• Privacy-conscious deployment, because keypoint inputs avoid signer-identifiable video features.

Industry relevance. Any organization building sign language interfaces needs parallel training data, which is the bottleneck this paper attacks. The released datasets and code lower the barrier to entry, and the finding that a standard Transformer plus synthetic data can beat specialized systems suggests that data strategy may matter more than model complexity in this domain.

Future Directions

1. Expand vocabulary coverage. The method is limited by the intersection of BLiMP vocabulary with WLASL and CISLR, which cover only about 2K to 4.5K words. The authors identify extending word-level datasets as critical but beyond the scope of this work due to a lack of publicly available resources.
2. Incorporate sign language grammar. The current pipeline uses English word order as a proxy because grammatical annotations for sign languages are limited and not standardized. Modeling sign-specific syntax would require linguistic resources and annotation efforts that do not currently exist for most sign languages.
3. Move beyond a standard Transformer. Architectural simplicity was a deliberate choice to isolate the effect of the training strategy, but the authors acknowledge it may underutilize recent sign-specific architectures — so combining this pretraining scheme with specialized models is an open avenue.
4. Extend to more sign languages. The approach is described as applicable beyond ASL and ISL, but broader adoption depends on word-level datasets existing for other sign languages. The paper also notes that direct comparison with prior methods such as GloFE was limited because implementation details were unavailable.

Target Audience

This paper is most useful to researchers and graduate students working on sign language processing, low-resource multimodal translation, or synthetic data generation for NLP. It is also relevant to practitioners who need to build or pretrain sign language translation systems under privacy constraints and without gloss annotations. Readers with a general NLP background will follow the core argument, but the appendix-level detail on pose processing, frame-rate matching, and vocabulary overlap assumes familiarity with keypoint pipelines and sequence-to-sequence training.

Authors’ abstract

Sign language translation remains a challenging task due to the scarcity of large-scale, sentence-aligned datasets. Prior arts have focused on various feature extraction and architectural changes to support neural machine translation for sign languages. We propose POSESTITCH-SLT, a novel pre-training scheme that is inspired by linguistic-templates-based sentence generation technique. With translation comparison on two sign language datasets, How2Sign and iSign, we show that a simple transformer-based encoder-decoder architecture outperforms the prior art when considering template-generated sentence pairs in training. We achieve BLEU-4 score improvements from 1.97 to 4.56 on How2Sign and from 0.55 to 3.43 on iSign, surpassing prior state-of-the-art methods for pose-based gloss-free translation. The results demonstrate the effectiveness of template-driven synthetic supervision in low-resource sign language settings.