Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing

AI summary

Overview

Research area: Sign language processing and cross-modal video–text alignment, at the intersection of natural language processing and computer vision.

Technical level: Intermediate. The method builds on pretrained sign segmentation and vision–language (CLIP-style) models and a dynamic programming alignment step; no novel network architecture is trained end-to-end.

Scope: The paper proposes SEA (Segment, Embed, and Align), a modular, largely training-free recipe that re-times spoken-language subtitles so they match the signing in continuous sign language video, and evaluates it on four datasets spanning BSL, ASL, and DSGS.

What This Paper Is About

Subtitles for sign language video (for example, broadcast interpreting) are usually derived from the original audio track, so they frequently lag or lead the actual signing by a non-deterministic amount. Manually re-aligning them is expensive: the paper reports that an expert fluent in sign language needs approximately 10–15 hours to align subtitles to one hour of continuous signing, at roughly 40 USD per hour according to the WMT-SLT 22 campaign. SEA's goal is to fix these timestamps automatically with a single framework that transfers across languages and datasets rather than being tied to one language or corpus through end-to-end training.

Key Contributions

1. A universal, modular alignment framework (SEA) consisting of three steps — segment continuous signing into individual units, embed signs and subtitle units into a shared latent space, and align them — designed to work across multiple sign languages and data sources without in-domain alignment supervision.
2. A global, lightweight alignment algorithm based on dynamic programming that optimizes a cost function over the whole episode, combining temporal/prosodic costs (onset and offset distance, duration difference, internal inter-sign gaps) with an optional text–sign semantic similarity term. It runs on CPUs within a minute even for hour-long episodes, in contrast to prior work that operates in local windows of about 20 seconds followed by dynamic time warping.
3. State-of-the-art results on four benchmarks covering BSL, ASL, and DSGS (BOBSL, How2Sign, WMT-SLT, and SwissSLi), including three datasets on which this task had not previously been evaluated. SEA is reported as SOTA on all four test sets.
4. A new expert-annotated test set: the authors hired DSGS experts to annotate 14 episodes of the mitenand program from SwissSLi as a test-only, highly low-resource DSGS/German benchmark, and release all code and models openly.

Main Findings

• SEA beats prior state of the art on all four test sets. Using F1@0.50 (a subtitle counts as correct when its predicted segment overlaps the ground truth with IoU ≥ 0.50), the best SEA variants reach 65.81 on BOBSL test, 39.57 on How2Sign test, 77.69 on WMT-SLT test, and 85.57 on SwissSLi test. On BOBSL this surpasses SAT (54.57) and SAT+ (63.81); on WMT-SLT, SAT scores 75.32 while SEA with finetuned embeddings scores 77.69.
• Original audio-based subtitles are poor, and fixed offsets help unevenly. Original alignment scores 29.09/14.11 (BOBSL val/test), 30.63/33.06 (How2Sign), 47.83/46.85 (WMT-SLT), and 69.58/60.48 (SwissSLi). Adding fixed offsets from training data raises BOBSL to 49.63/44.61 and WMT-SLT to 74.17/74.83, but only lifts How2Sign to 31.91/36.21, because that dataset's production workflow differs.
• Segmentation alone improves alignment. The "Segment and Align" variant, which uses only timing and prosodic costs, reaches 66.24/49.58 on BOBSL, 33.38/36.17 on How2Sign, 75.66/76.83 on WMT-SLT, and 71.48/84.19 on SwissSLi — already matching or exceeding SAT on WMT-SLT.
• Adding embeddings gives consistent gains across languages and datasets. With the multilingual SignCLIP model (which saw no DSGS/German data during pretraining), scores rise to 66.70/50.68 (BOBSL), 35.51/37.51 (How2Sign), 75.16/76.43 (WMT-SLT), and 71.86/85.57 (SwissSLi).
• Language-specific finetuning of the embedding model helps substantially where data exist. Finetuning on in-domain BOBSL sign spottings gives roughly 4 to 6 F1@0.50 improvement on BOBSL validation and test (72.78/54.50). Finetuning on three large out-of-domain isolated sign recognition datasets helps How2Sign (38.32/39.57). Finetuning on the small Signsuisse lexicon yields weak effects for WMT-SLT (75.34/77.69) and SwissSLi (71.86/85.22).
• SEA can build on other systems' output. Initializing SEA with subtitles predicted by SAT+ produces 75.27 on BOBSL validation and 65.81 on test, inheriting and refining those gains.
• Better isolated sign recognition correlates with better alignment. SignCLIP-BSL reaches 43.0 ISLR on BOBSL sign spottings versus 0.5 for SignCLIP-multilingual, corresponding to alignment of 72.78 versus 66.70; for ASL on PopSign, 84.3 versus 3.0 corresponds to 38.32 versus 35.51.
• Soft similarity beats rigid gloss matching. Using CSLR2 pseudo-glosses to fill the similarity matrix gives 68.80 on BOBSL validation, below SignCLIP-BSL's 72.78; an oracle using CSLR2 human glosses reaches 78.75.
• Segmentation quality and alignment quality do not move in lockstep. Finetuning the segmenter on 3.3 hours (149 videos) of the BSL Corpus achieves the best segmentation scores (51.46 F1, 55.46 mF1S, versus 31.13/33.32 for the original Moryossef et al. model) but yields worse alignment (63.61 versus 66.24 on BOBSL validation). Renz et al. (2021a) scores 47.71 segmentation F1 with 57.98 alignment, and Renz et al. (2021b) gives 57.53 alignment.
• Dataset-specific biases matter. The random search recovers offsets similar to the "Original +" baseline, and a fixed 1-second offset applied after the DP process is reported as always useful.

Methodology in Plain English

The authors deliberately avoid training one big end-to-end network for alignment. Instead they reuse two existing pretrained tools and put a small optimizer between them.

1. Segment. A pretrained automatic sign segmentation model (Moryossef et al., 2023) — an LSTM over MediaPipe Holistic poses, trained on about 73 hours of annotated Public DGS Corpus data — chops a continuous video into sign-like units. It was trained only on German Sign Language but the paper reports successful zero-shot transfer to other sign languages, including French Sign Language at 0.76 ROC-AUC on the O-tag, and finds it useful for ASL, BSL, and DSGS. Two decoding thresholds (b-threshold and o-threshold) control when a sign starts and ends.
2. Embed. Each segmented sign clip and each subtitle unit are mapped into a shared latent space with SignCLIP, a CLIP-style model that encodes subtitles with a BERT-like text encoder and signing clips as MediaPipe pose sequences through a BERT-initialized pose encoder. The default checkpoint is SignCLIP-multilingual, pretrained on SpreadtheSign and covering 41 sign languages with English on the text side only. The authors also finetune variants: SignCLIP-BSL (on BOBSL sign spottings, about 3.5M examples, using a contrastive text objective instead of classification over 8,697 signs), SignCLIP-ASL (about 200K examples from PopSign, ASL Citizen, and Sem-Lex), and SignCLIP-Suisse (Signsuisse, 16,213 lexical items). Finetuning takes under 3 days on a single NVIDIA RTX A6000 with 48 GB.
3. Align. Every subtitle is assigned a contiguous span of signs, and its timestamps are rewritten to that span's boundaries. A cost function penalizes how far the subtitle moves, how much its duration changes, and how large the gaps are between signs inside the span, and (in the full version) rewards spans whose sign embeddings are similar to the subtitle text. A dynamic programming recurrence finds the globally optimal assignment for the whole episode, with a window size of 50 limiting each subtitle to its temporally nearest signs and a softmax over rows turning dot-product similarities into a usable matrix. A maximum inter-sign gap constraint discards unsuitable groups. The DP table is indexed by subtitle and sign, does not require every sign to be used (since false positives tend to appear near episode ends), and is back-tracked to recover the assignment.

Parameter settings such as the DP weights, segmentation thresholds, and pre/post offsets were found by random search of up to 10,000 iterations (about one day on 128 CPU workers) for the timing-only variant, then largely inherited when the embedding term was added.

Why This Matters

Impact on research. Prior alignment work such as SAT and SAT+ depends on manually aligned in-domain data (SAT was finetuned on 17.7 hours of manual annotation) and on a strong in-domain visual backbone like I3D that does not necessarily transfer to another dataset or sign language, as the WMT-SLT results show. SEA reframes alignment as a modular, largely training-free pipeline whose components can be swapped plug-and-play, and shows that the segmentation module in particular has relatively strong cross-linguistic universality while the embedding module benefits from language-specific expertise. This lowers the barrier to producing parallel text–sign data, which sign language translation systems require for training and evaluation.

Real-world applications:

• Correcting broadcast subtitle timing for interpreted programming, as in the BBC-sourced BOBSL data used here.
• Automatic subtitling of user-generated sign language content on platforms such as YouTube, reducing post-hoc captioning effort and improving accessibility.
• Pre-segmenting and synchronizing material for linguists building and annotating corpora, such as the DGS Corpus, with tools like ELAN or iLex.
• Generating higher-quality parallel data to train and evaluate sign language translation systems.

Industry relevance. Broadcasters and streaming platforms that carry interpreted content, accessibility and captioning vendors, sign language corpus and lexicon producers, and developers of sign language translation and recognition products all stand to benefit from a CPU-efficient, run-once-per-episode alignment step.

Future Directions

• Iterative co-improvement of embedding and alignment. The authors state that the embedding and alignment models could be improved iteratively with respect to each other and leave this to future work.
• Robustness to dataset-specific biases and data quality. Suggested mechanisms include dropping irrelevant signing or subtitles, which would help the bad cases observed on How2Sign, and merging segments when boundaries are hard to discern, which would also support paragraph-level downstream applications.
• Human-in-the-loop workflows. Post-editing and evaluation with human annotators' intervention were not studied; establishing a semi-automatic process is proposed as valuable for both alignment quality and practical usability.
• Scaling the harvested data. Turning SEA's improved alignment into large-scale, high-quality parallel text–sign data for advancing sign language processing remains an open direction, along with testing broader language coverage beyond BSL, ASL, and DSGS.

Target Audience

Researchers and practitioners in sign language processing, cross-modal video–text alignment, and accessibility engineering who want a strong baseline that does not require in-domain alignment supervision; sign language linguists and corpus builders interested in pre-alignment tooling; and engineers on subtitle, captioning, or translation pipelines who need a CPU-efficient way to re-time existing subtitles to signing. Readers should be comfortable with contrastive vision–language models, dynamic programming, and standard alignment metrics, though the paper explains each component from the ground up.

Authors’ abstract

The goal of this work is to develop a universal approach for aligning subtitles (i.e., spoken language text with corresponding timestamps) to continuous sign language videos. Prior approaches typically rely on end-to-end training tied to a specific language or dataset, which limits their generality. In contrast, our method Segment, Embed, and Align (SEA) provides a single framework that works across multiple languages and domains. SEA leverages two pretrained models: the first to segment a video frame sequence into individual signs and the second to embed the video clip of each sign into a shared latent space with text. Alignment is subsequently performed with a lightweight dynamic programming procedure that runs efficiently on CPUs within a minute, even for hour-long episodes. SEA is flexible and can adapt to a wide range of scenarios, utilizing resources from small lexicons to large continuous corpora. Experiments on four sign language datasets demonstrate state-of-the-art alignment performance, highlighting the potential of SEA to generate high-quality parallel data for advancing sign language processing. SEA's code and models are openly available.