RoCoISLR: A Romanian Corpus for Isolated Sign Language Recognition

AI summary

Overview

• Research area: Computer Vision, specifically isolated sign language recognition (ISLR) for a low-resource language, with ties to video action recognition and accessibility technology.
• Technical level: Advanced. The dataset construction is accessible, but the benchmark section assumes familiarity with 3D CNNs, transformer-based video architectures, transfer learning, and skeleton/pose models.
• Scope in one sentence: The paper introduces RoCoISLR, the first large-scale standardized Romanian Isolated Sign Language Recognition corpus, and benchmarks seven deep learning video models on it against WLASL2000 and other sign language datasets.

What This Paper Is About

Romanian Isolated Sign Language Recognition has no large-scale, standardized dataset, so researchers cannot train or compare modern models for it. The authors build such a dataset by pooling videos from several public Romanian sign language sources, cleaning and merging the sign labels into a consistent vocabulary, and then running a common benchmark so future work has a reproducible baseline. They also use the results to show how hard low-resource sign recognition is when most signs have very few examples.

Key Contributions

1. They construct and release RoCoISLR, described as the first standardized dataset for Romanian Isolated Sign Language Recognition, built from multiple source collections and a gloss standardization pipeline.
2. They provide detailed dataset statistics and identify dataset-specific challenges, most notably a severe long-tail distribution and a low number of samples per class.
3. They benchmark seven representative deep learning architectures (I3D, SlowFast, Swin Transformer, TimeSformer, UniFormer V2, VideoMAE V2, and PoseConv3D) under consistent experimental protocols and contextualize the results with WLASL2000 and other sign language corpora.
4. They establish a reproducible foundation for future research on low-resource sign languages, including standardized video processing and generated metadata files.

Main Findings

• Dataset size and shape: RoCoISLR contains 9,141 original video files spanning 5,892 canonical glosses and 25 distinct contributing signers. It is drawn from three source groups: DLMG (6,744 videos), PeSemne (1,191), and Miscellaneous (1,206).
• Gloss standardization reduced vocabulary: The authors started with 6,129 identified glosses and reduced them to 5,892 canonical glosses using variant merging, complexity filtering (labels with more than three hyphens discarded), and near-duplicate matching at a 0.95 similarity threshold.
• Extreme class imbalance: Over 67% of glosses (3,966 out of 5,892) are represented by a single video, nearly 20% by two videos, and only a small fraction by more than five videos. The best-represented glosses are "bate" with 10 videos, "da" and "multumesc" with 9 each, and eight glosses with 8 videos each ("fantana", "influenta", "cosmar", "perfect", "piele-de-gaina", "palid", "proteza", "industrie").
• Benchmark performance is low: On the 1,926 classes with at least two videos, Swin Transformer achieved the best Top-1 accuracy of 34.10% (Top-5 40.70%). Other results were TimeSformer 30.20% / 34.80%, PoseConv3D 25.70% / 30.70%, I3D 24.00% / 32.30%, SlowFast 23.80% / 26.00%, VideoMAE V2 23.40% / 32.10%, and UniFormer V2 20.70% / 32.80%.
• Transformers beat convolutional baselines: The abstract states that transformer-based architectures outperformed convolutional baselines, with Swin Transformer leading.
• Accuracy correlates with samples per class: RoCoISLR has 2.7 samples per class on average, compared with 10.5 for WLASL2000, 22 for MM-WLAuslan, 30 for SignBD-Word, and 85 for Multi-VSL 1000. The same I3D model reaches 24.00% Top-1 on RoCoISLR versus 32.48% on WLASL2000, and Swin Transformer reaches 76.43% Top-1 on Multi-VSL 1000 versus 34.10% on RoCoISLR.
• Error behavior of the best model: In the Swin Transformer confusion matrix (1,926 x 1,926), the paper reports that 35% of the main diagonal is colored, representing top-1 accurate predictions, with the remaining predictions appearing largely random.
• Model confidence differs by outcome: For correctly classified samples the model shows confidence over 90%, while for misclassified samples it is usually "in doubt" between multiple classes, with the highest scoring ones averaging about 50%. The authors interpret this as some signs having distinctive features while many classes share overlapping features.

Methodology in Plain English

The authors assembled videos from three open-access Romanian sign language sources and manually removed duplicates. They then cleaned the label vocabulary in three ways: merging spelling variants of the same sign into one canonical form (for example "ruta-1", "ruta-2", "ruta-3" became "ruta"), discarding sentence-like labels with more than three hyphens, and reconciling near-duplicate labels using a 0.95 similarity threshold. Because most classes had only one video, they restricted benchmarking to the 1,926 classes with at least two examples, keeping n-1 videos for training and one for testing per class, giving roughly an 80/20 split.

All videos were normalized to 224x224 pixels, 25 frames per second, and 64 frames using uniform temporal sampling, and metadata files mapping glosses to classes and providing train/test annotations were generated automatically. Training and evaluation used the MMAction2 framework. Every RGB model started from weights pretrained on ImageNet, Kinetics-400, or Kinetics-710, with the classification head resized to the 1,926-class vocabulary and the maximum number of training epochs set to 125. PoseConv3D was the only keypoint-based model, taking 133 keypoints covering body, face, hands, and feet extracted with MMPose; the authors chose this large keypoint set to capture mouthing and facial expression as well as hand shape.

Why This Matters

Impact on research. RoCoISLR fills a documented gap: no standardized dataset existed for Romanian Isolated Sign Language Recognition, which blocked training, evaluation, reproducibility, and cross-lingual comparison. The paper supplies both the corpus and a consistent baseline, and its results make the low-resource, long-tail problem visible in a way that single-language benchmarks on English-centric data do not.

Real-world applications.

• Assistive tools that recognize isolated Romanian signs from video and convert them to text, helping Deaf signers communicate with hearing people.
• Educational and healthcare settings where Romanian sign vocabulary lookup or translation support could assist interpreters and service providers.
• Automated dataset and annotation pipelines for other under-resourced sign languages, reusing the standardization approach.
• Comparative evaluation of video recognition architectures on languages with limited training data, informing model selection in low-resource conditions.

Industry relevance. The benchmark shows that even state-of-the-art pretrained video models degrade sharply when transferred to a low-resource sign language, reaching only 34.10% Top-1 at best. That is a warning for companies building sign language products: models must be trained or adapted per language and per recording setup, and domain adaptation is a practical requirement rather than an optional refinement.

Future Directions

1. Develop stronger domain adaptation methods to close the gap between training accuracy and test accuracy, which the authors attribute to models potentially overfitting to particular visual appearances of specific signers or cameras.
2. Investigate multimodal techniques that use body posture and facial expressions alongside hand information.
3. Expand the dataset with more signers from diverse backgrounds and varied recording setups.
4. Apply large language models to complement gloss-to-text translation.
5. Collect missing metadata such as age and right-handedness, which the paper states is currently unavailable but planned for future inclusion.

Target Audience

Researchers in computer vision and sign language processing, especially those working on low-resource languages and isolated sign recognition; dataset builders who want a template for merging and standardizing scattered corpora; practitioners selecting video recognition architectures for small-data regimes; and accessibility engineers who need to understand the accuracy limits of current models on Romanian sign language.

Authors’ abstract

Automatic sign language recognition plays a crucial role in bridging the communication gap between deaf communities and hearing individuals; however, most available datasets focus on American Sign Language. For Romanian Isolated Sign Language Recognition (RoISLR), no large-scale, standardized dataset exists, which limits research progress. In this work, we introduce a new corpus for RoISLR, named RoCoISLR, comprising over 9,000 video samples that span nearly 6,000 standardized glosses from multiple sources. We establish benchmark results by evaluating seven state-of-the-art video recognition models-I3D, SlowFast, Swin Transformer, TimeSformer, Uniformer, VideoMAE, and PoseConv3D-under consistent experimental setups, and compare their performance with that of the widely used WLASL2000 corpus. According to the results, transformer-based architectures outperform convolutional baselines; Swin Transformer achieved a Top-1 accuracy of 34.1%. Our benchmarks highlight the challenges associated with long-tail class distributions in low-resource sign languages, and RoCoISLR provides the initial foundation for systematic RoISLR research.