Introducing A Bangla Sentence - Gloss Pair Dataset for Bangla Sign Language Translation and Research

AI summary

Overview

• Research area: Natural Language Processing / sign language translation, specifically Bangla (Bengali) text-to-gloss translation, a low-resource subfield of sign language processing.
• Technical level: Intermediate. The paper assumes familiarity with transformer encoder-decoder models, fine-tuning, BLEU/COMET evaluation, and retrieval-augmented generation, but explains the dataset construction and augmentation steps in accessible terms.
• Scope: The paper introduces Bangla-SGP, a 4,000-entry Bangla sentence-gloss parallel dataset (1,000 human-annotated plus roughly 3,000 synthetically augmented), and benchmarks four transformer-based models on sentence-to-gloss translation with BLEU and COMET scores.

What This Paper Is About

Bangla Sign Language (BdSL) research has been stuck at the word and alphabet level because no large-scale dataset offers sentence-level text-to-gloss translation. The authors build that missing resource: 1,000 professionally annotated Bangla sentence-gloss pairs, expanded to 4,000 entries through three augmentation strategies, including a rule-based Retrieval-Augmented Generation (RAG) pipeline. They then fine-tune and compare mT5-small, mBART-50, NLLB-200-1.3B, and GPT-4.1-nano to see whether the augmented data improves gloss translation.

Key Contributions

1. Bangla-SGP dataset. A parallel corpus of 1,000 human-annotated Bangla sentence-gloss pairs, augmented with roughly 3,000 synthetically generated pairs, yielding a final dataset of 4,000 entries at a 1:3 ratio of original to augmented data. Glosses were annotated by Mr. Ariful Islam, an experienced Bangla sign language presenter at BTV, using sentences drawn from Bangla newspaper corpora, Bangla literature, and a BdSL language development book.
2. A rule-based RAG pipeline for gloss generation. The pipeline embeds the training split with LaBSE, stores it in the Pinecone vector database, retrieves contextually similar annotated examples at a similarity threshold of 0.5 (capped at 20 examples per prompt), and uses GPT-4.1-nano with two-stage prompting to generate glosses.
3. Validation of synthetic data with professional signers. Two certified BdSL sign presenters independently reviewed 150 randomly selected samples (15% of the generated pairs) and rated them on comprehensibility and a 1-5 quality scale, with agreement measured via Cohen's kappa.
4. Benchmarking of four transformer models. mT5-small, mBART-50, NLLB-200-1.3B, and GPT-4.1-nano were fine-tuned and evaluated with BLEU-1 through BLEU-4 and COMET, and the results are compared against the RWTH-PHOENIX-2014T German Sign Language benchmark.

Main Findings

• mBART-50 performed best overall. On the augmented 4K dataset it reached BLEU-1 61.09, BLEU-2 46.01, BLEU-3 35.09, BLEU-4 27.31, and COMET 0.8261. On the base 1K dataset it scored BLEU-4 21.06 and COMET 0.7829, so augmentation improved every BLEU metric.
• mT5-small improved with augmentation. BLEU-4 rose from 17.56 on the 1K dataset to 20.28 on the 4K dataset, with COMET going from 0.8564 to 0.8602. Its BLEU-1 went from 51.79 to 55.33.
• GPT-4.1-nano was competitive on the augmented data. It achieved BLEU-4 22.42 and the highest COMET among augmented-data runs at 0.8913.
• NLLB-200-1.3B produced the highest single COMET score but was only evaluated on the base data. It scored COMET 0.907 and BLEU-1 67.99 on the 1K manually annotated dataset, but computational constraints prevented evaluation on the augmented dataset.
• RAG-generated glosses were judged moderately reliable. The combined validation rate was 75.3% (74.7% for Signer 1, 76.0% for Signer 2), with an average quality score of 3.19 out of 5 (2.96 and 3.41 respectively).
• Quality ratings were mixed. 42.7% of glosses were rated high quality (score at least 4), 24.3% acceptable (score 3), and 33.0% low quality (score at most 2). Signer 1 rated 35.3% high quality and 38.7% low quality; Signer 2 rated 50.0% high quality and 27.3% low quality.
• Inter-annotator agreement differed by task. Binary agreement was substantial at kappa = 0.7489, while agreement on the 1-5 quality ratings was only fair at kappa = 0.3496.
• Comparison with RWTH-PHOENIX-2014T favored the new dataset under matched settings. mBART-50 scored BLEU-4 21.49 and COMET 0.6673 on RWTH-PHOENIX-2014T versus BLEU-4 27.31 and COMET 0.8261 on the augmented Bangla dataset; mT5-small scored BLEU-4 18.73 and COMET 0.6205 on RWTH-PHOENIX-2014T versus BLEU-4 20.28 and COMET 0.8602 on the augmented Bangla dataset. The authors explicitly state direct comparison is not meaningful due to language and domain differences.
• Masked-token substitution and tense rules each contributed about 500 pairs. Rule-based morphological transformation produced around 500 text-gloss pairs and masked-token substitution, using Bangla-BERT predictions for masked common nouns and verbs, produced approximately 500 more. The RAG approach is reported as generating 2,000 pairs in the augmentation section, while the validation section describes first generating 1,000 pairs via RAG before sampling 150 for review.

Methodology in Plain English

The authors start with a real, professionally annotated dataset: 1,000 Bangla sentences paired with their gloss sequences, where glosses are Bangla sign-supported words acting as an intermediate textual representation of a continuous sign. That alone is too small to train transformer models well, so they expand it three ways.

First, they worked with a professional signer to extract recurring translation rules, most of them tied to verb tense, and used those rules to rewrite existing sentences into other tenses, producing about 500 new pairs. Second, they masked certain words in sentences and let Bangla-BERT predict contextually appropriate replacements, producing roughly 500 more grammatically consistent pairs. Third, they built a retrieval-augmented pipeline: they split the 1,000 annotated pairs 80/10/10 into training, test, and validation, embedded the training split with LaBSE, and stored it in Pinecone. When a new Bangla sentence arrives from the Bangla Corpus by Nuhash Afnan, the system retrieves up to 20 similar annotated examples (similarity threshold 0.5) and passes them, or the relevant tense rules as a fallback, to GPT-4.1-nano as a few-shot prompt. A two-stage prompt first asks the model to identify the sentence's tense and then supplies only that tense's rules, an approach the authors compare to chain-of-thought prompting.

To check the synthetic output, two professional sign presenters independently reviewed 150 samples, answering a yes/no comprehensibility question and giving a 1-5 quality rating. The authors then fine-tuned mT5-small (batch size 16, learning rate 0.001, warm-up steps 50, 20 epochs with early stopping, AdamW), mBART-50 (effective batch size 16, learning rate up to 3e-5, warm-up steps 300, 20 epochs with early stopping), NLLB-200-1.3B using LoRA (rank 8, alpha 16, dropout 0.05 on the q_proj, k_proj, v_proj, out_proj, fc1, and fc2 modules, batch size 1 with 24 gradient accumulation steps, learning rate 3e-4, 5 epochs, FP16 mixed precision), and GPT-4.1-nano. All models were scored with BLEU-1 to BLEU-4 and COMET, with the same test set used for both the base and augmented datasets.

Why This Matters

Impact on research. Continuous sign language recognition and translation for BdSL has been blocked by the absence of publicly available resources. Bangla-SGP provides the first sentence-level Bangla sentence-gloss resource described here, plus a reusable RAG-based augmentation recipe that other low-resource sign language projects could adapt. The authors also offer an empirical demonstration that augmentation helps consistently across model families, and they set a comparison point against RWTH-PHOENIX-2014T.

Real-world applications.

• Educational tools that convert written Bangla text into gloss sequences to support deaf students, who often lack the phonological associations hearing readers use.
• Virtual interpreters that turn spoken or written Bangla into sign-language representations in animated form.
• Input to a future 3D animation pipeline that maps glosses to isolated sign videos and reconstructs continuous signed sentences as animated meshes.
• Accessibility layers for text- and voice-heavy digital content that deaf Bangla speakers currently cannot easily consume.

The dataset serves roughly 13 million people in Bangladesh with hearing difficulties and 3 million who are deaf, per the figures cited in the paper.

Industry relevance. The work targets a low-resource language market that commercial translation systems largely ignore, and it shows how to stretch limited expert annotation with a RAG pipeline rather than expensive manual labeling. The NLLB-200 LoRA setup demonstrates that billion-parameter multilingual models can be adapted to a small dataset under VRAM constraints, which matters for organizations without large GPU budgets.

Future Directions

1. Phase 2 multimodal data collection. The authors plan to add synchronized sign video samples for the Bangla sentences and annotations for non-manual markers such as facial expressions, head movements, and torso movements, gathered from multiple signers with inter-annotator agreement checks.
2. A 3D sign language generation pipeline. The plan is to build a gloss video dictionary mapping isolated glosses to RGB videos, use architectures such as SMPLer-X and SMPL-X to reconstruct 3D human meshes frame by frame as obj files, and chain them into continuous animation. The authors report that a preliminary attempt failed due to the lack of a comprehensive sign video dataset and unstable transitions between consecutive signs.
3. Closing the augmentation evaluation gap. NLLB-200-1.3B was never evaluated on the augmented dataset because of computational constraints, so whether augmentation would improve its scores remains unverified.
4. Expanding coverage of vocabulary gaps. Words without a dedicated sign, such as names and landmarks that are fingerspelled at the alphabet level, and compound breakdowns such as "shoptaho" (week) rendered as "shat din" (seven days), are not well represented and need more examples.

Target Audience

Researchers and graduate students working on sign language processing, low-resource machine translation, or NLP for South Asian languages will get the most from this paper, particularly those interested in data augmentation, retrieval-augmented generation, or parameter-efficient fine-tuning of large multilingual models. Practitioners building accessibility tools for deaf communities, and dataset curators who need a worked example of validating synthetic data with professional signers via Cohen's kappa, are secondary audiences.

Authors’ abstract

Bangla Sign Language (BdSL) translation represents a low-resource NLP task due to the lack of large-scale datasets that address sentence-level translation. Correspondingly, existing research in this field has been limited to word and alphabet level detection. In this work, we introduce Bangla-SGP, a novel parallel dataset consisting of 1,000 human-annotated sentence-gloss pairs which was augmented with around 3,000 synthetically generated pairs using syntactic and morphological rules through a rule-based Retrieval-Augmented Generation (RAG) pipeline. The gloss sequences of the spoken Bangla sentences are made up of individual glosses which are Bangla sign supported words and serve as an intermediate representation for a continuous sign. Our dataset consists of 1000 high quality Bangla sentences that are manually annotated into a gloss sequence by a professional signer. The augmentation process incorporates rule-based linguistic strategies and prompt engineering techniques that we have adopted by critically analyzing our human annotated sentence-gloss pairs and by working closely with our professional signer. Furthermore, we fine-tune several transformer-based models such as mBart50, Google mT5, GPT4.1-nano and evaluate their sentence-to-gloss translation performance using BLEU scores, based on these evaluation metrics we compare the model's gloss-translation consistency across our dataset and the RWTH-PHOENIX-2014T benchmark.