DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation

AI summary

Overview

Research area: Natural language processing and computational sign language processing, specifically text-to-sign-language gloss translation, combining ASL linguistics, large language models, and machine translation evaluation.

Technical level: Intermediate. The paper is readable for someone familiar with LLM prompting pipelines and standard MT metrics, though it assumes no prior background in sign language linguistics; the framework's formal notation is modest and the modules are described conceptually.

Scope: The paper introduces DiscoSign, a framework that translates multi-sentence English text into American Sign Language (ASL) glosses while maintaining discourse-level coherence across sentences, plus a set of novel evaluation metrics for measuring that coherence.

What This Paper Is About

Existing text-to-sign-language systems translate one sentence at a time, so they have no way to keep track of entities, word-to-sign mappings, or rhetorical structures across a multi-sentence passage. This causes concrete errors: a character assigned to one spatial location can be reassigned later, the same English concept can be rendered with different ASL signs, and question-answer clause structures appear where they do not belong. The goal of DiscoSign is to translate whole passages rather than isolated sentences, using explicit discourse state that is carried forward from sentence to sentence.

Key Contributions

1. Discourse-aware text-to-gloss translation framework: The first systematic approach to sign language gloss translation that maintains cross-sentence coherence through explicit state registries and programmatic constraint enforcement, covering spatial coreference resolution, pseudocleft (QAC) usage, and concept-gloss consistency.
2. ASL-centric instruction design methodology: A systematic method for embedding ASL linguistic expertise into LLM prompts, showing how specific discourse phenomena can be handled through careful instruction design rather than model retraining.
3. Comprehensive evaluation framework: Novel metrics for discourse-level text-to-gloss translation, including Spatial Coreference Accuracy (SCA), Question-Answer Clause Appropriateness (QAC_Ap), and Concept-Gloss Consistency (CGC), each tied to one module of the framework.
4. Systematic instruction component analysis: Ablation studies isolating the contribution of each instruction component, plus a demonstration that the gains hold for both a proprietary and an open-weight LLM backbone.

Main Findings

• Traditional MT metrics do not detect discourse quality: On Aesop's Fables, all three Gemini-based configurations scored comparably on chrF (39.4–41.7) and COMET (0.76–0.78) despite very different discourse coherence, showing that these metrics cannot capture spatial consistency, coreference, or pseudocleft structure.
• Explicit modeling beats context access alone: Going from sentence-level to context-aware improved SCA from 0.29 to 0.48, CGC from 0.59 to 0.68, and QAC_Ap from 0.70 to 0.72, but the full framework reached SCA 0.84, CGC 0.97, and QAC_Ap 0.76. Access to prior sentences is helpful but insufficient.
• Statistical significance holds for most comparisons: With Gemini 2.5 Pro, all differences between the Proposed framework and both baselines are statistically significant (p<0.05, paired bootstrap, 10k samples), except QAC_Ap versus the context-aware baseline (p=0.066) and COMET versus the sentence-level baseline (p=0.345).
• The framework generalizes to an open-weight backbone: Repeating the Aesop's Fables experiment with Qwen3.6-35B-A3B, an open-weight mixture-of-experts model (35B total, 3B active parameters), preserved the ordering of conditions: the framework again scored highest on discourse metrics, improving SCA by +0.49 and CGC by +0.13 over the sentence-level baseline, though absolute scores were lower, most visibly on SCA.
• Sentence-level quality is not sacrificed: On ASL STEM Wiki, the proposed framework scored 30.1 chrF and 0.51 COMET on gloss-level evaluation versus 28.2 and 0.54 for the sentence-level baseline. On the licensed dataset, 39.1 and 0.57 versus 39.6 and 0.59. Back-translation quality improved consistently (ASL STEM Wiki chrF 54.8 vs 41.9, COMET 0.81 vs 0.73; licensed dataset chrF 67.2 vs 65.2, COMET 0.90 vs 0.88).
• Each module is responsible for its own metric: Disabling SCM dropped SCA from 0.81 to 0.71; disabling CGCM dropped CGC from 0.97 to 0.64; disabling QACM dropped QAC_Ap from 0.76 to 0.71–0.72. The authors note that QACM's effect on translation metrics is mostly stylistic.
• Structured modeling can substitute for context quantity: The framework outperformed the sentence-level baseline even with no context (SCA 0.45 vs 0.29), and with just one prior sentence it exceeded the context-aware baseline that had full access (SCA 0.57 vs 0.48). Increasing the context window improved all discourse metrics (SCA 0.40 → 0.84; CGC 0.78 → 0.97) while chrF stayed around 42 and COMET around 0.78; QAC_Ap plateaued at two sentences (0.76).
• Human evaluation supports the metrics, with a caveat for QACs: Two ASL-fluent evaluators rated 32 stories (152 sentences) on a 1–5 Likert scale. SCA showed the strongest validation (ρ=0.50, p=0.005 and ρ=0.42, p=0.022 individually; combined ρ=0.52, p=0.003; inter-rater weighted κ=0.43, ρ=0.49). CGC correlated significantly (ρ=0.48, p=0.007), with evaluators trending the same way (ρ=0.48, p=0.007 and ρ=0.24, p=0.195). QAC_Ap correlations were non-significant, which the authors attribute to rater disagreement about what constitutes a QAC rather than a metric failure; QAC usage agreement with human judgments was strong for one evaluator (ρ=0.85, p<0.001) and moderate for the other (ρ=0.39, p=0.035).

Methodology in Plain English

The system translates a text one sentence at a time, but keeps a running memory of the discourse in three registries: one for which entity lives at which spatial index, one for which English concept maps to which ASL gloss, and one for whether a question-answer clause was used in each sentence. For each new sentence, the system retrieves those registries, builds a single prompt that contains the sentence, a window of previous sentence-translation pairs, and the registry contents stated as hard constraints, then receives back a JSON response with the gloss translation plus structured metadata such as spatial mappings, concept mappings, and a QAC flag. A deterministic post-processor then checks the output against the registries and corrects violations by string replacement, without querying the model again, and the verified metadata updates the registries for the next sentence. The three modules are therefore not separate model calls but structured prompt sections with their own verification logic inside one LLM call per sentence. Gloss output is constrained to the ASLLRP SignBank lexicon (2,859 glosses) and sampling is deterministic (temperature=0). Evaluation uses the ASL STEM Wiki corpus (500 examples, one sentence each, gold glosses available), a licensed dataset of 2,121 examples of everyday activities with ASL glosses collected and annotated by a team of Deaf signers fluent in ASL and English (one sentence each, gold glosses available), and Aesop's Fables (284 examples, 5 sentences each, no gold glosses, containing coreference chains, causal relationships, and topic transitions). For discourse-level evaluation the English coreference chains required by SCA were extracted with Gemini 2.5 Pro and validated by a human expert, and the structured outputs produced during translation were verified by a human expert for faithfulness.

Why This Matters

Impact on research: The paper reframes text-to-sign-language translation as a discourse-level task rather than a sentence-level one, and argues that standard MT metrics are structurally unable to measure the qualities that matter for signed languages. It provides both a framework and a matching evaluation suite, which the authors describe as the first systematic treatment of discourse-level text-to-gloss translation.

Real-world applications:

• Sign language dictionaries and lexicon-constrained gloss production: The framework's vocabulary compliance requirement (all glosses must come from the sign dictionary) means its output can be mapped directly to sign dictionaries for downstream sign production.
• Signing avatars and visual sign generation: The paper situates its work alongside avatar and production-side research, since glosses are an intermediate representation that feeds a sign production stage.
• Dialogue and conversational accessibility systems: The authors point to prior work on embedding sign language processing into large-scale dialogue systems, an application where cross-sentence coherence is unavoidable.
• Educational and technical content: ASL STEM Wiki is included precisely because scientific terminology requires substantial fingerspelling, indicating a use case in accessible technical education.

Industry relevance: The work is a collaboration involving Apple, Northeastern University, and Gallaudet University, and tests a proprietary backbone (Gemini 2.5 Pro) alongside an open-weight mixture-of-experts model (Qwen3.6-35B-A3B). The framework is described as LLM-agnostic by construction, requiring only an instruction-following model that can produce the specified JSON schema, and the authors report that the discourse gains hold on both backbones.

Future Directions

• Replacing the LLM-based modules: The limitations section notes that SCM, QACM, and CGCM could instead be instantiated through rule-based systems, trained neural models, or human experts, and that the modular design makes this possible depending on application requirements.
• Building a reference-annotated discourse-level dataset: Aesop's Fables has no ground truth ASL annotations, so discourse-level evaluation currently relies on automatic metrics and back-translation. The authors regard building such a dataset as a substantial annotation contribution in its own right.
• Extending beyond English-to-ASL: The linguistic phenomena considered are described as present across signed languages, not specific to ASL, so extending the framework is an open direction.
• Incorporating non-manual markers (NMMs): The framework addresses manual sign production through glosses and does not incorporate facial expressions or prosody. The authors note that the registry state each module maintains is closely related to what a downstream visual generation system would need to produce NMMs, and framing NMM annotation from that state as a natural next step.
• Modeling QAC optionality: The QACM enforces QAC decisions through deterministic trigger rules, which the authors state cannot represent contexts where both a QAC and a non-QAC rendering are natural; learned alternatives are discussed in the paper's Appendix B.

Target Audience

Researchers and practitioners in sign language processing, machine translation, and accessibility technology, particularly those working on text-to-gloss translation, sign language generation, or evaluation methods for low-resource and visual-spatial languages. It is also relevant to ASL linguists interested in how discourse phenomena such as spatial indexing, question-answer clauses, and concept-gloss consistency can be operationalized computationally, and to NLP engineers building discourse-aware LLM pipelines with explicit state registries and deterministic post-processing.

Authors’ abstract

Sign language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and (iii) concept-gloss consistency, ensuring stable mappings between English concepts and American Sign Language (ASL) signs. Traditional translation metrics fail to capture discourse-level quality, so we introduce a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by our framework. Experiments on sentence-level and discourse-level datasets show that our approach for discourse-aware processing significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality. Our work establishes the first systematic framework for discourse-level text to sign language gloss translation with corresponding evaluation methodology.