GLOSS-MT: Controllable Low-Resource Machine Translation with an Editable Dictionary

AI summary

Overview

Research area: Natural Language Processing — low-resource machine translation, terminology control, and parameter-efficient fine-tuning.

Technical level: Intermediate. The pipeline is conceptually simple (dictionary lookup plus in-place substitution), but the evaluation uses standard MT metrics (chrF2, BLEU, sacreBLEU) and the model set includes LoRA and full fine-tuning of NLLB-200, OPUS-MT and TranslateGemma.

Scope: The paper asks whether a single fine-tune on dictionary-pretranslated input turns an editable English–Twi dictionary into a control surface for pretrained MT models, and whether this can be done without degrading translation quality.

What This Paper Is About

Adapting a pretrained translation model to new vocabulary normally means collecting new parallel data and retraining, which is expensive for a language like Twi (Akan) that has little parallel text. The authors propose Gloss-MT, a pipeline that looks up English noun phrases in an offline English–Twi dictionary, substitutes the Twi translation directly into the English source inside angle brackets, and fine-tunes a model once to translate these mixed inputs. The goal is for an operator to change terminology later by editing a wordlist rather than retraining, while the model still translates at least as well as the pretrained base it started from.

Key Contributions

1. The Gloss-MT pipeline and its training set. A dictionary of 613,259 English→Twi phrases is applied in place to source noun phrases via spaCy extraction, and a training set of 31,909 line-aligned chunks is built from 15,134 English–Twi documents by having Gemini align English noun phrases to verbatim spans in the gold Twi. The data is released as gloss-mt-parallel-en-twi.

2. Five fine-tunes across three model families, each compared to its own base. NLLB-200 600M, 1.3B and 3.3B, OPUS-MT en-tw (~75M parameters) and TranslateGemma 4B are each evaluated on an independent document-level benchmark (84 documents, four domains) against their pretrained counterpart.

3. A direct controllability test with a counterfactual dictionary. On 300 passages of three consecutive Bible verses (2,766 inserted terms) with human Twi references, each model translates three times — with the dictionary, without it, and with a "swap" dictionary in which every entry is replaced by a different real Twi term — measuring adherence, natural rate, swap adherence and leak.

4. Released models, data and code (Hugging Face ghanaopenai; GitHub GhanaOpenAI/gloss-mt; benchmark at GhanaNLP/nsanku-mt-benchmark).

Main Findings

• Every fine-tune beats its own base in chrF2. On the nsanku benchmark, gains range from +2.85 to +5.48: NLLB-200 3.3B 47.74 → 50.80 (+3.06), NLLB-200 1.3B 47.42 → 50.27 (+2.85), NLLB-200 600M 46.71 → 50.12 (+3.41), OPUS-MT 42.52 → 48.00 (+5.48). TranslateGemma 4B reaches 49.81 with no base row, because zero-shot it does not produce Twi.

• Gains concentrate in contemporary prose. Watchtower rises 5.7–7.3 points for the NLLB models and Finance 3.7–5.7. The only domain below its base is UDHR for NLLB 3.3B (−0.39), and the 1930 folktales in pre-standard orthography gain least.

• The three NLLB fine-tunes are the best open systems in the comparison (50.12–50.80), behind only commercial Gemini 3.8 Flash (54.64) and Google Translate (51.26), and ahead of the open baseline MADLAD-400 3B (37.14).

• Model size matters little once fine-tuned. The fine-tuned NLLB models span only 50.12–50.80 from 600M to 3.3B, and the 600M fine-tune scores 2.38 above the base 3.3B model. OPUS-MT at ~75M parameters (48.00) scores above the base NLLB 3.3B (47.74).

• There is a small BLEU cost for NLLB. NLLB fine-tunes fall 0.51 to 1.04 BLEU below their bases, mostly on UDHR and Watchtower, even as chrF2 rises on both; the authors report this as a limit on the no-degradation claim. OPUS-MT gains BLEU (+1.20).

• The dictionary controls the fine-tuned models without retraining. On the human-referenced Bible set, Gloss-MT TranslateGemma and NLLB 3.3B write 96.6% and 94.4% of inserted terms (26.2% and 38.4% of the same terms appear without the dictionary), and after swapping every entry they write 97.3% and 95.2% of the new terms, with only 1.9% and 2.2% of the original terms surviving.

• Fine-tuning is what creates the control. Base OPUS-MT essentially ignores bracketed terms (1.9% adherence) and degrades when they are present (chrF2 30.1 → 19.1 with the dictionary). Base NLLB 3.3B already copies some Twi from its input (66.0%), and fine-tuning raises this to 94.4%.

• The smallest model follows terms least reliably. Gloss-MT OPUS-MT adherence is 79.2% on the Bible and swap adherence 69.7%, versus 95–97% for the larger fine-tunes; the paper attributes this to the swapped terms often being wrong in context.

• No quality cost on human references. On the Bible, translating with the dictionary scores at least as well in chrF2 as without it: TranslateGemma +5.0, OPUS-MT +2.8, NLLB 3.3B +0.3 (BLEU 10.75 vs 10.91) — even though only 35.9% of the inserted terms occur in the human reference.

• Validation scores are a weak proxy for benchmark quality. TranslateGemma beat NLLB 3.3B on validation chrF2 (64.96 vs 63.66) but fell below it on the benchmark, and the 600M model beat the 1.3B on validation (63.99 vs 62.14) but was slightly below it on the benchmark. On validation chunks, the dictionary lowers chrF2 for every fine-tune (−1.7 to −5.6), which the authors read as a property of those references rather than a quality loss.

Methodology in Plain English

The approach has three moving parts.

An offline dictionary. 613,259 English phrases, each with exactly one Twi translation, drawn from machine-translated phrase lists (re-translated with Gemini where an entry had several candidates) and from alignments in the training data.

A substitution step at inference. Documents are split into lines and packed into chunks of at most 192 source tokens (128 for OPUS-MT), cut only at line boundaries so Markdown structure survives. spaCy extracts the chunk's noun phrases, they are looked up in the dictionary, and every phrase found is replaced in the English, longest first, by its Twi translation in angle brackets. The model reproduces the bracketed terms and the pipeline strips the brackets afterwards. Unfound phrases stay English, so the model receives a mixed English–Twi input.

One fine-tune. For training, Gemini aligns each English noun phrase of a chunk to its span in the reference Twi, keeping a span only if it occurs verbatim on the phrase's line; both input and target then carry the bracketed span, so the model learns to copy the terms and translate around them. NLLB-200 3.3B is fine-tuned with LoRA (r=32 on all attention and feed-forward projections, 2% of parameters, merged after training); NLLB-200 1.3B and 600M with full fine-tuning (Adafactor, lr 10⁻⁴); OPUS-MT en-tw with full fine-tuning (AdamW, lr 10⁻⁴); and TranslateGemma 4B with LoRA (r=32) using its own prompt with target code ak. A literal new-line token is added to the NLLB tokenizer. Every 1,000 steps the model translates 200 fixed validation chunks, and the checkpoint with the best mean of BLEU, chrF2 and 100·e^(−loss) is kept, with early stopping.

How control is measured. Each test set is translated three times by the same model, changing only the dictionary: dict (as deployed), none (no dictionary) and swap (a counterfactual dictionary where every phrase found receives a different real Twi term chosen so it occurs neither in the source nor in the dict or none outputs — so a swapped term can only reach the output through the dictionary). A term counts as used when its words appear in order as whole words. The benchmark reports document-level chrF2 and BLEU over 84 documents in four datasets — three Citizens' Budget documents (Finance 2021–2023), UDHR, five Watchtower study editions (2026), and 75 Akan folktales (Rattray, 1930) — with the language score as the mean of the four per-dataset scores. Because 75 documents are folktales, that averaging matters.

Why This Matters

For research: The paper reframes terminology adaptation from a data problem to an interface problem. It shows that in-place pretranslation, previously explored with a handful of per-sentence constraints, scales to a large community-style dictionary and can be learned by a single fine-tune — and it isolates the fine-tune as the step that creates controllability, since base OPUS-MT ignores bracketed terms entirely. It also offers a reusable evaluation design (swap dictionary, leak rate) for measuring whether a model actually routes through an external resource.

Real-world applications:

• Government and civic translation, where ministry budget terms, programme names and legal vocabulary must be rendered consistently — the Finance domain shows the largest gains alongside Watchtower.
• Community and language-committee maintenance of terminology: a wordlist is cheaper to produce and update than parallel corpora, and the paper argues communities can maintain translation by editing wordlists rather than collecting data.
• Domain or dialect preference switching without retraining, since replacing every dictionary entry changes the output as intended.
• Deployment on modest hardware: the 600M fine-tune scores 2.38 above the base 3.3B model, making the cheapest model a practical choice.

Industry relevance: The result undercuts the assumption that terminology control requires either slow lexically constrained decoding or a fresh fine-tune per term set. For organizations serving low-resource languages, it suggests a maintenance model where the model is trained once and terminology is a configuration artifact.

Future Directions

• Enforcing rather than learning control. Adherence is high but not complete and lowest for the smallest model; the authors note control is learned, not guaranteed.
• Handling morphology. Term matching is by exact whole words, so a term altered by a prefix or plural counts as missed and adherence is a lower bound; target-lemma annotation is suggested as a remedy for inflection.
• Improving the dictionary and measuring domain gains. The dictionary is largely machine-generated and its entries often differ from human references' choices; the authors state they have not yet measured the quality gain from adding curated terminology for a new domain.
• Broadening evaluation beyond one language pair and one control domain. All evaluation is English→Asante Twi, and the human-referenced controllability set is a single domain (the Bible), translated freely on both sides; the NLLB BLEU loss also remains unexplained.

Target Audience

Researchers and practitioners working on low-resource or multilingual MT, especially those adapting pretrained models such as NLLB-200 or OPUS-MT to new domains; NLP engineers who need controllable terminology without retraining; and language-technology teams and community organizations for Akan/Twi who want to maintain terminology through editable wordlists. Readers without MT background will follow the pipeline and the controllability results but will need familiarity with chrF2 and BLEU to interpret the benchmark table.

Authors’ abstract

Adapting a pretrained machine translation (MT) model to new terminology usually requires new parallel data and another round of fine-tuning, which is costly for low-resource languages. We ask whether controllability can be added to MT, and the fine-tuning burden reduced, without degrading the pretrained model. GLOSS-MT looks up the source's noun phrases in an offline English-Twi dictionary, replaces every phrase found by its Twi translation in place, and fine-tunes a translation model once to translate these mixed inputs. We apply the same data and pipeline to NLLB-200 (600M, 1.3B, 3.3B), OPUS-MT en-tw and TranslateGemma 4B. On a document-level English-to-Asante Twi benchmark (84 documents, four domains), every fine-tune scores above its pretrained base in chrF2 (+2.85 to +5.48), and the three NLLB fine-tunes are the best open systems (50.12-50.80). A direct controllability test on human-translated Bible passages shows that the three fine-tunes tested write 79-97% of the dictionary terms, and that replacing every dictionary entry by a different term changes the output accordingly (70-97% of the new terms appear) with no retraining. We release the models, data and code.