Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

AI summary

Overview

Research area: Natural Language Processing — diacritic restoration (also called diacritic recovery), evaluated on Romanian text.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting concepts (zero-shot, few-shot), standard accuracy/error metrics, and Levenshtein distance, but the setup is easy to follow.

Scope: A comparative benchmark of twelve large language models plus one mock baseline across multiple prompt templates, measuring how well each restores Romanian diacritics that have been stripped from input text.

What This Paper Is About

Romanian uses five diacritical marks (ă, î, ș, ț, â) that are frequently dropped in digital text because of keyboard limitations and data-entry errors, which introduces ambiguity and degrades downstream text processing. This paper asks whether modern large language models can put those marks back correctly when given diacritics-stripped text, and whether more elaborate prompts (with worked examples) help. The authors test a wide range of proprietary and open-source models against a deliberately trivial "Echo" baseline that simply returns the input unchanged, so that any claimed accuracy can be judged against what doing nothing already achieves.

Key Contributions

1. A comparative evaluation of twelve LLMs plus a mock baseline for Romanian diacritic restoration, covering OpenAI's GPT-3.5, GPT-4, GPT-4 Turbo, and GPT-4o; Google's Gemini 1.0 Pro; Meta's Llama 2 (7B and 70B) and Llama 3 (8B and 70B); MistralAI's Mixtral 8x7B Instruct; airoboros 70B; and OpenLLM-Ro's RoLlama 2 7B.

2. A staged prompt-template suite that escalates from a bare zero-shot instruction to verbose zero-shot, one-shot, two-shot, and three-shot variants, letting the authors measure how much performance comes from prompt engineering rather than the model itself.

3. A neutral "Echo" baseline and the Relative Performance Ratio (RPR), a metric that divides a model's Maximum Total Average Score (MTAS) by the Echo model's MTAS, exposing models that look good in isolation but do not beat copying the input.

4. A systematic error analysis on the DLRLC-1000 split, using a public error_stats.py module that logs diacritic-level confusion counts, context-aware positional statistics, and model-level over- or under-generation rates.

Main Findings

• OpenAI models lead, but not by doing nothing. GPT-4o achieved the best Total Average Score (TAS) of 0.9639 with the restore_diacritics_verbose_3s-240331_1858.md (3-shot) template, which the paper states is 19% higher than the mock Summa baseline score of 0.8100.

• The peak single configuration was very high. The highest Average Performance Score (APS) reported was approximately 0.9946, reached with the 3-shot restore_diacritics_verbose_3s-240331_1858.md template, the RER_CI_CL evaluator, and the CRAWLER-1000 dataset.

• Three-shot prompting won nearly everywhere. The 3-shot prompt consistently secured the highest scores across all models except those from Meta.

• Many open models failed to beat the baseline. Table 4 reports RPR values below 1.000 for Llama 3 8B (MTAS 0.7663, RPR 0.946), RoLlama 2 7B (0.6463, 0.798), Mixtral 8X7B (0.608, 0.751), Llama 2 70B (0.146, 0.180), and Llama 2 7B (0.002, 0.002). The paper text separately reports MTAS figures of 0.6463, 0.6079, 0.7663, 0.2501, and 0.6008 for RoLlama 2 7B, Mixtral 8x7B, Llama 3 8B, Llama 2 7B, and Llama 2 70B respectively — these differ from the values in Table 4 for Llama 2 7B and Llama 2 70B, and the paper does not reconcile the discrepancy.

• airoboros 70B barely cleared the bar. Its MTAS of 0.8289 on the 3-shot template was only 2.33% higher than the baseline.

• Gemini 1.0 Pro broke OpenAI's ranking dominance. Per Table 4, it recorded MTAS 0.9108 and RPR 1.124, ahead of GPT-3.5 Turbo (MTAS 0.904, RPR 1.116). The narrative text claims Gemini outperformed GPT-3.5 Turbo "in RPR by 0,72", which does not match the table values.

• More shots helped incrementally. GPT-4o saw a 1.66% improvement in Three-Shot TAS over Two-Shot TAS.

• Model size correlates with out-of-the-box performance. The authors note a correlation between model size (and by extension training-data volume) and immediate performance on automatic diacritic restoration.

• The â/î orthographic pair is the dominant error source. In the error analysis over n = 74,489 reference diacritics, 21.3% of all errors came from misplacing â, usually over-correcting to î inside words. The paper suggests rule-aware post-processing could eliminate up to 19% of remaining mistakes.

• Context matters. Sentence-initial capital Ș and Ț were restored correctly only 94.6% of the time, versus 98.7% for their lower-case counterparts. Syllable-final ă saw a 1.8 pp drop in recall relative to syllable-initial occurrences.

• Mixtral over-generated; another model under-generated. Mixtral-8x7B-Instr added on average 16.35 diacritics per 10-word sentence — almost three times the corpus expectation — while Codest-Ral-2501 showed a conservative strategy trading recall for precision. (Codest-Ral-2501 appears only in the error-analysis section and is not listed in the paper's model overview table.)

• Text type affected accuracy. Multi-sentence definitions in DLRLC-1000 exhibited 12.4% more errors than single-sentence entries, with a standard deviation of 0.6 pp across models.

## Methodology in Plain English

The authors drew text from the dexonline Project in two streams: DLRLC, a dictionary of contemporary literary Romanian, and CRAWLER, phrases harvested from online literary sources. To keep inference costs manageable, they sampled 1,000 statements from each source — 2,000 in total. Every statement was preprocessed by stripping all diacritics, simulating the everyday reality of Romanian typing.

Each model was then run against that stripped text using a graded series of prompts. The prompts began with the minimal instruction "Restore the diacritics: {input}", moved to a verbose version that forbade extra commentary and separated instruction, input, and output, and then added one, two, and three worked input-output examples. Models were also asked to handle the post-1993 orthographic rule governing "î" versus "â" inside words.

Outputs were scored two ways: restoration accuracy (at character and word level, each in case-sensitive and case-insensitive variants) and restoration error rate based on Levenshtein distance. Aggregates included APS, TAS, and MTAS. Crucially, all results were checked against an "Echo" model that returns the stripped input untouched — a 0.8100 MTAS bar that the authors argue prevents over-crediting models that merely score well on superficial metrics.

Why This Matters

Impact on research. The paper argues that high accuracy figures for diacritic restoration can be misleading without a do-nothing baseline, because a stripped Romanian text that is left alone already matches much of the original character by character. By publishing the RPR metric and full evaluation code, the authors give the community a more honest yardstick — and they show that several widely used open models fall below the trivial baseline on this task.

Real-world applications:

• Cleaning up user-generated content — reviews, forum posts, chat messages — typed on keyboards without Romanian diacritic support.
• Preparing corpora for training or fine-tuning Romanian language models, where diacritic integrity affects downstream quality.
• Preprocessing digitized or OCR-scanned Romanian documents that lost their diacritics in transcription.
• Improving search and information retrieval, since undiacritized text creates lexical ambiguity.

Industry relevance. The results matter for anyone choosing an LLM for a Romanian-language pipeline. The paper shows that model scale alone is not sufficient: Mixtral's over-generation contrasts with the more balanced output of GPT-4o, and Meta's Llama 2 models underperformed dramatically. It also quantifies the payoff from prompt engineering — three-shot templates beat zero-shot nearly everywhere — which translates directly into deployment and cost decisions.

Future Directions

1. Finetuning underperforming open models. The authors explicitly propose investigating how much finetuning Meta's Llama 2 7B and similar models would need to reach state-of-the-art diacritic restoration.

2. Rule-aware post-processing for the â/î pair. Since 21.3% of errors trace to this orthographic rule, targeted correction could remove up to 19% of remaining mistakes.

3. Extending to pre-1993 orthography and other languages. The current corpus covers post-reform Romanian only; the authors plan analysis of pre-reform data and acknowledge the findings may not transfer to languages with different diacritical systems.

4. Scaling the evaluation and enriching the metrics. The study used 1,000 statements per dataset due to computational cost, and the authors note that larger samples and additional criteria such as contextual appropriateness and human judgment could yield more robust, nuanced results. Prompt engineering is also described as a rapidly evolving field with techniques not yet explored here.

Target Audience

Researchers and engineers working on Romanian NLP or on low-resource and morphologically rich languages will get the most from this paper, particularly those selecting a model for a text-cleaning pipeline or designing benchmarks. It is also valuable for practitioners interested in prompt engineering, since the staged prompt templates provide a concrete template for measuring how much few-shot examples actually buy. Linguists and computational-linguistics students studying diacritic restoration and orthographic reform will find the error analysis section directly useful.

Authors’ abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.