LASER: An LLM-based ASR Scoring and Evaluation Rubric

AI summary

Overview

Research area: Automatic Speech Recognition (ASR) evaluation metrics, with a focus on Indian languages; large language model prompting and fine-tuning.

Technical level: Intermediate. Readers should be comfortable with concepts such as Word Error Rate, edit-distance metrics, embeddings-based metrics, in-context learning, and LoRA fine-tuning.

Scope: The paper introduces LASER, an LLM-based scoring rubric that assigns graded penalties to ASR errors according to their semantic impact, and evaluates how well these scores agree with human judgments across Hindi, Marathi, Malayalam, Kannada, and (in an appendix experiment) English.

What This Paper Is About

Standard ASR metrics such as Word Error Rate (WER) and Character Error Rate (CER) are exact-match, edit-distance measures: they count every substitution, insertion, and deletion equally, so a harmless spelling variant is penalized the same as a meaning-changing error. This hurts evaluation of Indian languages in particular, which are morphologically rich, use compound words with multiple accepted forms, and contain English loan words with no standardized native-script spelling. The paper's goal is a nuanced scoring rubric that ignores semantically equivalent variation, lightly penalizes minor grammatical or spelling slips, and heavily penalizes errors that alter meaning — and to show that LLM scores from this rubric align closely with human annotations.

Key Contributions

1. A three-tier penalty rubric for ASR errors. LASER defines no-penalty error types (numerical phrases, abbreviations, compound words, transliteration variants, alternate spellings, proper nouns, slang and colloquial terms), minor-penalty types (small single-character spelling errors, small grammatical errors in gender/tense/number), and major-penalty types (incorrect word substitutions, significant omissions or additions, meaning-changing reorderings, meaning-altering spelling errors).

2. A sentence-level scoring formula and an engineered prompt. Major errors cost 1 point, minor errors cost 0.5 points, and non-penalizable errors cost nothing; the sentence score is 1 − (total penalty / number of reference words). The prompt contains English instructions with Hindi examples, a worked example, and a chain-of-thought response format.

3. Demonstration that a Hindi-example prompt transfers cross-lingually to Marathi, a related Indo-Aryan language, and to Malayalam and Kannada, which are Dravidian languages that are syntactically and morphologically very different from Hindi.

4. A cheaper alternative via fine-tuning. A Llama 3 8B model was LoRA-fine-tuned on word-pair examples (3.4M trainable parameters, 950 word-pairs) to classify pairs as "no mismatch", "non-penalizable error", "major penalty", or "minor penalty", achieving 88.69% overall test accuracy. Scripts and checkpoints for Hindi LASER are released at https://github.com/Amparulekar/LASER-metric.

Main Findings

• Gemini 2.5 Pro gave the strongest agreement with humans on Hindi: Hindi LASER scores using Gemini 2.5 Pro reached a 94% correlation with human annotations, outperforming every other LLM tested and markedly surpassing WER.

• Humans were enlisted as ground truth under identical rules: Annotators with linguistic knowledge of the target languages received the same instructions, examples, and worked-out sample as the prompt, and were paid Rs. 24 per sentence pair to list and count major, minor, and no-penalty errors.

• The Hindi prompt transferred well across Indian languages: Using the Hindi-example prompt on Marathi, Malayalam, and Kannada data produced trends similar to Hindi, with Gemini 2.5 Pro again the best-performing LLM. Marathi correlations were higher than Hindi correlations, which the authors suggest may be because Marathi sentences averaged 20.86 words versus 27.34 words for Hindi. Malayalam and Kannada correlations were lower than Hindi and Marathi, potentially because the Hindi prompt could not capture all nuances of Dravidian languages.

• A Marathi-example prompt did not beat the Hindi one: A second prompt with English instructions and Marathi examples was tested on both Hindi and Marathi sentences; the Hindi prompt performed better for both languages, likely because the LLM is more familiar with Hindi.

• The model inferred penalties for error types absent from the prompt: Gemini 2.5 Pro was able to assign appropriate penalties for sandhi (phonetic transformation at word boundaries during word fusion) and synonyms, which were not included in the prompt. It also consistently recalled the scoring instructions and retained correct output formatting after each batch.

• Word-pair classification accuracy was 88.69% overall: On the fine-tuned Llama3-8B test split, class-wise accuracies were 94.12% for identical pairs (32/34 correct), 88.57% for non-penalizable errors (31/35), 66.67% for minor penalties (6/9), and 89.29% for major penalties (25/28). Minor-penalty errors were the hardest to identify, as they were the least frequent class.

• The fine-tuned model matched or exceeded Gemini on held-out sentences: On 17 held-out Hindi sentence pairs, Llama3 performed even better than Gemini 2.5 Pro and was more aligned with human scores, which the authors attribute to manual corrections of the Gemini 2.5 Pro outputs before training.

• High-WER samples were checked qualitatively: For samples with WER greater than 0.35, high LASER scores corresponded to a high percentage of non-penalizable errors that preserved word-pair meaning, while low LASER scores corresponded to significant semantic mismatches; human scores agreed with both cases.

• On English, the advantage over WER narrowed: Using the Hindi prompt on 80 Common-voice English test samples transcribed with Whisper, LASER still correlated best with human evaluations, but the gap between WER-human and LASER-human correlation was significantly smaller than for Indic languages — attributed to English being less morphologically complex, with only 3 of the 9 no-penalty error types (slang/contractions, proper nouns, numerical phrases) predominantly observed.

• Known inconsistency and latency limitations: LLMs can process ambiguities differently across runs (for example, treating a slang spelling as a spelling error), and this variation was higher for lower-resource languages. The prompt-based method also has higher latency than other metrics.

Methodology in Plain English

The authors began by studying which errors standard ASR metrics penalize, then reassigned penalties based on how much each error type actually changes meaning. They built a prompt that gives the LLM three things: detailed instructions (tokenize the sentences, align the reference and prediction into word pairs, classify mismatches, total the penalties, compute the score), one concrete example for every error type in the rubric, and a required step-by-step output format so the model reasons through the task consistently.

They tested this prompt on several reasoning-oriented LLMs, including Gemini 2.5 Pro, Gemini 2.5 Flash, GPTo3, and GPTo4mini (Deepseek R1 failed to produce scores with the prompt). To have a reference point, they gave human annotators the same instructions, examples, and sample, and had them report sentence-wise error lists and counts; these counts were converted into LASER scores. They then measured Pearson's correlation between the human scores, the LLM scores, WER, and BERTScore(F1).

The data came from a subset of the IndicVoices test set — multilingual, multi-speaker, spontaneous speech — with ASR predictions generated by the SeamlessM4T model. Sentence pairs with zero WER were removed, leaving 172 Hindi, 154 Marathi, 229 Malayalam, and 216 Kannada sentence pairs. Cross-lingual experiments reused the same Hindi-example prompt on the other three languages.

Finally, to avoid expensive LLM API calls, they aligned reference and ASR output into word pairs and fine-tuned Llama 3 8B with LoRA to classify each pair directly. Evaluation used both a test-train split and a held-out set of 17 of the 172 Hindi sentences, which were aligned with a custom greedy alignment script.

Why This Matters

Impact on research: WER is a lexical, exact-match metric that treats all errors as equally damaging, which misrepresents system quality for morphologically rich and code-mixed languages. LASER offers a semantics-aware alternative whose scores track human judgment closely, and it shows that an LLM prompted with examples from one language can transfer its penalty reasoning to languages from a different family. It also demonstrates that a compact fine-tuned model can reproduce the rubric cheaply and consistently, which addresses the run-to-run variability of prompt-based LLM scoring.

Real-world applications:

• Benchmarking and comparing ASR systems for Indian languages, where WER understates real transcription quality.
• Voice assistants and dictation tools for Hindi and other Indic languages, where users speak colloquially and mix in English loan words.
• Accessibility tools such as captioning, where transcription that changes meaning is far more harmful than a spelling variant.
• Post-editing and quality triage, since flagging major-penalty word pairs directs human reviewers to the errors that actually matter.

Industry relevance: ASR vendors and product teams need evaluation metrics that reflect user-perceived quality rather than string-level edits. The paper's fine-tuned word-pair classifier, trained on only 950 word pairs, is a practical, low-latency path to using this rubric at scale, and the released Hindi scripts and checkpoints lower the barrier to adoption.

Future Directions

• Standardizing LLM scoring across runs. The authors note that ambiguity is handled differently on different runs, especially for lower-resource languages, and call for a standardized technique that guarantees the same score every time. They suggest fine-tuning with fixed weights as a partial remedy.
• Further latency reduction. Prompt-based LASER is slower than other metrics, and although LoRA fine-tuning of Llama improved efficiency, the authors identify reducing latency as an open direction.
• Improving performance on Dravidian languages. Malayalam and Kannada correlations were lower than Hindi and Marathi, which the authors attribute to nuances the Hindi prompt could not address — an obvious target for language-specific prompts or examples.
• Extending the rubric beyond the error types already covered. The observation that Gemini inferred penalties for sandhi and synonyms not present in the prompt raises the question of how far this inference can be pushed, and whether the rubric should be extended explicitly.

Target Audience

This paper benefits speech and NLP researchers working on ASR evaluation, particularly those working on Indian languages and code-mixed or spontaneous speech; engineers building or benchmarking ASR products that need metrics closer to human judgment; and applied researchers interested in how far in-context learning from a single well-designed prompt can transfer across languages. Readers with a background in speech recognition metrics or LLM fine-tuning will get the most from the methodology and results sections.

Authors’ abstract

Standard ASR evaluation metrics like Word Error Rate (WER) tend to unfairly penalize morphological and syntactic nuances that do not significantly alter sentence semantics. We introduce an LLM-based scoring rubric LASER that leverages state-of-the-art LLMs' in-context learning abilities to learn from prompts with detailed examples. Hindi LASER scores using Gemini 2.5 Pro achieved a very high correlation score of 94% with human annotations. Hindi examples in the prompt were also effective in analyzing errors in other Indian languages such as Marathi, Kannada and Malayalam. We also demonstrate how a smaller LLM like Llama 3 can be finetuned on word-pair examples derived from reference and ASR predictions to predict what kind of penalty should be applied with close to 89% accuracy.