Evaluating Large Language Models on Urdu Idioms

AI summary

Overview

Research area: Natural Language Processing — multilingual evaluation of large language models, with a focus on figurative language and low-resource languages.

Technical level: Intermediate. The paper's topic (idioms, translation, prompting) is approachable, but the discussion of cross-script evaluation and machine translation comparisons assumes some familiarity with NLP evaluation practice.

Scope: The paper builds and uses a benchmark of Urdu-to-English idioms to test how well large language models handle figurative meaning across native and Romanized Urdu script.

What This Paper Is About

Idioms are hard for computers because their meaning cannot be read off the individual words — an idiom means something figurative and culturally specific rather than what it literally says. Large language models have improved at idioms in some languages, but Urdu, a low-resource language, has received little attention. This paper addresses that gap by creating a verified Urdu idiom dataset and using it to compare how different language models and machine translation systems handle idiomatic meaning.

Key Contributions

1. A manually verified Urdu–English idiom benchmark containing 4,000 aligned idiom sentence pairs, provided in both native Perso-Arabic Urdu script and Romanized Urdu.
2. Coverage of two writing systems for the same content, enabling evaluation of how script and orthography affect model performance — a comparison the abstract frames as underexplored for Urdu.
3. Evaluation across several tasks rather than a single one: translation, paraphrasing, idiom span detection, and back-translation.
4. A comparison of prompting strategies, including cultural prompting, idiomatic prompting, and few-shot learning, alongside a comparison against traditional neural machine translation systems.

Main Findings

• Large language models beat traditional NMT systems: Frontier LLMs are reported to consistently outperform conventional neural machine translation across all evaluation settings tested.
• Figurative meaning is where the gap shows most: The advantage of LLMs is described as particularly noticeable in preserving figurative and metaphorical meaning.
• Native script is comparatively stable: Models perform relatively consistently when working with Urdu in its native Perso-Arabic script.
• Romanized Urdu is a weak point: The lack of standardized spelling in Romanized Urdu causes substantial problems for consistency and for detecting idiom spans.
• Prompt design matters: The abstract emphasizes prompt engineering as important for preserving figurative language meaning across languages, implying that how a task is prompted affects outcomes.

The abstract does not report quantitative scores, per-task accuracy figures, or specific model baselines, so those details are not available here.

Methodology in Plain English

The researchers assembled a dataset of Urdu idioms together with their English equivalents, pairing each idiom with sentences and verifying the pairs by hand. They wrote everything twice — once in Urdu's native script and once in Romanized Urdu, where Urdu is written using Latin letters. They then set the models a series of tasks: translate the idiom, paraphrase it, locate the idiom within a sentence, and translate back again. To see whether the way you ask matters, they varied the prompts, using strategies that hint at cultural context, that emphasize idiomatic rather than literal reading, and that supply a few examples before the question. Finally, they compared model results against traditional machine translation systems to see which approach holds figurative meaning better.

Why This Matters

Impact on research: The paper adds Urdu to the set of languages with a curated idiom benchmark and makes cross-script evaluation part of the picture, which is unusual since most idiom resources exist in a single script. It also argues that prompting choices are a variable worth reporting, not just the model choice.

Real-world applications:

• Translation and localization: Better handling of Urdu idioms helps produce translations that read naturally rather than literally, useful for media, publishing, and government communication.
• Chatbots and assistants for Urdu speakers: Systems that can interpret idioms avoid the awkward literal misreadings that make conversational tools feel foreign.
• Search and information retrieval: Recognizing where an idiom sits in a sentence helps search engines index and match idiomatic queries correctly.
• Romanized text processing: Because many Urdu speakers type in Latin letters online, understanding Romanized Urdu matters for social media analysis and moderation tools.

Industry relevance: Any company serving Urdu-speaking users — messaging platforms, e-commerce, customer support, content platforms — depends on handling informal and figurative language. The finding that Romanized Urdu lacks consistent spelling is a practical warning for teams whose training data comes from user-generated text.

Future Directions

• Standardizing or normalizing Romanized Urdu: The abstract identifies inconsistent orthography as the main obstacle, which points to spelling normalization or transliteration as a needed next step.
• Extending the benchmark beyond Urdu: The approach could be repeated for other low-resource languages with rich idiomatic traditions.
• Understanding why prompting helps: The paper shows prompting strategies matter but leaves open which strategies work best, for which tasks, and why.
• Connecting span detection to downstream use: Improving the weakest task could feed into translation, retrieval, and other applications that need to know which words form the idiom.
• Testing newer models: Since frontier models are compared here, tracking how subsequent model generations perform on the same benchmark is a natural follow-up.

Target Audience

Researchers working on multilingual NLP, figurative language, and low-resource language evaluation; developers building Urdu-language or cross-lingual applications; and readers interested in how script and spelling variation affect model reliability. The paper is also relevant to anyone studying prompt engineering as an evaluation variable rather than a fixed setup.

Authors’ abstract

Idioms remain a persistent challenge in natural language processing due to their figurative and culturally grounded meanings, which distinguish them from literal expressions. Although recent advances in large language models (LLMs) have improved idiom handling across several languages, limited attention has been given to low resource languages such as Urdu. In this work, we present a comprehensive benchmark for Urdu to English idiomatic translation, consisting of a manually verified dataset of 4,000 aligned idiom sentence pairs in both Perso Arabic (native Urdu script) and Romanized Urdu. We evaluate multiple tasks, including translation, paraphrasing, idiom span detection, and back-translation, using diverse prompting strategies such as cultural prompting, idiomatic prompting, and few-shot learning. Our findings show that frontier LLMs consistently outperform traditional neural machine translation systems across all evaluation settings, particularly in preserving figurative and metaphorical meaning. While models demonstrate relatively stable performance on native Urdu script, the absence of standardized orthography in Romanized Urdu introduces substantial challenges for consistency and idiom span detection. This work establishes a high quality benchmark for cross script idiomatic evaluation in Urdu and underscores the importance of prompt engineering in preserving figurative language meaning across languages.