Benchmarking Machine Translation on Chinese Social Media Texts

AI summary

Overview

Research area: Machine translation (MT) benchmarking for informal, user-generated Chinese social media text, with a focus on LLM evaluation.

Technical level: Intermediate — readers benefit from familiarity with MT metrics (XCOMET, COMET, BLEU, chrF), embedding-based similarity, and LLM-as-a-judge evaluation.

Scope: The paper introduces CSM-MTBench, an expert-annotated Chinese-to-five-language benchmark with targeted slang and style-preservation metrics, and reports results for 22 translation and general-purpose models.

What This Paper Is About

Machine translation systems handle formal text such as news or Wikipedia reasonably well, but they struggle with social media writing, where Chinese users constantly create slang, neologisms, abbreviations, and stylized expressions that shift quickly over time. The authors identify two obstacles: data scarcity (high-quality parallel data needs bilingual annotators who understand platform-specific slang) and metric limitations (traditional evaluators such as COMET often miss stylistic fidelity and non-standard expressions). The paper's goal is to build a curated benchmark and evaluation methods that expose how current models actually handle this kind of language.

Key Contributions

1. CSM-MTBench, a new benchmark. It covers five Chinese-to-foreign-language directions (Spanish, French, Japanese, Korean, Russian) and comprises 10,915 total instances. It is organized into two subsets: Fun Posts (1,183 source samples, 5,915 total samples), containing longer, context-rich posts with frequent slang and neologisms, and Social Snippets (1,000 source samples, 5,000 total samples), containing short, emotion- and style-driven comments. Source data were collected from Xiaohongshu (RedNote) between January 2025 and July 2025 from a pool of over 100k posts and comments, using an engagement-aware selection strategy (1,500 posts chosen by exposure level, view counts, and likes; 5,000 candidate comments chosen by engagement signals and expressiveness). After filtering, bilingual experts translated the retained samples.

2. Targeted evaluation metrics for each subset. For Fun Posts, the authors introduce the Slang Success Rate (SSR), which uses GPT-5 to identify slang/neologistic expressions and their gold translations, augments them into candidate dictionaries, and applies fuzzy matching (RapidFuzz) with a threshold of 0.8. For Social Snippets, they introduce Embedding Similarity (ES), averaging cosine similarity over style embeddings, emotion embeddings, and sentiment embeddings, complemented by a GEMBA-stars LLM-as-a-judge evaluation of tone and style preservation.

3. A large-scale empirical study. The paper evaluates 22 models spanning closed-source LLM APIs, translation-specialized models, and open-source general-purpose LLMs, using XCOMET-XL for overall quality.

4. Behavioral analysis. The paper adds a prompt sensitivity analysis across five prompting strategies, qualitative case studies, and an annotation-difficulty analysis.

Main Findings

• Closed-source APIs lead on Fun Posts. GPT-5 averages 85.26 XCOMET and 77.70 SSR; GPT-4o averages 85.50 XCOMET and 64.91 SSR; Claude-Sonnet-4 averages 85.19 XCOMET and 71.31 SSR. GPT-4o and GPT-5 are described as comparable on XCOMET (not statistically significant), while GPT-5 leads by a large and consistent margin on SSR.

• Translation-specialized models lag behind, especially on slang. NLLB-3.3B averages 65.49 XCOMET and 23.52 SSR. Google Translate achieves the highest SSR within this group at 53.57 (XCOMET 81.93). GemmaX2-9B averages 82.72 XCOMET and 52.68 SSR, and Hunyuan-MT-7B averages 83.71 XCOMET and 51.60 SSR.

• Open-source general-purpose LLMs show a clear scaling trend. DeepSeek-V3 leads this group at 84.10 XCOMET and 68.46 SSR, surpassing GPT-4o in SSR. Qwen3-235B-A22B reaches 84.10 XCOMET and 61.04 SSR. Within the Gemma3 (4B, 12B, 27B) and Qwen3 families, larger models consistently outperform smaller ones, and instruction-tuned variants outperform earlier versions.

• Social Snippets are harder overall. XCOMET scores are noticeably lower than on Fun Posts. GPT-5 surpasses GPT-4o on both XCOMET (76.55 vs 76.50) and ES (68.27 vs 68.20), with GEMBA scores of 3.69 vs 3.65. Claude-Sonnet-4 averages 75.14 XCOMET, 68.10 ES, and 3.58 GEMBA. NLLB-3.3B is the weakest at 58.37 XCOMET, 61.74 ES, and 2.13 GEMBA. DeepSeek-V3 leads open-source models at 76.98 XCOMET, 68.08 ES, and 3.64 GEMBA.

• Slang coverage is substantial but partial. Of the 1,183 Fun Posts samples, 619 contain detected slang or neologisms, and each slang term has 6.6 candidate translations on average.

• The two subsets are distributionally distinct. Using Qwen3-Embedding-8B for t-SNE visualization, the cosine similarity between subset centroids is approximately 0.44 and the normalized centroid distance is 0.24.

• Prompt effects differ by subset. For Fun Posts, a Reminder prompt that alerts the model to slang or neologisms consistently improves both XCOMET and SSR (for example, Qwen3-8B rises from 80.23 XCOMET / 51.95 SSR under Direct (en) to 80.93 / 53.08), while two-stage reasoning and few-shot prompts give little or negative benefit. For Social Snippets, two-stage reasoning and few-shot prompting help. Switching the prompt language from English to Chinese does not significantly affect performance on either subset. Adding the original post as context for Social Snippets consistently improves quality (Qwen3-8B: 72.51 XCOMET without context vs 73.97 with context). Paired bootstrap tests confirm the largest effects in each direction are significant (p < 0.001).

• Annotations are harder for Social Snippets. Fun Posts require an average of 1.2 review rounds before agreement, while Social Snippets require 2.7 rounds. Agreement specifically on slang or neologism usage in Fun Posts requires an average of 1.4 rounds.

• Emojis and punctuation break otherwise easy translations. Qualitative analysis shows that when a source sentence contains emojis or repeated exclamation marks, even simple expressions become difficult for models to translate, though two-stage reasoning, few-shot prompting, or extra context can mitigate this.

Methodology in Plain English

The researchers gathered a large pool of real posts and comments from a Chinese social media platform and narrowed it down using engagement signals rather than random sampling, so the retained text reflects language people actually see and interact with. Everything went through strict filtering to remove personal information, violent content, politically sensitive content, hate speech, and inappropriate material, plus an internal safety screening. The remaining Chinese text was translated into the five target languages by bilingual expert annotators who were trained on a pilot set of 100 instances and followed guidelines emphasizing preservation of slang, tone, and pragmatic nuance, with peer review and adjudication producing the final references.

They then split the data by communication function. Longer posts became Fun Posts, evaluated for whether specific slang terms survive translation: an LLM proposes slang items and acceptable translations, humans clean the candidate list, and fuzzy matching checks whether any accepted expression appears in the model's output. Shorter comments became Social Snippets, where slang is hard to isolate, so instead three embedding models encode the source and the translation and cosine similarities are averaged into a single style/emotion/sentiment score, with an LLM judge rating tone and style preservation. All model outputs were also scored with XCOMET-XL. Finally, they ran the same models under several different instructions, with and without extra context, to see how sensitive results are to how the task is framed.

Why This Matters

This work matters because the benchmark targets the language that ordinary people actually write, not the polished text on which most MT evaluation rests. As the authors note, existing Chinese social media resources such as BOLT and HADQAET largely reflect earlier-stage social media language, while Redtrans-Bench contains 2,858 zh-to-en SNS translation pairs, is limited to a single language pair, and is evaluated only with XCOMET. CSM-MTBench differs by using up-to-date data, covering multiple target languages beyond zh-to-en, using careful human curation, and adding targeted metrics for social media text. It also demonstrates that higher overall quality scores do not automatically mean slang or tone survive translation, which is a distinction standard metrics hide.

Real-world applications:

• Cross-border social platforms and content localization, where posts and comments must read naturally to foreign users rather than merely be semantically correct.
• Marketing and creator-economy workflows, where the tone of a post or comment often carries more weight than its literal content.
• Subtitling, community management, and customer support for platforms with multilingual audiences, where short emotional replies (laughter, surprise, frustration) are common.
• Model development and diagnostics, since the benchmark's slang-level and style-level metrics can guide training or prompting decisions rather than just ranking systems.

Industry relevance is direct: the work comes from Xiaohongshu Inc. with collaborators at The University of Tokyo and Tsinghua University, and the dataset is released at https://github.com/KYuuto1006/CSM-MTBench under a CC BY-NC-SA 4.0 license. Kaiyan Zhao was supported by JST SPRING, Grant Number JPMJSP2108.

Future Directions

• Move beyond prompting. The authors state as a limitation that they only investigate prompting strategies, and suggest slang-aware pretraining and targeted fine-tuning as directions for future work.
• Extend language coverage. The benchmark currently covers five Chinese-foreign directions; adding more is constrained by annotation availability and cost.
• Avoid over-interpreting rankings. The paper states that CSM-MTBench is designed to reveal systematic patterns rather than produce a definitive system ranking, noting that several top-tier models score almost identically on some metrics and leaving the definitive ranking to future work.
• Improve evaluation of short informal text. Social Snippets required 2.7 annotation rounds on average and produce lower XCOMET scores, indicating room for better metrics and annotation protocols for very short, emotion-driven content. The paper also notes that ES offers a computationally efficient alternative to LLM judges, which invites further validation of embedding-based evaluation.

Target Audience

Researchers and engineers working on machine translation, multilingual evaluation benchmarks, and low-resource or informal language processing will gain the most, particularly those interested in Chinese-language NLP. Practitioners building translation features for social platforms, content localization teams, and model developers who need to diagnose slang and tone handling will also find the benchmark and its targeted metrics directly useful. Readers without a background in MT evaluation can still follow the paper's central point that meaning and style are different things, and that current models frequently get the first right while failing the second.

Authors’ abstract

The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese social media, poses significant challenges for Machine Translation (MT) benchmarking. Specifically, we identify two primary obstacles: (1) data scarcity, as high-quality parallel data requires bilingual annotators familiar with platform-specific slang, and stylistic cues in both languages; and (2) metric limitations, where traditional evaluators like COMET often fail to capture stylistic fidelity and nonstandard expressions. To bridge these gaps, we introduce CSM-MTBench, a benchmark covering five Chinese-foreign language directions and consisting of two expert-curated subsets: Fun Posts, featuring context-rich, slang- and neologism-heavy content, and Social Snippets, emphasizing concise, emotion- and style- driven expressions. Furthermore, we propose tailored evaluation approaches for each subset: measuring the translation success rate of slang and neologisms in Fun Posts, while assessing tone and style preservation in Social Snippets via a hybrid of embedding-based metrics and LLM-as-a-judge. Experiments on over 20 models reveal substantial variation in how current MT systems handle semantic fidelity and informal, social-media-specific stylistic cues. CSM-MTBench thus serves as a rigorous testbed for advancing MT systems capable of mastering real-world Chinese social media texts.