Assessing Quality of Experience in Natural Language Generation of German Text

AI summary

Overview

• Research area: Natural Language Processing / Natural Language Generation evaluation, with an emphasis on Quality of Experience (QoE), human-centered evaluation, and automatic quality prediction for German text.
• Technical level: Intermediate (accessible to readers who know basic NLP evaluation concepts; some statistical and modeling terminology assumes familiarity).
• Scope: The paper introduces TextQ-German, a dataset suite of human QoE ratings for German automatic text summarization (ATS) and machine translation (MT), and benchmarks transformer-based, linguistic-feature-based, and hybrid models for automatically predicting those ratings, including on LLM-generated text and held-out validation sets.

What This Paper Is About

Traditional NLG evaluation relies on automatic metrics such as BLEU and ROUGE, which compare surface-level lexical overlap with reference texts and are known to correlate poorly with human judgment. The authors argue that the real success of NLG systems — large language models included — depends on the Quality of Experience they deliver to actual users, and that this is underexplored for German. Their goal is to build a German-language resource of human perceptual ratings across two NLG tasks (summarization and translation) and to train models that can predict perceived quality automatically.

Key Contributions

1. A human-annotated QoE dataset suite for German NLG. TextQ-German comprises six subsets covering ATS and MT: the original corpora TextQ-ATS (91 samples) and TextQ-MT (106 samples) with dimension-level ratings, LLM-generated extensions TextQ-ATS-LLM and TextQ-MT-LLM (77 samples each) with an added overall QoE score, and held-out validation sets TextQ-ATS-Val (77 items) and TextQ-MT-Val (76 items). Datasets are publicly released under a CC BY-NC 4.0 license at https://github.com/DFKI-NLP/TextQ/.

2. Empirically derived task-specific perceptual quality dimensions. Rather than pre-specifying criteria, the authors used crowdsourcing with Semantic Differential scaling and Exploratory Factor Analysis to derive four dimensions per task: Precision, Complexity, Grammaticality, and Transparency for MT; Linguistic Logic, Complexity, Clarity, and Predictability for ATS. Complexity is the only dimension shared across both tasks (indicated by the simple–complicated pair).

3. A suite of automatic QoE prediction models. The paper develops and compares transformer-based models (five fine-tuned German pretrained models), feature-based models built on 121 implemented linguistic features, and hybrid approaches that combine both, at both dimension-level and overall-QoE levels.

4. Generalization testing on held-out data spanning LLM and non-LLM generation. The validation sets deliberately mix LLM-generated and non-LLM neural outputs and are not used for feature selection, hyperparameter tuning, checkpoint selection, or model selection.

Main Findings

• Hybrid models lead: Hybrid models — combining transformer representations with linguistic features — outperform pure transformer baselines in almost all experimental settings reported in the abstract.
• Linguistic features are competitive on their own: Models using linguistic features alone can approach the performance of fine-tuned language models.
• Held-out validation shows generalization: Final validation on TextQ-ATS-Val and TextQ-MT-Val indicates that the best QoE predictors generalize to unseen data.
• Four quality dimensions per task: Exploratory Factor Analysis yielded a four-factor structure with eight adjective pairs for each text type. MT fit statistics: Pearson's chi-squared test gave p = 0.36 (χ² = 2.06, df = 2). ATS fit statistics: p = 0.63 (χ² = 0.92, df = 2).
• Explained variance by factor (MT): F1 = 53.2%, F2 = 8.4%, F3 = 10.5%, F4 = 8.0%.
• Explained variance by factor (ATS): F1 = 54.4%, F2 = 14.3%, F3 = 10.6%, F4 = 2.6%.
• Reduced measurement scheme holds up: Correlating the full and reduced adjective-pair experiments gave Spearman correlation coefficients consistently around 0.8 for both text types. Jarque–Bera tests showed no significant departures from normality; Levene's test indicated homogeneous variances for all factors except MT's F2 (Complexity) and ATS's F4 (Predictability), which were analyzed with Welch's t-test. For ATS, no factor differed significantly between the two experiments; for MT, only F2 (Complexity) differed, by 0.5 points on a 7-point scale, which the authors treat as acceptable.
• Complexity is not interpreted the same way across tasks: The small MT Complexity discrepancy is attributed to translations favoring simpler language for fluency while summaries benefit from structural and conceptual depth, pointing to a need for more task-specific quality evaluation.
• Differing dominant dimensions: MT's F1 (Precision) and ATS's F1 (Linguistic Logic) share the pairs precise–vague and complete–incomplete but diverge otherwise — MT adds ambiguity and clarity of phrasing, ATS adds coherence and logical consistency.
• An important caveat on the reported experimental results: The supplied paper content is truncated during the methodology section (at the description of the 2¹²¹ − 1 feature subset search), so the specific performance numbers for the prediction models and the final validation experiments are not available in the provided text and cannot be reported here.

Methodology in Plain English

The authors first collected human judgments. They gathered German source texts from the GeWiki corpus (also used in GermEval 2020 Task 3) and generated summaries with both extractive methods (Lead-3, TextRank) and abstractive methods (Pointer-Generator, Transformer, Convolutional Self-Attention Networks, BERT-Transformer). For MT they sampled English–German translations from the WMT19 News Translation Task, deliberately taking outputs from top-, mid-, and bottom-ranked systems. Linguists annotated error types, and the corpus was balanced for quality and error variety.

To find out which qualities users actually perceive, they ran crowdsourcing studies using Semantic Differential scaling with bipolar adjective pairs (e.g., simple–complicated) rated on a 7-point scale from 0 to 6. Starting from roughly 40 candidate adjective pairs per task, a pre-study reduced this to about 20, and factor analysis narrowed it to eight pairs forming four dimensions per task. Studies ran on the Crowdee platform with self-identified native German speakers in the DACH region, with attention checks, an "being observed" effect to improve reliability, and data cleaning rules (discarding responses completed in 240 seconds or less, or with identical slider values across all pairs). A second quantification study validated that a reduced set of four adjective pairs per task could reproduce the original findings.

The LLM extension was produced with a mix of commercial API models (GPT-4o, GPT-3.5 Turbo) and locally hosted open-weight models via Ollama (Llama 3.2 3B, StableLM 2 1.6B, DeepSeek-R1 1.5B, SmolLM2-German-Instruct 360M, SauerkrautLM-7B-v1), then manually annotated and rated in further crowdsourcing runs that added an overall good–bad QoE item.

For prediction, all tasks are framed as regression on item-level Mean Opinion Scores (averages of annotator ratings). Models include fine-tuned German transformer models (bert-base-german-uncased, bert-base-german-cased, gbert-base, gbert-large, gelectra-large) with a linear output layer, feature-based models using 121 linguistic features (readability, lexical richness, syntactic, and morphological features computed with spaCy and textstat), and hybrids of the two. Evaluation proceeds in four stages: dimension-level prediction, overall QoE prediction, extension to LLM-generated data, and final evaluation on held-out validation sets.

Why This Matters

Impact on research. The work shifts NLG evaluation for German away from surface-overlap metrics and single criteria such as readability toward a multidimensional, user-grounded benchmark. It supplies both a public dataset and baseline models, and it links NLP evaluation to the established QoE paradigm from telecommunications and multimedia, where subjective ratings are used to train objective quality predictors.

Real-world applications:

• Evaluating and monitoring deployed German-language NLG systems such as summarization or translation features in news, government, or enterprise tools, where a single automatic quality score is operationally useful.
• Model and vendor comparison for German translation or summarization services, using perception-based scores rather than n-gram overlap.
• Quality-aware content pipelines, where predictions could flag low-QoE generated text for human review before publication.
• Accessibility and readability assessment for German texts, extending prior German complexity and readability work to a broader set of perceived quality dimensions.

Industry relevance. Organizations deploying LLMs in German-speaking markets lack perception-aligned evaluation tools; this resource offers a task-specific alternative to BLEU/ROUGE and a demonstration that relatively inexpensive linguistic features can be competitive with fine-tuned models — relevant where compute or labeled data are limited. The inclusion of LLM-generated outputs in both the extension and validation sets makes the benchmark directly applicable to current generative systems rather than older neural architectures.

Future Directions

• More task-specific quality evaluation: The MT Complexity discrepancy suggests quality dimensions do not transfer cleanly across NLG tasks, motivating task-tailored dimension sets and measurement designs.
• Extending QoE evaluation to further NLG tasks and languages: The paper covers German ATS and MT; question answering and content generation are named as common NLG tasks that remain unaddressed here.
• Broadening the model comparison beyond the reported results: The full model performance figures, feature importance rankings, and validation outcomes are not available in the provided (truncated) text; readers should consult the paper for the complete experimental tables.
• Leveraging the datasets as a foundation for systems that align better with human perception: The authors frame the resource as groundwork for developing NLG systems that match human quality perception rather than optimizing technical scores.

Target Audience

Researchers and practitioners in NLP evaluation, machine translation, and summarization who work with German or who are interested in human-centered evaluation methods; QoE and subjective-quality-prediction researchers moving from speech, audio, image, or video into text; and industry teams needing German-language benchmarks for judging generative systems. Readers seeking detailed model-performance numbers should note that the supplied content is truncated before those results are reported.

Authors’ abstract

The rapid advancement of Natural Language Generation (NLG) has made the reliable evaluation of generated text increasingly critical, as these systems, such as large language models (LLMs), are now widely deployed in real-world applications. However, traditional automatic metrics fail to capture the multifaceted nature of perceived quality. In this paper, we introduce TextQ-German, a novel dataset suite for human-centered evaluation of German NLG from a Quality of Experience (QoE) perspective, covering automatic text summarization and machine translation. Through crowdsourcing studies with German speakers, we collect human quality ratings and identify relevant perceptual quality dimensions for each task. We develop automatic QoE prediction models, including transformer-based, linguistic feature-based, and hybrid approaches. Hybrid models outperform pure transformer baselines in almost all experimental settings, while linguistic features alone can approach the performance of fine-tuned language models. The dataset is extended with LLM-generated outputs annotated with overall QoE scores. Final validation on held-out sets indicates generalization to unseen data. Our work contributes a publicly accessible resource for NLG evaluation and baselines for automatic QoE prediction, providing a foundation for developing NLG systems that better align with human quality perception.