Readability Reconsidered: A Cross-Dataset Analysis of Reference-Free Metrics

AI summary

Overview

Research area: Natural Language Processing — automatic readability assessment and evaluation-metric validation.

Technical level: Intermediate. The paper is readable without deep technical background, but assumes familiarity with standard NLP concepts such as rank correlation, fine-tuned encoders, and LLM prompting.

Scope: A cross-dataset empirical study of how well traditional readability formulas and newer reference-free model-based metrics correlate with human readability judgments, alongside a small human-rationale study of what actually drives human perception of readability.

What This Paper Is About

Automatic readability assessment is held back by inconsistent definitions of "readability" — some work means surface text difficulty, some means syntax and discourse organization, and some includes information content — so the field keeps using proxy metrics that may not match human comprehension. The authors ask what actually shapes human perceptions of readability, then test how well 15 popular readability metrics and six model-based metrics track human judgments across five English datasets. The goal is to expose the mismatch between existing metrics and human perception, and to identify which approaches are more promising.

Key Contributions

1. A quantitative study of human readability rationales. The two lead authors annotated the human-provided rationales in 90 ELI-Why (GPT-4) question–answer pairs, balanced evenly across classes, into five categories: Wording/Terminology, Sentence Structure, Examples/Analogies, Details and Depth, and Curriculum-based. Inter-annotator agreement was high, with an average sample-level Jaccard index of 0.91.

2. A cross-dataset benchmark of reference-free readability metrics. The authors evaluate 15 popular readability metrics and six model-based metrics across five English datasets (Scientific Papers, CLEAR, ELI-Why (GPT-4), ELI-Why (Human), and ScienceQA), using Kendall Tau-b rank correlation against human labels. The conclusion describes this as benchmarking 20+ reference-less metrics.

3. Evidence that model-based metrics outperform traditional ones. Four model-based metrics consistently occupy the top four positions in rank correlation with human judgments, while the best-performing traditional metric reaches an average rank of 8.6.

4. A documented length bias in LLM-generated explanations. In the ELI-Why (GPT-4) dataset, responses judged as more readable tend to be longer, which the authors attribute to the added detail and specificity that human annotators emphasized.

Main Findings

• Model-based metrics dominate the top of the rankings. All three LLM-as-a-judge variants (0-shot, 5-shot, continuous 0–100) place in the top three, with average ranks of 2.4, 3.2, and 2.4 respectively, followed by Meta Rater (professionalism) at 4.2 and ReadMe++ at 6.2. Model-based metrics surpass surface-form and psycholinguistic metrics by up to 0.24 absolute points.

• The best traditional metric ranks 8.6 on average. Among surface-form and psycholinguistic metrics, the strongest average rank belongs to # Difficult Words (8.6), followed by # Polysyllables (9.6), Automatic Readability Index (11.0), Flesch Reading Grade (11.6), Flesch-Kincaid Reading Ease (11.8), SMOG Index (12.2), and Dale-Chall Readability Score (12.4).

• Meta Rater (readability) fails badly. It achieves an average rank of 21.0, below both psycholinguistic and surface-form metrics. The authors hypothesize that because the examples are generally clear, grammatically correct, and coherent, the model systematically assigns the same readability class.

• Wording/Terminology is the dominant rationale for humans. In the 90-example rationale study, lexical complexity or simplicity is the most frequent justification. Curriculum-based reasoning is invoked far more often for High School and Graduate judgments than for Elementary, while Examples/Analogies is disproportionately used to support Elementary judgments.

• No single model-based metric consistently dominates. The continuous LLM-as-a-judge metric achieves the highest correlations on three datasets but underperforms relative to LLM-as-a-judge (0-shot) on ELI-Why (Human) and ScienceQA.

• Metrics disagree in characterizable ways. LLM-as-a-judge metrics are more sensitive to specialized terminology and sentence structure, whereas fine-tuned models such as ReadMe++ are more sensitive to information density and the presence of connectors and cohesive devices. The continuous LLM judge penalizes texts containing numbers and named entities, while the discriminative judge is more sensitive to scientific terminology, complex sentence structures, and equations.

• Score saturation in the continuous LLM judge. On ScienceQA, 81.30% of continuous LLM-as-a-judge scores are confined to three values.

• Surface-form metrics beat psycholinguistic ones on 4 of 5 datasets. With the exception of ELI-Why (Human), a simpler surface-level metric (for example # Difficult Words or # Syllables) is always on par with or better than popular formulas such as the Automatic Readability Index or the Flesch-Kincaid Reading Ease.

• Even the best metrics are far from perfect. The strongest correlations with human judgments reach up to 0.73, which the authors note still leaves substantial room for improvement.

Methodology in Plain English

The work has two parts. First, the authors take an existing dataset — ELI-Why (GPT-4), which contains GPT-4–generated explanations for 299 "Why" questions labeled by humans as Elementary, High School, or Graduate — and look at the written justifications humans gave for those labels. Two of the paper's authors reread justifications for 90 question–answer pairs, evenly split across the three levels, and tagged each with one or more categories such as wording, sentence structure, examples, detail, or curriculum links. Where both annotators agreed, the tags were counted to see which reasons dominate.

Second, they test how well automated readability metrics match human judgments. They use five English datasets with different properties: Scientific Papers (180 human-edited summaries at Low, Medium, and High complexity), CLEAR (continuous reading-ease scores, subsampled to 1,000 from 4,724 excerpts), ELI-Why (GPT-4) (897 examples) and ELI-Why (Human) (a smaller manually curated set), and ScienceQA (2,295 examples drawn from grades 1–12).

The metrics fall into three groups: surface-form counts (words, sentences, syllables, monosyllables, polysyllables, difficult words, estimated reading time, and other counts), psycholinguistic "readability tests" such as the Automatic Readability Index, Dale-Chall, Flesch-Kincaid Reading Ease, and SMOG, and model-based metrics. The model-based group includes two ModernBERT-based fine-tuned raters (Meta Rater for readability and Meta Rater for professionalism), a BERT-based metric called ReadMe++ that predicts readability on the 6-point Common European Framework of Reference for Languages scale, and three LLM-as-a-judge setups run with Llama-3.3-70B-Instruct using greedy decoding (temperature=0): a zero-shot continuous 0–100 score, a categorical zero-shot prompt, and a 5-shot prompt with five example annotations (two Elementary, two Graduate, one High School).

Because readability labels are ordinal, the authors map categorical labels to monotonically increasing numbers from 0 to k−1, apply a similar transformation to model outputs where needed, and compute Kendall Tau-b correlations between metric outputs and human annotations using the scipy.stats implementation. They summarize overall performance with an average rank order across all datasets.

Why This Matters

Impact on research. The paper shows that human readability perception reaches beyond lexical and syntactic surface features into information content and topic, while the metrics most commonly used in the literature remain surface-level. It also shows that a fine-tuned model targeted directly at readability (Meta Rater (readability), average rank 21.0) can perform worse than the formulas it was meant to improve upon, while a model aimed at a different construct (Meta Rater (professionalism), average rank 4.2) performs better — a warning that metric naming does not guarantee metric validity. The authors call for clearer definitions of readability and more rigorous validation of metrics.

Real-world applications:

• Science communication: judging whether explanations of technical topics are accessible to non-expert readers.
• Health communication: assessing whether patient-facing materials, a domain the paper cites as jargon-heavy, are understandable.
• Legal and regulatory text: the paper cites law as another domain where readability assessment matters for access to information.
• Education: matching texts and explanations to grade levels, as in ScienceQA's K-12 curriculum sourcing, and supporting learners with varied domain familiarity.

Industry relevance. The authors are affiliated with Capital One, and the paper explicitly weighs the trade-off between accuracy and inference cost: LLM-as-a-judge metrics achieve the strongest correlations but require generating text for each instance, making them slower and more resource-intensive than fine-tuned models. For teams deploying readability checks at scale, this trade-off — plus the finding that a fine-tuned readability model can degenerate to a constant prediction — is directly actionable.

Future Directions

• Extending beyond English. The analysis is limited to English-language datasets. The authors note that few high-quality labeled readability datasets exist in other languages, naming French as a rare exception, and suggest creating new multilingual readability datasets or expanding the analysis to other languages.

• Broadening the human-rationale study. The Section 3 findings come from a single QA dataset (90 annotated examples), because reasoning judgments are not widely available in readability datasets. Future work could build datasets with rationales across other domains and tasks.

• Mitigating length bias. Since longer GPT-4 explanations are perceived as more readable in ELI-Why (GPT-4), future work could enforce strict generation lengths, or, when a reference document is available, control the information content within each generation.

• Resolving the metric-selection problem. Because no single model-based metric consistently dominates across datasets, and because the continuous LLM judge saturates (81.30% of ScienceQA scores in three values), the field still needs metrics that are both accurate and stable.

Target Audience

Researchers and practitioners in NLP evaluation and readability assessment who need to choose a readability metric; teams building educational technology, health communication, or science communication tools that depend on matching text to an audience; and industry engineers who must weigh evaluation accuracy against inference cost when deploying LLM-as-a-judge pipelines. Readers interested in human-centered evaluation methodology — particularly the use of annotator rationales as evidence — will also find the paper useful.

Authors’ abstract

Automatic readability assessment plays a key role in ensuring effective and accessible written communication. Despite significant progress, the field is hindered by inconsistent definitions of readability and measurements that rely on surface-level text properties. In this work, we investigate the factors shaping human perceptions of readability through the analysis of 897 judgments, finding that, beyond surface-level cues, information content and topic strongly shape text comprehensibility. Furthermore, we evaluate 15 popular readability metrics across five English datasets, contrasting them with six more nuanced, model-based metrics. Our results show that four model-based metrics consistently place among the top four in rank correlations with human judgments, while the best performing traditional metric achieves an average rank of 8.6. These findings highlight a mismatch between current readability metrics and human perceptions, pointing to model-based approaches as a more promising direction.