The Pulse
Study Finds Nine AI Models Use More Language Features Linked to Processing Difficulty
A study found that nine frontier and near-frontier language models used several language features associated with human processing difficulty more often than human-written comparisons. A separate experiment found that length penalties shift
AI.info Team · Written with AI · fact-checked ·
All nine frontier and near-frontier language models in a new study used more inanimate subjects paired with animate verbs, and more uncommon word sequences, than human-written comparison texts. Most also showed longer grammatical dependencies, more noun stacks and more reduced relative clauses in conversation, according to a preprint posted October 21 on arXiv.
The study, by Thomas Hikaru Clark of New York University and Matias B. Martin and Kyle Mahowald of the University of Texas at Austin, examines language features associated in prior research with processing difficulty. It does not report a controlled test of how long people take to read the model outputs or whether they understand them less accurately. Instead, the researchers measure those features and compare them with human writing and paraphrases produced by translating AI text into Japanese and back into English.
Six features linked to reading effort
The authors examine uncommon word combinations, long grammatical dependencies, object relative clauses, stacked noun compounds, verbs formed from nouns and inanimate subjects used with verbs typically associated with living actors. Examples include an error-handling function that “swallows” something, a noun phrase such as “reader-patience budget,” or an optional “that” omitted from a clause.
Thomas Hikaru Clark, Matias B. Martin and Kyle Mahowald, the paper’s authors, write:All of these features are associated with linguistic processing difficulty.
For the human comparisons, the team used movie and television subtitles for chat, StackExchange posts for coding, and student essays for essay writing. The AI corpus covered chat, coding-agent sessions and essays. The researchers also compared model responses with versions translated from English to Japanese and back using Meta’s NLLB-200 system, a method intended to preserve meaning while changing the wording.
Conversation and coding showed stronger patterns
The round-trip comparisons found that frontier and near-frontier models generally used more rare word sequences, improvised noun-based verbs and inanimate-subject constructions than the paraphrases. Similar patterns appeared for omitted “that” in object relative clauses. Longer dependencies and stacked nouns also tended to appear more in chat and coding outputs than in their translated paraphrases.
The results varied by task and model. The differences were smaller in single-prompt student-style essays than in chat and coding, while the Llama models and two Olmo models showed smaller changes under round-trip translation on several measures. The researchers report that Claude Opus 5.5 scored lower than Opus 5 on nearly every difficulty-linked measure they compared, across the settings tested.
A length penalty changed a smaller model’s choices
To test one possible explanation, the team fine-tuned Qwen2.5-1.5B-Instruct using reinforcement learning that rewarded correct responses while penalizing length. The task asked the model to turn structured event descriptions into English sentences. Some pairs expressed the same meaning with different lengths, such as “the book the professor wrote” and “the book that the professor wrote.”
Without a penalty for drifting from the base model, every tested run with a nonzero length penalty converged on the shorter option for the two trained “that” constructions. The authors report that the shorter version accounted for more than 98% of choices in those cases. The model also favored omitting “that” in related constructions not used in training, but transfer to other types of shorter constructions was limited.
The experiment offers evidence that a length penalty can shift grammatical choices in one small model and one narrowly defined task. It does not establish that the same mechanism caused the patterns observed in commercial frontier models. The authors say they rely on proxies because directly tracing the causes of language changes during frontier-model post-training is not feasible at the scale they studied.
The study measures features, not readers
The paper is a preprint and has several stated limits: it analyzes English only, covers a subset of commonly used models and relies on a limited number of chat topics and code repositories. The authors also say that greater exposure could make people more familiar with some of the constructions, even as they argue that many of the patterns pose challenges for intelligibility.
That distinction matters for interpreting the findings. The study identifies structures associated with processing difficulty in earlier research; it does not show that readers in a new experiment struggled with these particular samples. A direct test of reading time and comprehension would establish whether the measured language patterns translate into slower reading or more mistakes for people.