The Astonishing Ability of Large Language Models to Parse Jabberwockified Language

AI summary

Overview

Research area: Natural Language Processing and the cognitive science of language (construction grammar, language understanding, morphosyntax), with the work carried out in a Department of Psychology.

Technical level: Intermediate. No model internals are needed, but the paper relies on embedding-based evaluation metrics and regression statistics, and it assumes some familiarity with linguistic terms such as content words, stop-words, and constructions.

Scope: The paper tests whether frontier large language models can recover the original meaning of English passages in which every content word has been replaced by a nonsense string (or, in the most extreme condition, by the word "BLANK"), and what the success of that recovery implies about how syntax, lexical semantics, and world knowledge interact.

What This Paper Is About

Jabberwocky-type text — English in which the content words are nonsense but the grammatical scaffolding is intact — is usually assumed to convey only vague, abstract meaning. This paper asks how much of a text's specific meaning can actually be reconstructed when content words are randomly replaced with nonce words, and it shows that state-of-the-art LLMs can often translate such degraded passages back into English that is close to the original. The goal is not just to demonstrate a striking model capability, but to use it as evidence about the architecture of language processing itself, arguing that syntax, lexical semantics, and general world knowledge must be tightly integrated rather than kept in separate "firewalls."

Key Contributions

1. A systematic demonstration of "Jabberwockified" text recovery. The authors build a pipeline that parses English passages into part-of-speech-aware lemmas, replaces content-word lemmas with English-like nonce words from the ARC nonword database (Rastle et al., 2002), and reattaches original suffixes, then measure how well LLMs translate the result back into English.

2. A quantified benchmark across genres and provenance. The main test uses 150 short passages from the Human-AI-Parallel corpus (Reinhart et al., 2025), sampled 50 each from spoken podcast transcripts, TV/movie screenplays, and fiction, plus an additional set of 9 matched passage pairs that contrast texts known to be in OpenAI's pretraining corpus against unpublished student essays on the same topics.

3. Five degradation variants that isolate which cues matter. Beyond the standard Jabberwocky condition, the authors test removal of modal/auxiliary verbs, removal of auxiliary verbs and prepositions, lowercasing with numerals removed, punctuation removal, and a BLANKs condition in which all non-stop-word lemmas become the string "BLANK."

4. Evidence that context and function-word density drive recovery. Sentence-by-sentence incremental testing shows that later context improves translation of earlier text, and the proportion of stop-words in a passage is the one passage-level variable that consistently predicts better meaning recovery.

Main Findings

• LLM translation of Jabberwockified text is far above chance. Mean translation accuracy across the 150 main passages was M = .59 (range .34 to .99), compared with a baseline similarity to a random same-genre passage of M = .43, t = 15.9, p << .0001. Translation similarity and baseline similarity were essentially uncorrelated (r = .05), indicating better translations were not simply more generic.

• Genre matters. Spoken texts (podcast transcripts) were significantly harder to translate than screenplays and fiction (t > 3.5, p < .001). This effect held when controlling for number of unique words and type-token ratio, neither of which predicted accuracy. A higher proportion of content words had a small negative effect on accuracy (b = -.74, t = 2.5, p = .01).

• Function words act as a "semantic fingerprint." A higher proportion of stop-words predicted better meaning recovery (b = .71, t = 3.0, p = 0.004). First-order grammatical effects — present-tense use associated with worse translations (r = -.29, p = 0.0002) and preposition use with better ones (r = .27, p = 0.001) — were subsumed by genre.

• Pretraining helps but is not required. Topic-matched pairs showed slightly higher accuracy for texts in pretraining (M = .69) than for novel texts (M = .61), but the difference was not reliable (t = .11, p = .75). Perfect recovery was only achieved for previously seen texts, and only once (an excerpt from Great Expectations, M = .99), yet being in pretraining was no guarantee of success. The abstract's example is from an unpublished student essay and scored M = .92, far above the M = .54 for a Wikipedia excerpt on the same topic.

• The BLANKs condition is nearly as good as standard Jabberwocky. Every manipulation lowered accuracy significantly (t > 3.2) except BLANKs, which removed all intra-passage repetition of nonce tokens and was statistically indistinguishable from the standard condition (t = .003), although with greater variance in quality — some texts became unrecoverable and others improved. All manipulations except BLANKs prevented the near-perfect recovery seen for Great Expectations.

• More context resolves earlier text. Incremental sentence-by-sentence prompting showed that additional context improves translation (b = .08, t > 9). In one worked example, a sentence translated at sim = .37 in isolation reached sim = .64 with 225 content words of additional context.

• Word-level translation quality follows word properties. Passage-embedding scores correlated moderately with averaged FastText word scores (r = .43 for Screenplays, .52 for Spoken, .56 for Fiction). Repetition within a passage helped (b = .05, t = 8.0). Easiest word types were auxiliaries omitted from the stop-word list, verbs, numbers, and adverbs; hardest were foreign words, conjunctions, and nouns. The ten best-translated words were: thank, matter, think, know, talk, guess, want, tell, speak, and able. The original word's Zipf frequency strongly predicted translation quality (β = .28, t = 30.9), as did within-passage occurrence count (β = .06, t = 8.4) and overall passage translation quality (β = .15, t = 23).

• Model class determines success. Base models cannot perform the task; non-reasoning instruction-tuned models do it but badly; only reasoning models succeed, with Gemini 3 Pro performing best, followed by GPT5.1. Passage-level correlations between reasoning models were r ≈ .7.

• People can do more than intuition suggests — but far less than the models. In prior work cited by the authors (De Luca and Lupyan, 2019), English-speaking adults heard a roughly 600-word Jabberwockified children's story once; 38% later correctly identified "stronk" (which replaced "cricket") as "an insect," compared with 0% of participants who had not been exposed.

Methodology in Plain English

The researchers took normal English passages and mechanically stripped the meaning out of them. Using NLTK, each passage was parsed into lemmas with their suffixes separated. Content-word lemmas were then swapped for nonce words drawn from the ARC nonword database, chosen to be as unlike real English words as possible — the mean number of phonological neighbors was 0.5, the median 0, the maximum 2, and nonce words ranged from 3 to 9 characters (M = 5.5). Suffixes were reattached, so "walked" might become "clirsed." The same nonce word was used for the same original word throughout a passage. Punctuation, capitalization, numerals, and standard NLTK stop-words (articles, pronouns, prepositions, most auxiliary verbs) were left intact in the standard condition.

Each passage was sent to a model through a single API call with a prompt describing the task ("In this passage, open-class English words were replaced with nonsense words. Translate the passage to regular English as best you can"), instructions to ignore resemblance to real English words, and a push toward specificity rather than placeholders like "something" or the nonce word itself. After the translation, the model was asked for single-word glosses of every unique nonce word.

Quality was scored by computing embedding similarity between the original and the translation using OpenAI's text-embedding-3-large. To guard against generic translations scoring well, the authors also computed a baseline similarity between each translation and a randomly chosen original passage from the same genre; the gap between the two serves as a specificity score. Word-level accuracy used FastText embeddings. The reported analyses use GPT5.1 with medium reasoning effort, tested on 150 passages (50 each from three genres), plus 9 content-matched novelty pairs, plus manipulation conditions and an incremental sentence-by-sentence condition.

Why This Matters

The paper reframes a striking LLM capability as evidence about linguistic structure. If a system can recover detailed meaning from text with almost no content words, then meaning must be distributed across morphosyntax, function words, and world knowledge to a degree that theories treating syntax as only abstractly semantic (able to signal causativity or transfer but not specific lexical content) do not predict. The authors argue this supports the connectionist view of language processing as constraint satisfaction, and the construction-grammar view of lexico-syntactic patterns carrying rich meaning, while also suggesting that any efficient language system — biological or artificial — benefits from integrating these domains rather than separating them.

Real-world applications implied by the findings:

• Robustness to noisy or degraded input — OCR errors, autocorrect damage, speech-to-text corruption, and dropped characters all resemble partial Jabberwockification, and the results suggest models can recover intended meaning rather than failing.
• Handling informal and disfluent text — the genre analysis shows the approach extends to screenplay and casual spoken material, relevant to chat, captioning, and conversational systems.
• Model evaluation and capability testing — Jabberwockified passages, especially the unseen student-essay pairs, offer a way to probe understanding with stimuli that cannot have been memorized.
• Insight into human language learning and reading — the finding that adults extract real semantic detail from a single exposure to a Jabberwockified story has implications for how much meaning comes from structure alone in education and clinical assessment.

Industry relevance centers on the finding that only reasoning models succeed, while base and non-reasoning instruction-tuned models largely fail. That is a concrete signal for teams choosing models for tasks involving corrupted, truncated, or heavily abbreviated text, and it suggests reasoning capacity — not just scale — underpins robust meaning recovery. The prominence of function words as predictors also matters for preprocessing choices such as stop-word removal, which the study implies discards much of the signal.

Future Directions

• Mechanism. The authors state they have only the vaguest outlines of how recovery works: whether the model first infers high-level information like genre and topic and uses it to constrain lower-level inferences, or whether small construction-like patterns drive inference of higher-level semantic content.
• The LLM-human gap. It remains unknown whether the superhuman gap reflects LLMs learning abstract "semantic fingerprints" unlike those people learn, or whether LLMs simply have a more powerful mechanism for using human-like patterns. The authors identify measuring the limits of human performance on this task as a clear next step.
• Languages beyond English. The work uses only English texts, and the authors note that parsing Jabberwockified text may be easier in languages with richer morphology, leaving open whether richer morphology affects meaning recovery similarly in people and models.
• Better evaluation metrics. The current embedding-similarity measure works as expected at sim < .30 and sim > .7, but in the middle range it scores gist-correct/detail-poor translations similarly to gist-wrong/detail-correct ones. The authors plan separate measures distinguishing fidelity of overall gist from fidelity of specific details.

Target Audience

This paper suits researchers in computational linguistics and NLP who study evaluation, robustness, or the relationship between syntax and semantics; cognitive scientists and psycholinguists interested in construction grammar, syntactic bootstrapping, and the limits of human comprehension of degraded language; and machine learning practitioners who need to reason about which model classes handle corrupted or low-content input. Readers without a statistics background can follow the main results from the tables and figures, but the regression analyses and embedding-based metrics reward some familiarity with quantitative methods.

Authors’ abstract

We show that large language models (LLMs) have an astonishing ability to recover meaning from severely degraded English texts. Texts in which content words have been randomly substituted by nonsense strings, e.g., "At the ghybe of the swuint, we are haiveed to Wourge Phrear-gwurr, who sproles into an ghitch flount with his crurp", can be translated to conventional English that is, in many cases, close to the original text, e.g., "At the start of the story, we meet a man, Chow, who moves into an apartment building with his wife." These results show that structural cues (e.g., morphosyntax, closed-class words) constrain lexical meaning to a much larger degree than imagined. Although the abilities of LLMs to make sense of "Jabberwockified" English are clearly superhuman, they are highly relevant to understanding linguistic structure and suggest that efficient language processing either in biological or artificial systems likely benefits from very tight integration between syntax, lexical semantics, and general world knowledge.