STRUCTURALCOST: A controlled reading time dataset for modeling human sentence processing difficulty

AI summary

Overview

Research area: Computational psycholinguistics and NLP — specifically, whether language models are cognitively plausible models of human sentence comprehension, tested through reading-time prediction.

Technical level: Intermediate. The paper assumes familiarity with surprisal, self-paced reading, and mixed-effects modeling, but its core argument is accessible to anyone who knows basic NLP.

Scope in one sentence: The paper releases a large, experimentally controlled self-paced reading dataset (475 participants) that isolates the processing cost of long-distance subject–verb dependencies, and uses it to show that n-gram models, state-space models, and transformers all capture global predictability but underestimate the memory-based integration cost humans pay.

What This Paper Is About

Psycholinguists have long claimed that human reading slows down when a subject and its verb are separated by a long stretch of intervening material, because working memory must hold the subject until it can be integrated. Existing evidence for this claim comes either from small laboratory experiments (well controlled but low-powered) or from large naturalistic corpora like Natural Stories and Dundee (large but confounded, since frequency, word length, and structure vary together). The paper builds a dataset that combines experimental control with NLP-scale data, then asks whether the surprisal of modern language models reproduces the graded difficulty profile humans show at the main verb.

Key Contributions

1. StructuralCost, a released dataset. 475 participants and 40,800 self-paced reading observations (27,103 critical sentences and 13,663 control sentences after filtering), with 450 unique sentences total — 60 per condition across six critical conditions (360 sentences) and 30 per condition across three control conditions (90 sentences). Individual working memory capacity scores were also collected via an operation span test.

2. A controlled design that separates distance from structure. Critical sentences hold the main subject–verb pair constant while varying dependency length (0, 4, or 9 intervening words) and syntactic type (no clause, prepositional phrase, subject relative clause, object relative clause), so that linear distance and hierarchical embedding can be dissociated.

3. A large-scale replication of a low-powered psycholinguistic finding. Reading times at the main verb increase with dependency length, and at a fixed 4-word distance the three conditions pattern differently (PP < SRC < ORC), showing that syntactic structure beyond linear distance drives difficulty.

4. Evidence that surprisal-based fit is a weak test of cognitive plausibility. Global surprisal–RT correlations cluster tightly around r = 0.51–0.53 across all architectures and sizes, but at the main verb the picture diverges sharply, and no model reproduces the magnitude of the human integration cost.

Main Findings

• Reading time rises with dependency length, then plateaus. Collapsing across clause types, mean verb RTs go from 490 ms at 0-word dependencies to 544 ms at 4 words and 528 ms at 9 words. The 0-vs-4 and 0-vs-9 contrasts exceed the Region of Practical Equivalence (ROPE), but the 4-vs-9 contrast does not (d = 0.05, 95% CI [0.02, 0.07]) and is treated as negligible.

• Structure matters independently of distance. At the same 4-word dependency length, PP shows a negligible effect (d = −0.08, CI [−0.12, −0.04]), SRC a small effect (d = −0.15, CI [−0.19, −0.12]), and ORC the largest (d = −0.19, CI [−0.24, −0.15]), against a baseline mean of 490 ms. ORC mean RT is 583 ms versus PP at 512 ms — a 70 ms penalty at matched surface distance.

• Dependency length survives psycholinguistic controls. A mixed-effects model adding dependency length over a baseline with word length and log frequency improved fit (χ²(1) = 21.6, p < .001); word length (β = 0.012, t = 5.04), log frequency (β = −0.009, t = −4.10), and dependency length (β = 0.0065, t = 4.73) were all significant. This corresponds to roughly 3 ms per additional intervening word.

• An unexpected reversal at double embedding. Doubly embedded conditions (2×SRC, 2×ORC) were consistently faster than their singly embedded counterparts: 2×SRC mean 504 ms vs. SRC 539 ms, and 2×ORC 553 ms vs. ORC 583 ms. The differences (Δd = 0.04 and Δd = 0.05) are negligible relative to the ROPE. No model predicts this reversal; the authors flag it as an open question.

• Global surprisal alignment is uniform and uninformative. Across all word positions, Spearman correlations between model surprisal and RT cluster at r = 0.51–0.53 with CIs of ±.02, regardless of architecture or size. Adding surprisal improves fit for all neural LMs (p < .001, χ²(1) > 846).

• Local alignment at the main verb is where models diverge. At dependency length 4, correlations drop to r = 0.31–0.39 for neural models and r = 0.14–0.15 for n-gram models (whose CIs include zero, [−.01, .29]). At dependency length 9, neural models settle at r = 0.26–0.35 with wider CIs.

• N-gram models fail entirely at the integration site. With the subject noun beyond the 5-token window, surprisal adds nothing over baseline predictors at any dependency length (p > 0.5, χ²(1) < 0.1).

• Scaling does not help. Larger models within a family do not fit human integration profiles better, and the largest variants lose correlation points relative to smaller counterparts.

• Mamba-130M is the one partial exception. It reaches r = 0.41 at dependency length 9 — the only model whose CI does not overlap with any 0-word dependency interval — approaching the global baseline and outperforming larger attention-based models. The authors suggest selective state-space compression may introduce a recency bias resembling distance-dependent memory decay.

• Separately estimated surprisal-to-RT conversion factors badly under-predict. Only four models (Mamba-130M, Pythia-70M, Pythia-1.4B, KenLM/GPT-2) yielded a significant spillover coefficient at p < .01 on Natural Stories. Applied to StructuralCost, KenLM-GPT2 predicted effects of 0.02–1.99 ms against observed effects of 14–93 ms. Pythia-70M predicted a speedup in every condition, and no model recovered the correct direction in the 2×SRC condition.

• Reader profiles show a further limitation. LMs fit slow and medium readers better than fast ones, who are described as simultaneously faster and more accurate — suggesting efficient integration strategies fall outside what surprisal captures.

Methodology in Plain English

The authors first designed sentence sets where the main subject and verb are always the same pair, and only the material between them changes. In critical sentences, that material is absent (baseline), a prepositional phrase (4 words of pure linear distance), a subject relative clause (4 words including an embedded clause), an object relative clause (4 words, same length but harder), or doubly embedded versions of the relative clauses (9 words). Control sentences vary only linear distance using prepositional phrases of 0, 4, and 9 words, giving a reference class with no extra clause. Sentences were drafted with LLaMA-3.2 and Perplexity.ai under tight constraints, then manually reviewed and corrected; all sentences were matched at 13–15 words to avoid reading-strategy differences.

Native English speakers were recruited on Prolific and read sentences one word at a time (non-cumulative moving-window self-paced reading). Participants answered comprehension questions; only those scoring at least 70% were kept, giving 475 participants. Reading times below 100 ms or above 3,000 ms were excluded as inattention or technical error (under 5% of the data). Afterwards, participants completed an operation span test of working memory capacity.

For the human analysis, the authors used a Region of Practical Equivalence test, treating differences smaller than 0.1 SD of the RT distribution (35 ms in this data) as negligible, and fit log-transformed RTs with linear mixed-effects models including random intercepts for participants, items, and presentation order.

For the model analysis, they extracted word-level surprisal (aggregating subword tokens) from nine models spanning a memory-constraint spectrum: two KenLM n-gram models with a hard 5-token window, three GPT-2 sizes (small, large, XL), three Pythia sizes (70M, 1.4B, 12B), and three Mamba sizes (130M, 1.4B, 2.8B). They then measured alignment two ways: globally (Spearman correlations across all word positions) and locally (correlations restricted to the main verb), plus nested mixed-effects comparisons adding surprisal and then dependency length. To test whether surprisal could predict the magnitude of the effect, they estimated a millisecond-per-bit conversion factor on Natural Stories, regressing RT on surprisal at the current and three preceding words plus length and frequency, then applied it to StructuralCost.

Why This Matters

Impact on research. The paper argues that the field's dominant evaluation metric — correlation between surprisal and reading times on naturalistic text — is a weak diagnostic. Because most RT variance is lexical, almost any model with good word statistics scores similarly. The paper pushes evaluation toward controlled datasets that probe theoretically motivated positions, and toward architectural interventions (limited context, recency-biased attention, capacity-constrained recurrence) rather than further scaling. It also replicates a low-powered psycholinguistic result at scale and reports a counterintuitive reversal at double embedding that current theories and models do not predict.

Real-world applications:

• Cognitive modeling and theory testing: A dataset like this lets researchers run hypothesis tests at scale that would be impractical to run in a lab, and use model failures to sharpen sentence-processing theories.
• Model evaluation and selection for human-facing text: Systems where predictability to humans matters (readability scoring, subtitle and caption timing, educational content pacing) could be benchmarked on structural difficulty rather than raw perplexity.
• Accessibility and reading-load estimation: The dissociation between distance and structural embedding could inform tools that flag text likely to strain working memory, for example for people with reduced working memory capacity.
• Architecture design: The Mamba-130M result hints that memory-constrained recurrence may yield more human-like processing profiles, which is directly relevant to anyone designing models with bounded state.

Industry relevance. Most commercial language models are evaluated on perplexity or downstream task accuracy. This paper provides an argument and a resource for adding a human-alignment axis to evaluation, particularly for products where the model must track or predict human comprehension difficulty rather than simply predict the next token. It also offers a caution against assuming that scaling improves human-likeness.

Future Directions

• Better dissociate linear and hierarchical structure. The authors state that a denser condition gradient (for example a PP+SRC combination) would help separate linear from hierarchical contributions and clarify the double-embedding reversal.
• Explain the double-embedding reversal. The paper offers within-sentence syntactic adaptation as one possible account — readers may have already processed the relevant structure once in the same sentence — but explicitly flags this as an open question rather than a settled mechanism.
• Determine whether Mamba's advantage is architectural. The authors write that whether these advantages reflect genuine architectural sensitivity or confounds in training data and tokenization remains to be established.
• Leverage the working memory capacity scores. The WMC scores from the operation span test were collected but, as the paper states, are not yet fully leveraged. (Note: the paper's limitations section is truncated at this point.)

Target Audience

Computational psycholinguists and NLP researchers working on cognitive plausibility, surprisal-based reading time modeling, or human-aligned language model evaluation. It is also relevant to psycholinguists who want large-sample evidence on dependency length effects, and to model architects interested in memory-constrained alternatives to transformers. Readers should be comfortable with mixed-effects models, likelihood ratio tests, and surprisal, though the paper's central claim — that models predict words well but do not pay the memory cost humans pay — is stated plainly enough for a broader audience.

Authors’ abstract

We introduce STRUCTURALCOST, a self-paced reading dataset of 475 participants and 40,800 observations isolating the processing cost of long-distance subject-verb dependency resolution. We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Different language models -- spanning n-gram models, SSMs, and transformers -- partially mirror this graded difficulty profile, yet underestimate the integration cost humans incur, with a gap that persists across architectures and model sizes. This suggests these models capture the predictive component of human processing but not the full integration cost that working memory imposes. STRUCTURALCOST provides data needed to drive progress toward evaluating the cognitive plausibility of language models.