The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

AI summary

Overview

Research area: Natural Language Processing / computational linguistics, at the intersection of clinical aphasiology, psycholinguistics, and LLM evaluation.

Technical level: Intermediate. The paper is primarily a benchmark design and validation paper rather than a methods paper — the subtest mechanics are simple to grasp, while the inter-rater reliability analysis requires some familiarity with Cohen's Kappa and annotation methodology.

Scope: This paper introduces and describes the Text Aphasia Battery (TAB), a text-only adaptation of the Quick Aphasia Battery for measuring aphasia-like language deficits in large language models, and validates an automated scoring protocol against expert human raters.

What This Paper Is About

Clinical aphasia batteries such as the WAB-R, BDAE, CAT, and QAB were built to probe language breakdown in humans, and they depend on pictures, speech perception, gestures, and other non-textual context that text-only LLMs simply do not have. The authors' goal is to build a benchmark that keeps the clinical structure of the Quick Aphasia Battery while stripping out everything that cannot be assessed from text alone, so that models can be evaluated for aphasia-like patterns in a standardized, scalable way. A second goal is to check whether an LLM can score that benchmark automatically, rather than requiring expert speech-language pathologists to rate every response.

Key Contributions

1. A clinically-grounded text-only benchmark. The TAB adapts core components of the Quick Aphasia Battery (QAB) into four subtests — Connected Text, Word Comprehension, Sentence Comprehension, and Repetition — developed in collaboration with speech-language pathologists and aphasiologists. It is explicitly framed as a behavioral benchmark, not a neuropsychological diagnostic tool.

2. A 19-feature binary scoring scheme for connected speech. Connected Text responses are rated on 19 aphasic features adapted from the APROCSA framework (which rates 27 features of connected speech on a five-point scale), converted here to present/absent binary scoring for text-only evaluation.

3. A validated automated evaluation protocol. Using in-context prompting with Gemini 2.5 Flash, the authors show automated scoring reaches reliability comparable to expert human raters when agreement is weighted by feature prevalence (Cohen's kappa = 0.255 for model–consensus vs. 0.286 for human–human).

4. A methodological critique of agreement aggregation. The paper shows that unweighted macro-averaging of Cohen's kappa across features with wildly different prevalence rates is misleading, and recommends prevalence-weighting for future annotation studies.

Main Findings

• Comparable reliability between the automated judge and human experts. On the prevalence-weighted metric, the Gemini 2.5 Flash system scored Cohen's kappa = 0.255 against the expert consensus, while human–human agreement was Cohen's kappa = 0.286. Both fall in a commonly described "fair" range.

• Unweighted averaging produces a distorted picture. In Table 1, the unweighted average gives the model Cohen's kappa = 0.794 versus only 0.061 for humans. The authors attribute this to features with zero positive instances in the validation set: kappa is undefined for humans when there is no variation, while the automated system trivially "agrees" by never detecting those features.

• Excluding zero-prevalence features flips the comparison. Under that third aggregation method, humans score Cohen's kappa = 0.215 and the model scores 0.381 — a different ordering than the prevalence-weighted result, which is why the authors designate prevalence-weighting as their primary metric.

• Feature prevalence varies sharply. Some features never occurred in the validation set, while Perseverations appeared in 34% of samples.

• Multiple models were lesion-simulated to generate AI data. The 255 AI samples were produced by applying lesions to GPT-2, Pythia, and Llama via parameter zeroing, swapping, or global scaling applied to components such as attention heads, feed-forward networks, and layers, at severities from 10% to 40%.

• The authors acknowledge the agreement scores are low. They attribute this to the inherent difficulty of identifying subtle linguistic features in text without prosodic or gestural cues, and suggest that automated evaluation is best used as a screening tool paired with expert review.

Methodology in Plain English

The researchers started from an existing clinical instrument, the Quick Aphasia Battery, and kept only the parts that can be delivered and answered in writing. The result is four short subtests, each with five items.

In Connected Text, the model answers five open-ended prompts (for example, "Tell me about the best trip you ever took") in three to five full sentences, and each response is checked for the presence or absence of 19 aphasic features drawn from the APROCSA framework — things like anomia, semantic and phonemic paraphasias, agrammatism, empty speech, perseverations, and overall communication impairment. In Word Comprehension, the model picks one of six options for each of five questions ("Which one is an animal with a mane?"). In Sentence Comprehension, it answers five Yes/No questions that involve passive voice, negation, and conditional reasoning. In Repetition, it must reproduce five items exactly, ranging from a single word like "house" up to "The ambitious journalist discovered where we'd be going."

Subtests 2 through 4 can be scored automatically with exact string matching or binary correct/incorrect judgment. Connected Text is the hard one, so the authors used in-context prompting with Gemini 2.5 Flash: the judge model receives a definition and an example for each of the 19 features, two annotated example transcripts with ground-truth labels, and instructions to emit a JSON object with binary 0/1 values.

For validation, they assembled 561 English text samples: 306 transcribed responses from individuals with aphasia sourced from the AphasiaBank database, plus 255 samples generated by TAB administration to lesioned language models. Five expert speech-language pathologists annotated the full set through a custom web interface, rating the presence or absence of each of the 19 features. A subset of 82 samples was annotated by two or three of the five experts, and majority vote on those items established ground truth. Agreement was then measured with Cohen's kappa under three different aggregation schemes.

Why This Matters

Impact on research. The TAB gives the field a shared, reproducible instrument for talking about language breakdown in models, rather than ad hoc error analysis. It also opens a bridge in both directions: clinical frameworks can be used to characterize model deficits, and systematically "lesioning" models and running them through the TAB enables "virtual lesion studies" that would be impossible to run in humans, letting researchers test hypotheses about the functional architecture of language.

Real-world applications (as framed by the authors, for computational research — not clinical use):

• Model evaluation: systematic assessment of LLM linguistic competencies across lexical, syntactic, and discourse dimensions.
• Ablation studies: measuring how architectural changes or parameter interventions affect linguistic integrity.
• Adversarial robustness testing: probing model stability under perturbed or degraded conditions.
• Interpretability research: mapping internal representations onto observable linguistic patterns.

Industry relevance. The automated scoring protocol matters for scale — it replaces per-item expert rating with a validated LLM judge, which is the difference between a benchmark a lab can afford to run and one it cannot. The broader lesson also travels beyond aphasia: the paper's demonstration that unweighted kappa averaging can produce a misleadingly high score for an automated judge is directly relevant to anyone building LLM-as-judge pipelines for annotation, evaluation, or data quality at scale.

Future Directions

1. Improving annotation quality. The authors propose giving annotators more surrounding linguistic context, writing more explicit guidelines with additional examples, and adding a training phase to calibrate raters, in response to the relatively low agreement scores.

2. Testing judge robustness. Automated evaluation currently relies on a single judge model and prompt; future work should evaluate robustness across multiple judge models and prompts and include human adjudication of disagreements to bound judge-specific bias.

3. Strengthening the psychometrics. The authors plan to increase subtest difficulty to mitigate ceiling effects — harder foils, a broader range of syntactic phenomena, and balanced Yes/No items — and to perform item-response analysis to estimate item difficulty and discrimination, plus analyze repetition errors under a specified normalization policy.

4. Expanding scope and coverage. Each subtest currently contains only five items, prioritizing rapid assessment over comprehensive coverage; expanded versions may be needed for fine-grained evaluation, and adapting the TAB beyond English will require careful linguistic and cultural consideration, since the current version reflects cultural assumptions from the original QAB.

Target Audience

Researchers in computational linguistics and NLP who need a standardized way to characterize language degradation in models; cognitive scientists and psycholinguists interested in using LLMs as model organisms for language processing; and clinical aphasiologists or speech-language pathologists who want to understand how clinical constructs are being translated into computational benchmarks. The paper is also relevant to engineers building LLM-as-judge evaluation pipelines, because of its analysis of how aggregation choices can distort reported reliability. It is explicitly not intended for clinicians seeking a diagnostic instrument for human aphasia — the authors state that the TAB is not validated for clinical diagnosis and should not replace the WAB-R, CAT, or QAB.

Authors’ abstract

Large language models (LLMs) have emerged as a candidate "model organism" for human language, offering an unprecedented opportunity to study the computational basis of linguistic disorders like aphasia. However, traditional clinical assessments are ill-suited for LLMs, as they presuppose human-like pragmatic pressures and probe cognitive processes not inherent to artificial architectures. We introduce the Text Aphasia Battery (TAB), a text-only benchmark adapted from the Quick Aphasia Battery (QAB) to assess aphasic-like deficits in LLMs. The TAB comprises four subtests: Connected Text, Word Comprehension, Sentence Comprehension, and Repetition. This paper details the TAB's design, subtests, and scoring criteria. To facilitate large-scale use, we validate an automated evaluation protocol using Gemini 2.5 Flash, which achieves reliability comparable to expert human raters (prevalence-weighted Cohen's kappa = 0.255 for model--consensus agreement vs. 0.286 for human--human agreement). We release TAB as a clinically-grounded, scalable framework for analyzing language deficits in artificial systems.