Understanding or Memorizing? A Case Study of German Definite Articles in Language Models

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability of language models, with a focus on morphosyntax in German.

Technical level: Intermediate. The paper uses a gradient-based interpretability method (Gradiend) plus statistical evaluation, so some familiarity with transformer training and gradient analysis helps, but the underlying question is stated plainly.

Scope: A single-paradigm case study of how six language models internally represent German definite singular articles, testing whether their behaviour reflects abstract grammatical rules or memorized surface associations.

What This Paper Is About

German definite singular articles depend jointly on grammatical gender and case, and they are highly syncretic — the same surface word (for example der) can serve several gender–case combinations. This makes them a useful probe for a broader question: when a language model predicts the right article, is it applying an abstract grammatical rule (the paper's hypothesis H2, rule-encoding) or reproducing memorized surface-level token–context associations (hypothesis H1, memorization)? The authors train a small gradient-based encoder–decoder, Gradiend, for specific gender–case article swaps, then check whether the learned updates stay confined to the trained setting or leak into other grammatical settings.

Key Contributions

1. Applying Gradiend to German article morphology. The authors train one Gradiend model per article transition between two gender–case "cells" that differ in exactly one dimension (gender at fixed case, or case at fixed gender), covering all two-dimensional and one-dimensional transition groups listed in Table 2.

2. A three-angle evaluation protocol. They analyse (i) the encoded scalar feature h and whether it separates the two swap directions from neutral inputs, (ii) probability shifts after applying the decoded update to the base model under a language-modelling-preservation constraint, and (iii) parameter-space overlap of the most important decoder weights across transitions.

3. An explicit test of three possible generalization patterns. They formalize local rule-based (LR), generalized rule-based (GR), and spillover (SO) patterns (Figure 4) and check which pattern the observed probability shifts match.

4. Evidence against a strictly rule-based account. Statistically significant generalization across gender and case, combined with substantial neuron overlap between conceptually different transformations, argues against H2 and supports at least partial memorization (H1).

Main Findings

• The learned scalar separates transition directions. All models reach correlations of at least 50% between the encoded value h and the expected labels (±1 for the two directed transition tasks, 0 for identity pairs), with German encoder-only models often exceeding 90%. Table 3 reports the Pearson correlations scaled by 100; for example GermanBERT ranges from 90.7 to 98.4 across the listed Gradiend variants.

• Identity pairs map near zero only for German encoder-only models. By construction, non-targeted cells should produce no update. German encoder-only models show h ≈ 0 for these identity pairs, while EuroBERT shows larger deviations — the authors suggest multilingual representations encode German article distinctions less sharply. Decoder-only models show a similar pattern and additionally spread identity pairs across much of [-1, 1], which the authors attribute to the lightweight article classification head they added.

• Neutral control behaves as expected. Gradients from the gender- and case-cue-minimized D_Neutral dataset map consistently close to zero across models and configurations.

• Updates leak beyond the trained cell. Under a constraint of preserving at least 99% of the base model's Language Modeling Score on D_Neutral, probability changes (Δℙ) are typically below 1%, but effects are statistically significant on article datasets that share the same source article, and generally stronger than on D_Neutral. Table 4 reports Δℙ (scaled by 100) with Cohen's d and significance for GermanBERT; changes on D_Neutral are mostly not significant, with Cohen's d between 0.01 and 0.05 in the rows shown.

• Mixed evidence on which generalization pattern holds. The heatmaps partially match the generalized rule (GR) pattern, but some GR-predicted cells are neutral or opposite-signed (for example GermanBERT on D^Fem_Dat for der), and other effects appear in cells no clean grammar-preserving rule predicts. The two clearest GR contradictions — ℙ(der) on D^Fem_Dat and D^Fem_Gen — align instead with the spillover (SO) pattern, which explains most of its predicted cells in terms of probability-change direction. The authors conclude the patterns reflect "a mixture of structured generalization and surface-article-linked effects."

• Transitive spillover appears across models. Cells sharing a surface article with the same gender or case (such as D^Masc_Dat / D^Neut_Dat and D^Masc_Gen / D^Neut_Gen) often behave similarly. For instance, G^{Fem, Masc}_Nom die → der increases ℙ(des) on D^Masc_Gen (GR-consistent) and concurrently on D^Neut_Gen.

• One model breaks the pattern. LLaMA is the only model that does not show the ℙ(der) increase on D^Fem_Dat / D^Fem_Gen, which the authors say may indicate "a trend of less memorization in larger models" (LLaMA is 3.2B parameters in this study).

• Broad capabilities stay intact. Mostly unchanged SuperGLEBer scores (a German NLP benchmark of 29 tasks) support the claim that the observed shifts are not caused by general degradation. In Table 4, the base GermanBERT scores 70.7 ± 0.4 and the Gradiend-modified variants score 70.1 ± 0.4 to 70.2 ± 0.4.

• Parameter overlap is high within article groups. Using Top-k decoder weights with k = 1000, groups with at least two variants realizing the same surface article pair show consistently high overlap (more than 75% on average), whereas a control group of four variants spanning disjoint surface articles is much lower (mean 38.9%). This suggests that gender–case transitions of the same articles rely on a shared subset of parameters rather than disjoint, transition-specific mechanisms.

Methodology in Plain English

The authors ask a counterfactual question for each grammatical setting: if the article in this context were different, which parameters would have to change to make the model prefer the alternative article?

1. Build per-cell datasets. Each gender–case cell gets its own dataset (denoted D^g_c) built by filtering German Wikipedia sentences with spaCy, keeping only sentences where spaCy identifies a definite singular article with the desired gender and case. Dataset sizes range from 19K to 61K, split 80% train / 10% validation / 10% test. A separate D_Neutral dataset, drawn from the Wortschatz Leipzig German news corpus and filtered to exclude determiners, definite or indefinite articles, and third-person pronouns, minimizes gender and case cues and serves as a reference.

2. Define factual and alternative targets. For a masked article, the factual target is the grammatically licensed article; the alternative target is the article of a paired gender–case cell. The difference between the corresponding gradients is ∇^ΔW_m = ∇^FW_m − ∇^AW_m. Non-targeted cells are included as identity pairs, where factual and alternative gradients are identical by construction, enforcing a do-not-change constraint.

3. Train Gradiend. A one-dimensional bottleneck maps gradient input to a scalar h = tanh(W_e^⊤∇_inW_m + b_e) ∈ [-1, 1], and a linear decoder reconstructs an update direction h·W_d + b_d, trained with a squared reconstruction loss. Departing from the original Gradiend work, the input is the alternative gradient rather than the factual gradient, because article prediction is often so confident that factual gradients are near zero and training becomes unstable.

4. Intervene and measure. The base model is modified as W̃_m = W_m + α · dec(h^⋆), with h^⋆ = ±1 selecting the transition direction. Among candidate learning rates, only those preserving at least 99% of the base model's Language Modeling Score on D_Neutral are kept, and α^⋆ is the candidate maximizing the mean probability of the target article on the relevant dataset. Effects are reported as mean probability changes with Cohen's d and permutation-test significance (Benjamini–Hochberg FDR correction applied per model).

5. Compare parameter importance. The absolute decoder weights |W_d| define importance; the Top-k = 1000 weights are compared across Gradiend variants via set overlap.

6. Handle decoder-only models. Because German article choice depends on the noun's gender, left-to-right context alone is insufficient, so the authors add an MLM-style article classifier for bidirectional conditioning, and a custom small MLM head for Gradiend training on decoder-only models.

Why This Matters

Impact on research. Mechanistic interpretability work often moves from probing results to claims about internal mechanisms, and the paper argues probing accuracy alone is not mechanistic evidence. By showing that a transition-specific intervention spills over to grammatically unrelated cells that share a surface article, the study provides a concrete counterexample to the assumption that high grammatical accuracy implies abstract rule use — and it points to a shared parameter subset for same-article transitions rather than cleanly separable per-transition circuitry. It also extends a primarily English-centric literature into a morphologically rich, syncretic paradigm, where case is not marked on nouns and article forms jointly encode case and gender.

Real-world applications:

• Grammar and writing assistance for German. Tools that correct or suggest articles rely on models whose internal article knowledge may be partly memorized from frequent contexts, which matters for uncommon nouns and constructions.
• Machine translation into gendered languages. When a model's article decision is tied to surface associations rather than gender/case variables, systematic errors can appear in less frequent gender–case combinations.
• Model editing and steering. The paper's intervention procedure is a form of targeted model editing; knowing that edits leak to same-surface-article cells informs how such edits should be scoped and validated.
• Evaluation design. Morphological productivity and nonce-word tests are complementary to accuracy benchmarks; this study supports the view that surface-level agreement can coexist with non-uniform generalization.

Industry relevance. Teams shipping German-language NLP — search, assistants, content generation, translation — depend on whether article behaviour generalizes to rare nouns and cases. The finding that Top-k parameters overlap more than 75% on average within an article group, versus 38.9% in the disjoint-article control, gives a concrete handle for auditing and controlling shared representations. The language-modelling preservation constraint used here is also a practical recipe for evaluating any intervention: check the benchmark cost, not just the targeted effect.

Future Directions

• Extend beyond a single paradigm. The authors frame this as a case study on German definite singular articles only; the same three-pattern test could be applied to other syncretic or agree­ment phenomena and to other morphologically rich languages.
• Test the scaling hypothesis. LLaMA was the only model that did not show the ℙ(der) increase on D^Fem_Dat / D^Fem_Gen, which the authors suggest may indicate less memorization in larger models. Whether memorization reliably decreases with scale is left open.
• Improve decoder-only analysis. The authors attribute unstable encodings and spread identity pairs in decoder-only models to their lightweight MLM-style classification head and custom prediction head, so a better bidirectional conditioning setup may be needed before drawing firm conclusions for such models.
• Probe the source of the mixture. The probability-shift patterns matched neither a clean grammar-preserving rule nor unrestricted spillover; isolating which training-data distributional properties produce the surface-article-linked effects remains an open question.

Target Audience

Researchers in NLP interpretability and mechanistic analysis; computational linguists working on German and other morphologically rich or syncretic languages; and practitioners who build or evaluate grammatical agreement systems, model-editing tooling, or German-language applications. Readers looking for a worked example of testing "understanding versus memorization" with causal interventions rather than probing accuracy will find it most directly useful.

Authors’ abstract

Language models perform well on grammatical agreement, but it is unclear whether this reflects rule-based generalization or memorization. We study this question for German definite singular articles, whose forms depend on gender and case. Using GRADIEND, a gradient-based interpretability method, we learn parameter update directions for gender-case specific article transitions. We find that updates learned for a specific gender-case article transition frequently affect unrelated gender-case settings, with substantial overlap among the most affected neurons across settings. These results argue against a strictly rule-based encoding of German definite articles, indicating that models at least partly rely on memorized associations rather than abstract grammatical rules.