Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

AI summary

Overview

Research area: Natural Language Processing — evaluation of large language models, specifically evidence-conditioned relational reasoning, multilingual robustness, and the faithfulness assumptions behind retrieval-augmented generation (RAG).

Technical level: Intermediate. The paper is readable without prior work on Bloom's taxonomy, but it assumes familiarity with standard LLM benchmarking practice, multiple-choice evaluation, and the parametric versus non-parametric knowledge distinction.

Scope: A controlled, item-matched, multilingual diagnostic (Rel-Bloom) that tests whether 43 language models reason over relations supplied in the prompt rather than falling back on memorized associations, across five cognitive levels, three veracity conditions, and six languages.

What This Paper Is About

Retrieval-augmented generation assumes a model will reason over facts placed in its context instead of relying on what it memorized during pretraining, but existing benchmarks rarely separate these two behaviors — and end-to-end RAG evaluations confound the question with retrieval quality. The authors remove retrieval entirely and put the relevant fact directly in the prompt, then vary how cognitively demanding the question is, whether the stated fact conflicts with real-world knowledge, and whether entity names are present at all. The goal is to measure, in isolation, whether models actually follow supplied relational evidence, and whether the resulting difficulty structure is stable across languages.

Key Contributions

1. A controlled benchmark design (Rel-Bloom). 450 English multiple-choice questions built from 30 curated Wikidata relations, where the same relations are instantiated at all five Bloom-inspired levels (Remember, Understand, Apply, Analyze, Evaluate) and under all three veracity conditions (Factual, Counterfactual, Neutralized), giving a 30 × 5 × 3 = 450 full crossing that allows level and condition effects to be compared on matched relations.

2. A large multilingual evaluation. 43 models — 41 open-weight models from seven families analyzed as eight series (Qwen2.5 Base, Qwen2.5 Instruct, Qwen3, Qwen3.5, Gemma, LLaMA, Granite, DeepSeek), plus two Gemini models (Gemini 3 Flash, Gemini 3.1 Pro) — evaluated in Arabic, Chinese, English, Hindi, Persian, and Spanish under two settings (hybrid cross-lingual and fully translated), yielding 226,800 predictions.

3. A pre-specified hypothesis framework. Ten hypotheses (not publicly preregistered) covering scale, Bloom level, veracity, relation usage frequency, and cross-language stability, tested with relation-clustered sign-flip tests, cluster-bootstrap intervals, permutation tests, and Holm correction.

4. Empirical findings on where reasoning breaks. A sharp Apply-to-Analyze drop in every open-weight series but not in Gemini, a persistent Evaluate bottleneck even for near-ceiling models, and cross-language stability of item- and relation-level difficulty rankings.

Main Findings

• A cognitive "cliff" between Apply and Analyze in open-weight models. Apply exceeds Analyze in all eight open-weight series in both settings. The magnitude varies sharply: at the Large bucket, Granite shows a 47.2-percentage-point drop, while Qwen3.5 drops only 7.2 points. The cliff is described as neither scale-invariant nor monotonic with model size.

• Gemini models do not show the Apply-to-Analyze drop. Neither Gemini model exhibits the drop observed across the open-weight series, and Gemini 3.1 Pro shows no Apply-to-Analyze drop at all.

• Evaluate is a separate, universal bottleneck. Gemini 3.1 Pro averages approximately 99.5% on the other four levels but only 60.7% on Evaluate — a gap of about 39 percentage points. The Evaluate gap appears in all eight open-weight series and in both Gemini models.

• Scaling helps, but diminishing returns are not universal. In the fully translated setting, Medium significantly exceeds Small in all eight open-weight series and Large significantly exceeds Medium in seven, with DeepSeek the only exception. For example, Qwen2.5 Instruct gains 23.8 points from Small to Medium and 6.1 points from Medium to Large. The diminishing-returns hypothesis holds in five of eight series fully translated and six of eight in the hybrid setting.

• Counterfactual contexts are generally the hardest condition, but the full ordering fails. The predicted Neutralized > Factual > Counterfactual ordering is common directionally but statistically supported in only a few size buckets and in no series as a whole; Neutralized does not consistently beat Factual and reverses in several high-capability models, including Qwen3.5, Gemma, and both Gemini models.

• Removing entity names can help at higher cognitive levels. In the fully translated setting, the Neutralized-over-Factual advantage at Analyze and Evaluate (relative to Remember and Understand) is significantly larger in the Small bucket for seven of eight open-weight series. The effect weakens or reverses in stronger series, including Gemini 3.1 Pro.

• The Factual–Counterfactual gap widens with cognitive demand most clearly for Gemini. Both Gemini models show significant widening despite near-ceiling performance on the lower-complexity levels; for open-weight models the expected pattern appears directionally in several series but is statistically supported in only a few size buckets.

• Relation usage frequency effects are inconclusive. No open-weight series satisfies the interval-based equivalence criterion in the fully translated setting. Gemini 3.1 Pro shows a small observed spread, while Gemini 3 Flash lies at the five-point boundary under the rounded result. The authors state this says nothing about pretraining exposure, since the tiers reflect Wikidata property counts.

• Difficulty rankings transfer across languages. Item-level cross-language stability is significantly positive for every series, size bucket, and both Gemini models in both settings — mean pairwise correlation of approximately ρ = 0.54 for Gemini 3 Flash (moderate) and ρ = 0.74 for Gemini 3.1 Pro (strong). Relation-level rankings show the same pattern.

• Errors are consistent with memory competition. In matched Neutralized–Factual pairs where Qwen2.5-32B-Instruct succeeds under Neutralized but fails under Factual input, the wrong answers tend to be salient attributes of the named entity rather than the requested relation (e.g., answering "Returned safely to Earth" instead of "Recovered by USS Hornet" for Apollo 11).

Methodology in Plain English

The authors start from 30 Wikidata relations, stratified into High, Medium, and Low usage tiers (10 relations each) using MediaWiki property-frequency counts as a proxy for how often a relation appears in Wikidata. For each relation they pick one representative factual triple (head, relation, tail). GPT-5 was used to verbalize the fact and help draft level-specific multiple-choice questions; every generated question, option, and answer key was then manually reviewed for factual correctness, preservation of the intended relation, answer uniqueness, and distractor plausibility. As an extra check, Gemini and Grok independently flagged disagreements with the provided answers, and ambiguous items were revised and re-evaluated.

Each triple is turned into questions at five levels, each defined by the operation it requires: extracting a stated value (Remember), selecting a paraphrase (Understand), applying an abstract relation schema to a new statement (Apply), discriminating the target relation from near-miss alternatives (Analyze), and selecting the strongest justification grounded in the relation's defining criterion (Evaluate). The sixth Bloom level, Create, was dropped as incompatible with controlled multiple-choice grading.

Three matched veracity variants then probe different sources of answer evidence: Factual uses the real entities, Counterfactual replaces exactly one argument of the triple while preserving entity type and relation context (creating conflict with what a model likely knows), and Neutralized swaps entity mentions for typed placeholders such as [PERSON 1], stripping entity-identity shortcuts without introducing a conflict.

The English items are then evaluated in six languages under two settings. In the hybrid cross-lingual setting the fact and question stay in English while only the answer options are translated; in the fully translated setting everything is presented in the target language while preserving the relation, the level operation, the veracity condition, the answer key, and option order. Matched accuracy contrasts are aggregated within the 30 source relations and evaluated with one-sided relation-cluster sign-flip tests plus 95% cluster-bootstrap intervals, with Holm correction applied within hypothesis-specific families. Cross-language stability is assessed through 15 pairwise Spearman correlations among the six languages, summarized by their mean and tested with a sign-flip test (item level) or a relation-identity permutation test (relation level).

Why This Matters

The paper isolates a component that RAG systems silently depend on: the ability to reason over a relation stated in context rather than substituting a memorized association. Its results suggest that high aggregate accuracy can mask a system that checks facts against a question but cannot identify which property justifies the answer — and that scaling model capability does not by itself fix this.

Real-world applications:

• Enterprise and domain RAG systems (legal, medical, financial) where retrieved passages must be followed faithfully and where entity names in the source document may actively mislead the model.
• Knowledge-graph question answering, where a system must interpret the specific relation carried by a triple rather than a plausible-sounding attribute of the subject entity.
• Multilingual deployment and localization QA, where the finding that difficulty rankings transfer across languages suggests translated item sets can act as an initial screen before costly per-language evaluation.
• Benchmark and audit design, offering a template for evaluating context faithfulness with counterfactual and entity-neutralized variants rather than factual passages alone.

Industry relevance: The paper's three recommended practices — report accuracy separately by the operation a question requires, include counterfactual and entity-neutralized variants of retrieved passages, and use translated item sets as a first multilingual screen while still measuring language-specific accuracy — translate directly into evaluation protocols for teams shipping retrieval-augmented products. The released resources are available on Hugging Face at the QCRI space named in the paper.

Future Directions

• Add multiple triples per relation. With one source triple per relation, the 30 relation clusters give limited statistical power for small effects (notably the usage-frequency hypothesis) and limit generalization to unseen triples of the same relations.
• Vary distractor plausibility within a fixed level. The authors note that Evaluate distractors are by construction more plausible than Analyze distractors, so separating a "different operation" account from a "harder alternatives" account requires items that hold the level fixed while varying distractor plausibility.
• Measure memorization directly. The paper does not measure whether each model has memorized each source fact, so Counterfactual effects are interpreted as sensitivity to likely prior-knowledge conflict rather than verified memorization.
• Move to end-to-end RAG with noisy retrieval and open-ended generation. The current design deliberately removes retrieval and restricts output to multiple-choice, and the error analysis uses a single checkpoint (Qwen2.5-32B-Instruct), both of which the authors flag as boundaries for future work.

Target Audience

Researchers and practitioners working on LLM evaluation, retrieval-augmented generation, knowledge-graph question answering, and multilingual model robustness. It is most useful for evaluation leads and applied scientists who need to diagnose where a RAG system's reasoning fails rather than just how often it is correct, and for cognitive-science-oriented NLP researchers interested in whether Bloom-inspired task hierarchies actually produce the ordering they imply. Readers without background in LLM benchmarking will find the experimental design accessible but the statistical framework (relation-clustered inference, sign-flip tests, equivalence margins) demanding.

Authors’ abstract

Retrieval-augmented generation (RAG) assumes that language models can reason over relations stated in their context rather than fall back on associations memorized during pretraining. We study this evidence-conditioned relational reasoning in isolation, with the facts supplied directly in the prompt rather than retrieved. \textsc{Rel-Bloom} is a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations; the same source relations are instantiated at five Bloom-inspired levels (Remember, Understand, Apply, Analyze, Evaluate) and under three veracity conditions (Factual, Counterfactual, Neutralized). Across 43 models, six languages, and two multilingual settings (226,800 predictions), three findings stand out. First, accuracy drops sharply from Apply to Analyze in every evaluated open-weight model series, reaching 47.2 percentage points for the Large Granite bucket, but not for either Gemini model. Second, Evaluate items remain a bottleneck even for models near ceiling on the other task types: Gemini~3.1 Pro averages 99.5\% on the other four levels but 60.7\% on Evaluate. Third, item- and relation-level difficulty rankings are consistently preserved across languages, indicating a difficulty structure partly shared across languages within this benchmark. Across knowledge conditions, Counterfactual contexts are generally harder than Factual ones, while removing entity identities can improve higher-level performance in some model regimes. Greater model capability therefore does not by itself ensure reliable reasoning over supplied relational evidence. Relevant resources are available at: https://huggingface.co/spaces/QCRI/non-parametric-memory-relational-reasoning.