The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models

AI summary

Overview

Research area: Mechanistic interpretability of large language models, specifically applied to analogical reasoning (Natural Language Processing / language model internals).

Technical level: Advanced. The paper relies on attention knockout, linear probing, Patchscopes, and activation patching, and assumes familiarity with transformer layers, attention heads, and hidden-state representations.

Scope: A mechanistic study of how three families of open-source LLMs encode, transfer, and structurally align relational information when solving proportional analogies ("A is to B as C is to D") and story analogies.

What This Paper Is About

Analogical reasoning — extracting an underlying relationship from one pair of things and applying it to a new situation — is a core part of human cognition, but it is unclear whether LLMs actually encode such high-level relational concepts or only surface-level patterns. The authors probe the internal activations of LLMs to find where relational information lives, why models fail even when they have extracted it, and how models map structurally analogous stories onto each other. The goal is to characterize both the parallels and the gaps between LLM behavior and human analogical reasoning.

Key Contributions

1. An internal-mechanism account of relational encoding. The authors show that mid-upper layers of the second and third entities carry the information needed to resolve proportional analogies, and that these layers encode both attributive and relational content — with relational content showing a sharp gap between correct and incorrect cases.

2. Identification of "application" as a distinct bottleneck. Building on the observation that the link word ("as") strongly influences incorrect generations, they run two interventions showing that supplying a correct first pair changes model behavior in up to 38.4% of incorrect cases, and that patching the second entity's representations into the link yields gains of up to 38.1% in the remaining cases.

3. A structural-alignment analysis of story analogies. Using the Mutual Alignment Score (MAS) they define, they show that successful analogical reasoning is associated with stronger token-level mutual best-match alignment between source and target stories, even when lexical overlap is minimal.

4. A comparison with human cognition. The paper frames its findings as showing emerging but limited relational abstraction in LLMs, with a key divergence: for humans, mapping is comparatively easy once the relation is retrieved, whereas for LLMs application remains as much of a hurdle as encoding.

Main Findings

• Position-level information flow (attention knockout on Qwen2.5-14B). Blocking attention from the resolution token to the first entity (e1) has little impact on performance or generation. Blocking attention to the second entity (e2) or the third entity (e3) causes noticeable drops or generation fluctuations, concentrated in the mid-upper layers. Information from e2 propagates in slightly earlier layers than information from e3.

• The link matters most in failures. Information propagating from the link heavily affects generations in incorrect cases, particularly in the early to middle layers, which the authors interpret as either incorrect encoding passed to the link or a failure of the link to transfer information to the target element.

• Attributive information persists; relational information does not. Using Patchscopes, attributive information is consistently encoded in e2 and e3 and persists until the mid-upper layers before declining sharply in the upper layers — in both correct and incorrect cases. Relational information, by contrast, shows a significant gap between correct and incorrect cases. Relational information in e3 remains consistent up to the upper layers.

• Extraction failure accounts for a substantial share of errors. Replacing incorrect cases' first pairs with correct ones from the same relation raised performance by +32.3% (Llama-2-13B), +38.4% (Gemma-7B), and +35.6% (Qwen2.5-14B), with overall gains of +49.8%, +61.9%, and +55.3% respectively.

• Transfer failure accounts for additional errors. Patching e2's hidden representations into the link for the remaining incorrect cases produced gains of +25.9% (Llama-2-13B), +38.1% (Gemma-7B), and +30.5% (Qwen2.5-14B). Patching from e2 is most effective up to its middle layers; patching into the link is most effective in the link's early layers.

• Answer resolution happens late. Per the appendix, the answer is mostly resolved in the upper layers of the resolution token across all models.

• Analogical structure becomes linearly separable in middle layers. A binary linear probe distinguishing analogical from lexically similar story pairs (5-fold cross-validation) shows above-chance accuracy in early-to-middle layers and a marked increase through the middle layers, with layers 20 through 30 averaging 82.9% accuracy.

• Correct analogies show stronger mutual alignment. The relative Mutual Alignment Score (source-target MAS minus source-distractor MAS) consistently favors the target in correct cases, peaking in the middle layers. In incorrect cases, the model builds stronger alignment with the distractor story across most layers, with only a slight preference for the target in the middle layers.

• Alignment can be concrete. A sample heatmap shows analogous token pairs such as water–air and house–lungs forming mutual best matches with high similarity scores despite surface-level disparities.

Methodology in Plain English

The authors built a controlled set of proportional analogies by pulling entity pairs sharing the same relation from AnalogyKB (Wikidata subset), filtering out relations with multiple or time-varying answers, and combining pairs to generate 50k analogies. Before evaluation they applied two filters: a knowledge filter (the model must be able to answer "The author of X is" style queries for both entities) and a shortcut filter (the model must not get the answer right when the second entity or the first pair is removed). They then sampled 500 correct and 500 incorrect analogies.

To find where the answer is computed, they blocked the final token from attending to each of four earlier positions (e1, e2, the link, e3) and watched what broke. To see what those positions actually contain, they used Patchscopes — a technique that reads a hidden state back out as natural language by substituting it into a prompt — with custom prompts that either elicit entity-related descriptions (attributive information) or force the model to state the relation explicitly (relational information). Related entities for the attributive check were drawn by retrieving 100 Wikipedia paragraphs per entity with BM25 and extracting entities with en_core_web_trf.

For the two error-analysis experiments, they either swapped in correct first pairs from the same relation, or patched hidden representations from each layer of e2 into each layer of the link, reporting the best-performing layer combination. For story analogies they used the StoryAnalogy dataset in a two-option format (each question shown twice with reversed indices, correct only if the model picks the target both times), trained linear probes on attention-head activations, and computed the Mutual Alignment Score — the proportion of tokens that are each other's nearest neighbor by cosine similarity between source and candidate spans.

All experiments ran on two Nvidia A100 GPUs with 80GB memory using PyTorch v2.3.1 and HuggingFace v4.44.2 with greedy decoding.

Why This Matters

Impact on research. The paper moves beyond behavioral accuracy on analogy benchmarks and asks where and how relational information is processed. It separates two failure modes — failing to extract the relation versus failing to apply it — that are usually conflated in accuracy numbers, and it introduces the Mutual Alignment Score as a measurable proxy for structural alignment. This gives interpretability researchers a concrete set of positions, layers, and metrics to build on.

Real-world applications:

• Education and tutoring systems that use analogies to teach — knowing where models fail to transfer a relation suggests where scaffolding or intermediate prompting would help.
• Scientific and technical discovery support, where analogies from one domain are used to generate hypotheses in another; the alignment findings indicate when a suggested analogy is grounded in structure rather than surface wording.
• Robustness auditing and benchmark design, since the paper's two-option story format with reversed indices is designed to control for positional bias, and its shortcut filter targets answers reachable without doing the analogy.
• Knowledge retrieval and knowledge-base construction, because the proportional-analogy pipeline is built on a million-scale analogy knowledge base and exposes which relations models can and cannot handle.

Industry relevance. Teams deploying LLMs for reasoning-heavy tasks can use the results as a diagnostic: if a model knows both entities but still fails the analogy, the failure is likely in the transfer step, not the knowledge — a different problem requiring different mitigation. The finding that patching specific layers recovers a meaningful fraction of errors also motivates targeted intervention methods over wholesale retraining.

Future Directions

1. Improving the transfer step. The authors explicitly note that failures stem not only from representational gaps but from limitations in relational application; how to strengthen that transfer during inference or training is left open.

2. Extending the analysis beyond the relations tested. The intervention experiments are restricted to three relations ("official language of", "author of", "composer of"), so it is unclear how well the layer-level conclusions generalize to other relation types or to multi-hop analogies.

3. Connecting MAS to intervention. The Mutual Alignment Score is shown to correlate with correctness, but whether manipulating alignment directly can cause or repair answers is not established.

4. Broadening model coverage. The main results are reported for Qwen2.5-14B models because they exhibit "representative behavior," with other models in the appendix — scaling the mechanistic account to a wider range of architectures and sizes remains a next step.

Target Audience

Interpretability and NLP researchers studying reasoning in LLMs; cognitive scientists interested in comparing machine and human analogical processing; and machine learning engineers who need to diagnose why a deployed model fails at relational tasks despite having the underlying knowledge. Readers should be comfortable with transformer internals, since the paper's central evidence comes from layer-wise activation analysis and intervention experiments.

Authors’ abstract

Analogical reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical token positions can facilitate information transfer to a certain extent. Lastly, successful analogical reasoning in LLMs is marked by strong structural alignment between analogous situations, whereas failures often reflect degraded or misplaced alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.