Kinship Data Benchmark for Multi-hop Reasoning

AI summary

Overview

Research area: Natural Language Processing — evaluation benchmarks for multi-hop reasoning and cultural knowledge in large language models.

Technical level: Intermediate. The core idea is accessible to anyone familiar with LLM benchmarking, but the paper assumes some familiarity with kinship terminology (bifurcate merging, cross-cousins, lineage skewing), knowledge-graph encoding (RDF/OWL, SPARQL), and standard evaluation protocols (exact match, chain-of-thought, few-shot prompting).

Scope: The paper introduces KinshipQA, a procedurally generated benchmark that tests whether LLMs can apply the cultural rules of seven documented kinship systems — not just chain together biological family facts — and shows that cultural rule application collapses far faster than biological chaining as reasoning depth grows.

What This Paper Is About

Existing kinship reasoning benchmarks such as CLUTRR only cover the Eskimo system, which is "descriptive": it maps one term per biological relationship and largely matches the English default. That makes it impossible to tell whether a model is genuinely reasoning about kinship or simply recalling familiar English family vocabulary.

This paper builds a benchmark that covers seven anthropologically documented kinship systems, including ones where cultural rules override biology — for example, in Hawaiian kinship your mother's sister is classified as "mother," and in Crow kinship your father's sister's son is classified as "father." The goal is to measure whether LLMs can apply these culture-specific reclassifications, and to separate that ability from ordinary multi-hop chaining.

Key Contributions

1. A procedurally extensible benchmark generator. A five-stage simulator builds multi-generational family trees under culture-specific marriage constraints, with a tunable horizon (50, 150, or 200 simulated years) that scales population size and chain depth while eliminating exact-instance overlap with pretraining data. Trees are encoded as RDF/OWL with a dual namespace — family: for biology, kin: for culture — so biological facts and cultural categories are never conflated.

2. KinshipQA, a 3,931-question benchmark. Spanning seven kinship systems (Eskimo, Sudanese, Hawaiian, Iroquois, Dravidian, Crow, Omaha), four reasoning categories, and reasoning depths from 1 to 6 hops. Ground truth is computed by SPARQL queries against the released ontology, so a deterministic executor returns the gold answer on 100% of questions by construction — the LLM gaps cannot be explained by gold-label inconsistency.

3. Evidence that cultural disambiguation is a distinct burden from multi-hop chaining. The paper documents a 40.9% exact-match drop when reasoning shifts from biological multi-hop to culturally marked classification, holding for every one of the five non-descriptive systems and largest for the two skewing systems (Crow, Omaha).

4. Two diagnostic interventions. A fictional-rule control (swapping system labels and kin terms for invented strings) raises accuracy by 6.1%, implicating familiar English surface forms. An in-context-rule probe helps skewing systems by +17.1% but hurts non-skewing systems with high baselines by −13.4%, consistent with a missing skewing prior alongside rule interference where the model already has a working approximation.

Main Findings

The cultural override gap is large and universal. Across the five non-descriptive systems, average Cat. 4 (cultural disambiguation) accuracy is 50.4%, against 91.3% on Cat. 1–3 (biological reasoning) over the same systems — a 40.9% EM drop (31.8% under F1). Every one of the five systems shows the drop.

Skewing systems are hardest. Omaha (patrilineal skewing) bottoms out at 32.6% Cat. 4 EM and Crow (matrilineal skewing) at 47.0%, versus 60–62% for Hawaiian, Iroquois, and Dravidian. The authors are careful to claim only the descriptive-versus-non-descriptive contrast and the skewing penalty, noting the ordering among Hawaiian, Iroquois, and Dravidian is not stable.

The gap compounds with depth. At 5–6 hops, cultural override falls to 10.6% EM while biological composition over the same chains holds at 58.6%. The cultural gap widens from 37.6% at 2-hop to 66.5% at 4-hop.

It is not a prompt or decoding artefact. The gap persists under zero-shot direct, zero-shot chain-of-thought, and few-shot CoT prompting, under stochastic decoding (T=0.7, n=5), under paraphrase of the rule wording, and at frontier scale (Claude Opus 4.7 reaches 100% on descriptive systems but only 69% on Other-5).

It is not a hop-count artefact. A hop-matched comparison of Cat. 2 (biological) against Cat. 4 (cultural) at 2 and 3 hops shows the gap widening with depth (38.0% → 56.3%), the opposite of what a lookup-disruption explanation would predict.

Humans solve the questions once the rule is supplied. A two-annotator baseline (n=280, inter-annotator agreement 96.8%) reaches 89.0% on Other-5 Cat. 4 with the same one-sentence rule the in-context-rule probe uses, versus 50.7% for LLMs — a 38.3% gap. The questions are intelligible and reliably solvable.

Errors are dominated by biological-default leakage. Of the 642 substantive non-EM Cat. 4 errors, 83.0% are either bare-default substitution (returning the Eskimo/English answer, 51.1%) or over-inclusion (31.9%); only 15.3% are wrong-disjoint.

Surface form matters, and the rule-recall picture is asymmetric. The fictional-rule control's +6.1% gain is an interaction, not additive — swapping either the system label or the kin terms alone is null; swapping both helps. The in-context-rule probe splits cleanly by typology and baseline: skewing systems uniformly benefit (mean +17.1%, 6/6 model-system pairs), while non-skewing systems already above roughly 60% baseline are uniformly hurt (mean −13.4%, 4/4 pairs).

Methodology in Plain English

The authors built a simulator rather than scraping or hand-writing questions. It works in five stages:

1. Population simulation. Starting from four seed individuals, the simulator grows multi-generational family trees over a chosen horizon. Each kinship system imposes its own marriage constraints — Eskimo prohibits only sibling marriage; Dravidian prefers cross-cousin marriage; Crow and Omaha enforce clan-based prohibitions under matrilineal and patrilineal descent. A 50-year run yields 60–70 individuals; 150- and 200-year runs yield 60–360.
2. RDF/OWL encoding. Trees are stored as a knowledge graph with two namespaces, separating biological relationships from cultural classifications.
3. Question generation. Templates organised by category and hop depth produce questions that map biological paths (e.g. mother → sister → child) onto system-specific terms. Ground truth comes from SPARQL queries, not from human annotation — which is how the authors guarantee that a symbolic executor gets 100% of questions right.
4. Proof graph extraction. The minimal subgraph supporting each question is isolated.
5. Natural-language serialisation. That subgraph is rendered into a short English paragraph ("Alice is the mother of Bob"), producing the context the model actually reads.

Questions fall into four categories: Cat. 1 is a one-hop lookup; Cat. 2 chains 2–4 biological steps; Cat. 3 adds counting and filtering; Cat. 4 asks the model to apply a named cultural system's classification rules. Crucially, Cat. 4 names the system but never states the rule, so it tests application of a latent cultural prior rather than application of a supplied rule — which is why the separate in-context-rule probe matters.

Six LLMs were evaluated under greedy decoding with identical prompts: three open-source 27–32B models (Qwen3-32B, Gemma3-27B, DeepSeek-R1-Distill-Qwen-32B) and three closed-source mid-tier APIs (GPT-4o-mini, Claude-Haiku-4.5, Gemini-2.5-Flash). Chain-of-thought and few-shot variants were run on the five models with stable coverage. Because the 512-token generation budget truncated reasoning models before their answer line on the 5–6 hop sweep, every empty prediction was re-run at 8,192 tokens.

Why This Matters

Impact on research. The paper makes a methodological point that generalises beyond kinship: benchmarks built on a single cultural typology — especially one that happens to match the English default — can overestimate how reliably models perform a task. Because KinshipQA's ground truth is machine-verified by construction and the simulator horizon prevents exact-instance overlap, the reported gaps can be attributed to interpretation and reasoning rather than contamination or evaluator drift. The pairing of a benchmark with two targeted interventions (fictional-rule control, in-context-rule probe) is also a template for diagnosing why a model fails rather than just that it fails.

Real-world applications:

• Cross-cultural NLP systems. Translation, dialogue, and information-extraction tools that operate across societies with different kinship terminologies — a domain where a wrong classification has concrete social consequences.
• Genealogy and family-history platforms. Automated reasoning over large family graphs where a user's cultural kinship system determines the correct label for a relative.
• Legal and administrative contexts. Inheritance, guardianship, and immigration proceedings where kinship classification carries legal weight and varies by community.
• Evaluation design generally. Any benchmark testing rule application in a domain with culturally contingent conventions can borrow the "name the rule but don't state it, then probe with the rule supplied" structure.

Industry relevance. The finding that supplying a rule hurts models that already have a working internal approximation is directly actionable for prompt engineering and RAG pipelines: adding retrieved rules to context is not uniformly beneficial, and can interfere with knowledge the model already encodes. The asymmetric results across typologies also argue against treating "cultural knowledge" as a single capability to be fine-tuned monolithically.

Future Directions

• Multilingual and native-speaker validation. The benchmark is currently implemented as idealised anthropological models in English, and the paper explicitly leaves native-speaker validation to future work. Whether the same gaps appear when questions are posed in the relevant languages — and with lexicons that lack an English default to fall back on — is open.
• Measuring pretraining frequency directly. The fictional-rule control is consistent with surface-form interference but does not isolate it; altered answer-space expectations and differential parsing around culturally loaded words are not excluded. A controlled study of corpus frequency for these specific glosses would sharpen the attribution.
• Distinguishing "missing prior" from "rule interference" mechanistically. The in-context-rule probe's opposite-signed effects are behavioural evidence for two distinct failure modes. Probing where in the network these diverge — following work that localises multi-step failures to middle-layer self-attention — is a natural next step.
• Extending beyond the idealised model. The simulator assumes a binary-sex, two-parent biological family structure, and the benchmark excludes distractor facts, so it does not test evidence selection or robustness to irrelevant context. Broader family structures, non-idealised contemporary practice, and distractor-laden contexts are all untested.
• Fine-tuning and retrieval-augmented regimes. Both are explicitly out of scope, yet they are precisely the interventions an industry practitioner would consider. Whether fine-tuning on skewing-system rules closes the gap, or degrades performance the way in-context rules do for high-baseline systems, is unresolved.

Target Audience

This paper is most valuable to researchers building and auditing evaluation benchmarks for LLMs, particularly those working on compositional reasoning, knowledge-graph question answering, or cultural and cross-lingual evaluation. Anthropologists and cognitive scientists interested in formal kinship theory will find Morgan's typology operationalised in a computationally tractable way. Practitioners designing RAG and prompt-engineering pipelines will benefit from the in-context-rule probe's counterintuitive result that supplying a rule can reduce accuracy. Readers looking for state-of-the-art model architecture work or training-method contributions should look elsewhere — this is a measurement paper, and its contribution is in what it reveals about existing models rather than in proposing new ones.

Authors’ abstract

Multi-hop kinship reasoning is a natural testbed for LLM compositionality, but existing benchmarks (notably CLUTRR) cover only the descriptive Eskimo system. We introduce KinshipQA, a procedurally-generated benchmark covering seven anthropologically-documented kinship systems (Eskimo, Sudanese, Hawaiian, Iroquois, Dravidian, Crow, Omaha) and up to six reasoning hops, with a tunable simulator horizon that eliminates exact-instance pretraining overlap. Evaluating six LLMs, we find a 40.9% accuracy drop when reasoning shifts from biological multi-hop to culturally-marked classification on the five non-descriptive systems. The drop holds for every non-descriptive system and is largest for the two skewing systems (Crow, Omaha), persists under chain-of-thought and few-shot prompting, and compounds with depth: at 5--6 hops cultural override falls to 10.6% while biological composition over the same chains remains at 58.6%. Under identical rule access humans reach 89.0% versus 50.7% for LLMs, so the questions are reliably solvable once the rule is supplied. Two follow-up experiments suggest distinct contributors. A fictional-rule control swapping system labels and kin terms for invented strings raises accuracy by 6.1%, implicating familiar English surface forms. An in-context-rule probe prepending the override rule helps skewing systems (+17.1%) but hurts non-skewing systems whose baseline already exceeds about 60% (-13.4%), consistent with a missing skewing prior alongside rule interference where the model already has a working approximation. Our code and data are publicly available on GitHub.