SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models

AI summary

Overview

• Research area: Natural language processing — evaluation of reasoning versus memorization in language models (LMs).
• Technical level: Intermediate. Familiarity with multi-hop question answering, retrieval-augmented generation (RAG), and agentic tool use helps, but the framework's core idea is explained plainly in the paper.
• Scope: The paper introduces SynthWorlds, a fully automatic framework that builds paired real-entity and synthetic-entity corpora so that a model's reliance on memorized world knowledge can be measured separately from its task reasoning.

What This Paper Is About

Benchmarks for LM reasoning are confounded by the fact that models have already memorized many facts about the world, so strong scores may reflect recall rather than reasoning. Existing fixes — manually curated "clean" evaluation sets, temporal filtering, paraphrasing, adversarial substitution — cannot cleanly separate the two, because either they are costly and go stale or they leak parametric knowledge through reused content. SynthWorlds addresses this by generating two parallel "worlds" with identical structure but different entity names, then measuring the performance difference between them as a knowledge advantage gap.

Key Contributions

1. A scalable framework for generating interconnected corpora and tasks. SynthWorlds samples a connected subgraph from a knowledge base, renames entities with type- and name-derivation-consistent transformations, and generates documents in the synthetic-mapped universe before mapping symbolic references back to real-world labels, producing two corpora that differ only in surface-form names.
2. Two parallel corpora with paired task datasets. SynthWorld-RM (real-mapped) and SynthWorld-SM (synthetic-mapped) are paired at the document, fact, and task levels. The paper reports each corpus contains 6,920 documents covering 161K facts in the contributions list, while Table 1 reports 6,290 documents and over 1.5M tokens; both figures appear in the paper. The datasets include 1.2K multi-hop QA instances and 1K page navigation instances, and are released publicly (GitHub: behavioral-data/synthworlds; Hugging Face: kenqgu/synthworlds).
3. Two mirrored case-study tasks with controllable difficulty. Multi-hop QA is built by composing single-hop questions over graph motifs (from 2 decomposed questions for motif A up to 4 for motifs D, E, F), and page navigation is built by sampling source–target page pairs across five difficulty buckets defined by expected random-walk distance (50–1K, 1K–10K, 10K–100K, 100K–1M, 1M–10M).
4. An empirical analysis of the knowledge advantage gap across parametric-only and knowledge-augmented settings, using six models: GPT-5-mini (reasoning effort set to medium), Gemini-2.0-Flash, gpt-oss-20B, gpt-oss-120B, Kimi-K2-Instruct, and Kimi-K2-Thinking.

Main Findings

• Closed-book QA shows a pure parametric knowledge gap. On multi-hop QA, baseline performance on real-mapped questions is approximately 20 F1 (P_R^base ≈ 20), while synthetic-mapped performance is near zero (P_S^base validates that these questions cannot be solved parametrically), giving KA^base ≈ 20. As task difficulty increases, real-mapped performance decreases while synthetic-mapped performance remains at 0.
• Reading comprehension equalizes or reverses the gap. When models are given gold documents (2–4, depending on graph motif) plus distractor documents to reach 10 total, synthetic-mapped performance is equal to or stronger than real-mapped, which the authors attribute to real-mapped settings letting parametric knowledge interfere with grounding reasoning in the provided content.
• Page navigation shows a larger gap for some models. Under the Links Only condition, KA^base ≈ 30 for GPT-5-mini and Kimi-K2 models versus KA^base ≈ 20 for Gemini-2.0-Flash and gpt-oss models. The gap persists across difficulty levels, narrowing slightly at the easiest difficulty.
• One-step RAG widens the gap. KA^base − KA^RAG = −4.0 for GPT-5-mini and −1.3 for Gemini-2.0-Flash, meaning RAG disproportionately benefits the real-mapped setting. The pattern is consistent across multiple difficulty levels and other models.
• IRCoT + RAG narrows the gap. KA^base − KA^IRCoT+RAG is positive: 5.2 for GPT-5-mini and 10.3 for Gemini-2.0-Flash. The authors note IRCoT + RAG does not improve absolute performance over one-step RAG for the gpt-oss models, which struggle to follow the IRCoT prompt format.
• Retrieval quality appears to drive part of the gap. Triangulating reading comprehension F1 with one-step RAG and IRCoT + RAG F1 and recall@5 suggests LM-based retrieval generalizes less well to the synthetic (novel) corpus; retrieval performance improves slightly with IRCoT in both worlds but synthetic-mapped retrieval remains lower than real-mapped.
• Page content narrows but does not close the navigation gap. Granting access to page content yields KA^base − KA^content = 9.3 for GPT-5-mini and 7.0 for Gemini-2.0-Flash. The narrowing is largest on simpler navigation pairs.
• Reasoning traces reveal reliance on unobserved knowledge. In the Links Only condition, at least one external entity (one not seen on any previously visited page) is mentioned in 48% of steps for GPT-5-mini and 60% for Gemini-2.0-Flash. Under Content + Links, these rates drop to 35% and 15%. In the synthetic setting this rate is 0 by construction and confirmed empirically.

Methodology in Plain English

The researchers start from a knowledge base of subject–relation–object triplets and sample a connected subgraph of facts. Because the structure of this subgraph carries the reasoning difficulty, they keep it fixed and change only the names. People get person names (Geoffrey Hinton becomes Caleb Ardent), cities get city names (Toronto becomes Metrovale), and derived names stay consistent (University of Toronto becomes University of Metrovale, not University of Grandvale). Names are also constrained to remain compatible with entity type, so a library stays library-like (Central Library becomes Oakwood Public Library, not Central Stadium).

Using the synthetic triplets, an LM writes documents in the synthetic world, symbolic references to other entities are inserted as hyperlinks, and those references are then swapped to the real-world labels so that every synthetic document has a real-world twin with identical sentence structure. This yields two corpora differing only in surface forms, and two tasks built on top of them: multi-hop questions composed from single-hop questions over reasoning motifs, and a page navigation agent equipped with click_link and backtrack tools, capped at 30 steps (well above the shortest-path distribution, whose median is 5 and maximum is 11). The knowledge advantage gap is then computed as the difference in performance between the real-mapped and synthetic-mapped versions of each task, both with parametric knowledge only and with knowledge augmentation. Evaluation metrics are token-based F1 for QA, recall@5 for RAG baselines, and success rate for navigation; the retriever used is HippoRAG 2.

Why This Matters

Impact on research. SynthWorlds provides a controlled, automatic alternative to manually curated "clean" benchmarks, which lose their value as models memorize them — the paper's example is MuSiQue (released 2021), on which Llama-3.3-70B achieves over 26% F1 with no documents. Because the framework is inexpensive and reusable with any input knowledge graph, new corpora can be generated to keep evaluation data novel and guard against inclusion in pre- and post-training datasets.

Real-world applications:

• Web navigation agents that must traverse linked pages to find information without prior familiarity with the site.
• Personal health insights, where a model connects medical data to external resources to inform advice.
• Scientific discovery, where agents integrate findings scattered across research articles to form new hypotheses.
• Deployment in novel environments, where systems cannot rely on knowledge seen during training.

Industry relevance. The results suggest that current retrieval pipelines and agentic workflows may underperform when deployed in domains or environments where models lack memorized background knowledge, and that LM-indexed retrieval may not generalize well in such settings — a direct concern for anyone building RAG or browsing agents.

Future Directions

1. Test alternative knowledge integration schemes. The paper asks whether retrieval could be integrated into page navigation to improve planning, and whether long-context methods or multi-agent workflows (where group discussion and feedback shape integration) can close the gap.
2. Generalize across constructed worlds. The authors note their experiments cover only the specific corpora and task designs they introduced, and that different relation types, connective structures, or contexts could produce different outcomes.
3. Extend beyond one knowledge graph or domain. The framework could apply to mathematics with parallel notation systems (real-mapped x, y, f(x) versus synthetic-mapped α, β, φ(α)) or to code generation by consistently renaming entire libraries such as numpy or pandas and their function calls.
4. Build graphs from unstructured text. SynthWorlds requires a high-quality knowledge graph for factual consistency; as extraction methods improve, graphs could be constructed directly from sources like Wikipedia.

The paper also states a limitation: findings may not generalize beyond the specific SynthWorlds corpora and task designs studied.

Target Audience

Researchers and practitioners working on LM evaluation, multi-hop question answering, retrieval-augmented generation, and agentic web navigation. It is most useful for those who need to distinguish genuine reasoning from memorization in model benchmarks, and for engineers building retrieval or agent systems intended to operate in unfamiliar environments. Readers without background in knowledge graphs or RAG will still follow the main argument but may want the appendices for construction and evaluation details.

Authors’ abstract

Evaluating the reasoning ability of language models (LMs) is complicated by their extensive parametric world knowledge, where benchmark performance often reflects factual recall rather than genuine reasoning. Existing datasets and approaches (e.g., temporal filtering, paraphrasing, adversarial substitution) cannot cleanly separate the two. We present SynthWorlds, a framework that disentangles task reasoning complexity from factual knowledge. In SynthWorlds, we construct parallel corpora representing two worlds with identical interconnected structure: a real-mapped world, where models may exploit parametric knowledge, and a synthetic-mapped world, where such knowledge is meaningless. On top of these corpora, we design two mirrored tasks as case studies: multi-hop question answering and page navigation, which maintain equal reasoning difficulty across worlds. Experiments in parametric-only (e.g., closed-book QA) and knowledge-augmented (e.g., retrieval-augmented) LM settings reveal a persistent knowledge advantage gap, defined as the performance boost models gain from memorized parametric world knowledge. Knowledge acquisition and integration mechanisms reduce but do not eliminate this gap, highlighting opportunities for system improvements. Fully automatic and scalable, SynthWorlds provides a controlled environment for evaluating LMs in ways that were previously challenging, enabling precise and testable comparisons of reasoning and memorization.