SurveyGraph: Generating Academic Surveys via Methods-Entity Knowledge Graphs

AI summary

Overview

Research area: Natural Language Processing — automated academic survey (literature review) generation, with emphasis on retrieval, knowledge-graph representations of research fields, and citation grounding.

Technical level: Advanced. The paper assumes familiarity with retrieval metrics (recall@k, precision@k, AUC, Cohen's κ), knowledge graphs, RAG pipelines, and clustering.

Scope (one sentence): The paper introduces SurveyGraph, a pipeline that builds a Methods-Entity Knowledge Graph (MEKG) of typed relations between papers and reuses it for core-paper selection, outline generation, graph-aware retrieval, and evolutionary narrative synthesis, evaluated across retrieval benchmarks, semantic-edge validation, citation-grounding audits, and same-backend generation diagnostics.

What This Paper Is About

Most LLM-based survey generators retrieve papers by lexical or embedding similarity, which the authors call "structural blindness": the system can find topically similar papers but cannot tell which work extends which idea, which methods compete, or which share a task or dataset. The goal of SurveyGraph is to build an explicit, typed, auditable graph of method-level relationships and reuse it throughout the writing pipeline — for choosing core papers, structuring the outline, retrieving graph-linked context, and narrating how a field evolved. The paper's stance is deliberately conservative: it frames the graph as an auditable control layer and reports several comparisons as bounded diagnostics rather than leaderboard claims.

Key Contributions

1. A graph-centric survey framework. SurveyGraph couples MEKG construction, graph-aware retrieval, hybrid outline generation, and evolutionary narrative synthesis into one pipeline, in contrast to flat-RAG or citation-only approaches. The MEKG schema defines heterogeneous entities (Method, Task, Dataset, Metric, Model, Innovation) linked by 21 typed relations across five categories.

2. A multi-level evaluation protocol. The authors combine a seven-method equal-budget retrieval benchmark, an archived graph-native diagnostic, semantic-edge validation with adjudicated human annotation, citation-grounding audits at claim and section levels, SurveyBench-facing collection probes, and same-backend generation-stage controls.

3. Quantified evidence for the graph's value and its limits. MEKG-anchored fusion gives the best recall–structure balance in the seven-method benchmark; graph-aware retrieval nearly doubles recall on the archived core diagnostic (p = 0.004); an evidence gate (AUC = 0.75) filters noisy semantic edges; human validation on 108 edges reaches κ = 0.64 for binary support with 68.5% supported-or-partial after adjudication.

4. A full reproducibility release. Eight completed runs with MEKG graphs and evaluator traces, retrieval replay scripts for all seven methods, the 108-row annotation packet with guidelines, and grounding/evidence-gate audits are released, with code at the linked GitHub repository.

Main Findings

• Graph traversal is the main driver of retrieval gains. On the archived graph-native target set, graph-aware retrieval reaches 0.373 core-paper recall versus 0.216 for flat lexical retrieval across all eight runs (sign test p = 0.004). Relation density rises from 0.162 to 0.259 and community coverage from 0.758 to 0.863. Removing 1-hop expansion collapses recall to 0.219, while removing PageRank centrality only drops it to 0.341.

• Confident edges carry most of the signal. Keeping only edges with confidence ≥ 0.9 (39% of all edges) still yields recall of 0.342.

• Typed MEKG alone is not a high-recall retriever; fusion is. In the seven-method benchmark, MEKG typed alone gets R@5 = 0.153 but the highest relation density (0.228); flat lexical gets R@5 = 0.252 with density 0.134; Biblio+TF-IDF gets the highest R@5 at 0.357. Fusion methods retain competitive recall while nearly doubling density (Biblio+MEKG: R@5 = 0.304, density 0.174).

• An independent co-citation check supports the graph. MEKG-linked methods produce retrieved sets with higher co-citation connectivity (0.147–0.163) than graph-free methods (0.089–0.131), measured via OpenAlex independently of MEKG edge definitions.

• Semantic edge quality varies sharply by relation type. Across 2,091 semantic edges (extends: 1,276; improves: 366; applies: 449), temporal direction consistency exceeds 98%, but only 58.8% are supported-or-partial under a deterministic protocol: improves 80.6%, applies 59.0%, extends 52.4%. Human annotation on 108 stratified edges after adjudication gives 68.5% supported-or-partial (40.7% fully supported, 27.8% partial, 31.5% unsupported), with improves strongest at 80.6%, applies at 72.2%, and extends weakest at 52.8%.

• Confidence scores are meaningful. High-confidence edges reach 75.0% supported-or-partial, mid-confidence 72.2%, low-confidence 58.3%. ROC analysis yields AUC = 0.75 with optimal threshold 0.35 (F1 = 0.81); improves and applies are well separated (F1 > 0.88), while extends needs a higher threshold.

• Evidence gating does not degrade retrieval. The gate removes roughly 103 noisy semantic candidates per topic, changing mean recall from 0.164 to 0.182 and precision from 0.105 to 0.117; recall improves only on the "Memory in the Age of AI Agents" topic (+0.071) and is unchanged on the other three. The authors treat this descriptively given only four topics.

• Citations are mostly grounded. Of 160 sampled cited sentences (20 per run across eight surveys), 93 are anchored, 36 partial, and 31 weak/missing — 80.6% anchored-or-partial. Cited evidence beats a matched non-cited hard negative in 65.4% of cases (anchored 71.7%, weak 48.4%). A 42-section audit finds zero uncited sections, 85.7% aligned-or-partial, and 19.5 citations per section on average.

• Outlines are richer than SciSage's. Graph-aware outlines average 7.25 sections with 36.5 nodes versus 2.75 sections from SciSage on the same topics.

• Benchmark-facing collection is viable but partial. A live collection probe on four SurveyBench topics under an eight-paper budget reaches 0.521 mean reference coverage versus 0.166 for exact-topic retrieval. In a reference-list probe, released SurveyForge references score 0.465 mean coverage and human survey references 0.629.

• Removing hybrid outlining hurts organization signals. In the same-backend Qwen diagnostic on a 30-paper MEKG with 74 typed relations, the full run scores 4.135 rubric / 3.80 global reflection; disabling the evolutionary analyzer drops global reflection to 3.50; forcing a rule-based outline fallback reduces the artifact to 5 sections (23 pages) with 3.955 rubric and 3.20 global reflection.

• The matched flat-vs-graph comparison is mixed. With the same 30 papers, backend, outline, and generation settings, SurveyGraph scores 4.135 vs 4.030 rubric (+0.105), 89.1% (49/55) vs 86.3% (44/51) citation grounding, and 3.80 vs 3.20 global reflection, but the evidence-level TF-IDF hard-negative contrast favors flat generation (65.5% vs 78.4%). Section alignment is reported only as directional (3/4 vs 3/5).

• Graph construction dominates cost. MEKG construction accounts for 55% of the pipeline's token budget, amortized across repeated retrieval, outline variants, and generation passes.

Methodology in Plain English

The pipeline has four stages.

1. Build the graph. Starting from a topic, the system collects papers from multiple sources and infers relations in four layers: bibliometric structure (citations, co-citation), pattern matching on citation contexts, shared-entity analysis, and LLM reasoning over pairs of paper records. The LLM layer returns a typed edge, a confidence, and a rationale, and is allowed to return "none" when papers are merely topically similar. Long papers are handled by a recursive chunking strategy that carries summaries forward. Core papers are ranked with a weighted score combining PageRank, log citation count, recency, and degree centrality (weights 0.4/0.3/0.2/0.1), with an elbow heuristic picking the cutoff.

2. Build an outline. A top-down survey template is merged with bottom-up themes found by clustering paper title+abstract embeddings with HDBSCAN (min_cluster_size = 3, min_samples = 2, KMeans fallback), then an LLM arranges clusters into a three-level hierarchy.

3. Retrieve with the graph. For each section, the system starts from core papers and expands one hop to predecessors, successors, and siblings, capped at k = 5 papers per relation type and ranked by a mix of graph centrality, category overlap, and embedding similarity. A depth sweep found 2-hop expansion raised relation density but lowered community coverage and unique-paper yield.

4. Write the narrative. An evolutionary narrative analyzer detects milestones and phase transitions across time and organizes them chronologically and logically. Each section then goes through a write–evaluate–refine loop (threshold 3.5, at most 3 iterations) with a reflection agent scoring coverage, coherence, citations, and critical analysis. Figures use data-template separation: the LLM emits structured JSON-like data that is rendered by predefined LaTeX/TikZ templates.

Evaluation is layered: graph-native diagnostics (weighted centrality recall, community coverage, structural coherence), external retrieval benchmarks, human edge annotation, citation-grounding audits against abstracts and cached text, and same-backend component ablations with a fixed Qwen model.

Why This Matters

Impact on research. The paper reframes survey generation as a structure-recovery problem rather than a similarity-search problem, and it is unusually candid about where its evidence is bounded — explicitly separating graph-derived diagnostics from external benchmarks and refusing to claim leaderboard parity. The per-relation quality breakdown (improves 80.6% vs extends 52.8%) is a concrete, reusable finding: implicit "extends" relations are much harder to validate than performance refinements, which tells future work where to invest.

Real-world applications:

• Automated first drafts of literature reviews for researchers entering a new subfield, with auditable evidence for why each paper was included.
• Research-library and database products that could surface method lineages ("what improved on this method, and on what benchmark?") rather than keyword matches.
• Funding agencies, systematic-review teams, and industry R&D groups mapping the state of a technical area before committing resources.
• Editorial and peer-review support: checking whether a claimed related-work landscape misses competing method families.

Industry relevance. The MEKG is a reusable asset: once built for a topic, it serves many surveys at marginal cost, which matters for organizations producing repeated technical reports. The confidence-gated, typed relation layer is directly transferable to enterprise knowledge management, patent landscaping, and internal technical-intelligence systems where "why was this document retrieved" must be answerable to a human reviewer.

Future Directions

• Full end-to-end SurveyBench evaluation. The current probes cover collection and reference lists only; the authors plan complete outlining, writing, and claim-support evaluation on the full suite, plus evaluations on SurGE and SurveyLens (not yet done).
• Same-backend cross-system comparison. A same-topic, same-budget, same-backend rerun against MVSS and StructSurvey is explicitly named as future work, since the present comparison is confounded by differing backbones and released artifacts.
• Larger and more diverse edge annotation. The 108-edge study covers three relation types from a single domain cluster, and three-way agreement is low (κ = 0.38); extending to more topics and relation types would strengthen the claims, and the released packet is designed to enable that.
• Improving the weakest relation type. "Extends" edges are both the most numerous (1,276 of 2,091) and the least reliable (52.4–52.8% supported-or-partial), making better inference or stricter gating for implicit follow-on relations an obvious open problem.

Target Audience

Researchers and graduate students working on automated survey generation, scientific document processing, and Graph-RAG; practitioners building retrieval systems over scholarly or technical corpora who need auditable, explainable retrieval decisions; and evaluation-focused readers interested in a template for combining graph-native diagnostics with external grounding audits. Readers wanting a leaderboard result or a turnkey writing tool will find the paper deliberately hedged; readers wanting a careful account of where typed graph structure does and does not help will find it directly useful.

Authors’ abstract

Most automated survey generation systems retrieve papers as flat documents, limiting their ability to capture research structure. We present SurveyGraph, a framework built around a Methods-Entity Knowledge Graph (MEKG) encoding method-level follow-on, competition, and shared-resource relations. The graph supports core-paper discovery, retrieval, hybrid outlining, and evidence-gated narrative synthesis. We evaluate retrieval, semantic edges, citation grounding, collection coverage, and generation-stage components. In a controlled seven-method equal-budget benchmark, MEKG-anchored fusion balances recall and structural density, with an independent co-citation check. In a separate archived graph-native diagnostic, graph-aware retrieval nearly doubles recall over flat lexical baselines across eight runs (p = 0.004). A semantic evidence gate achieves AUC = 0.75. Audits of 160 generated sentences find 80.6% anchored-or-partial citation support. Human adjudication of 108 stratified semantic edges finds 68.5% supported-or-partial relations (binary-support Cohen's kappa = 0.64), with improves strongest (80.6%) and extends weakest (52.8%). A SurveyBench-facing collection probe reaches mean reference coverage of 0.521 across four topics with an eight-paper budget. Same-backend Qwen diagnostics on a 30-paper/74-relation graph show reduced organization signals when hybrid outlining is removed. Concurrent-system comparisons provide bounded context rather than matched-backend end-to-end evidence. The results position MEKG as an auditable structural layer for typed diagnostics, relation-specific filtering, and controlled survey generation. Code and artifacts: https://github.com/xzzvsxd/SurveyGraph.