Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation

AI summary

Overview

• Research area: Automated academic survey generation using agentic LLM systems; posted under arXiv category Computer Vision (cs.CV), with a benchmark spanning 21 core computer science topics and 9 non-CS topics.
• Technical level: Advanced. The paper assumes familiarity with retrieval-augmented generation, multi-agent orchestration, taxonomies, citation planning, and LaTeX manuscript assembly.
• Scope: The paper proposes DAS, a stateful agentic framework that treats survey writing as closed-loop manuscript construction over a persistent literature metadata lake (DAS-2M), and evaluates it on a new 30-topic benchmark (DAS-Bench) with a 16-criterion rubric (DAS-Eval).

Note: the supplied paper content is truncated mid-sentence inside Appendix B, so appendix details in B through H are only partially available.

What This Paper Is About

Producing a publication-quality academic survey by hand requires broad literature coverage, coherent organization, careful citation support, and substantial expert effort, and existing Deep Research and automated survey systems each handle only part of that pipeline. The authors argue these systems fail to coordinate paper understanding, literature organization, evidence-grounded drafting, and manuscript validation through a single shared, revisable state. DAS addresses this by separating reusable paper analysis from topic-specific manuscript construction, then running construction as a scoped closed loop that repairs only the affected writing states and performs deterministic validation before assembly.

Key Contributions

1. DAS, a stateful agentic framework for publication-oriented academic surveys. The authors describe it as the first stateful agentic framework for this task, and pair it with DAS-2M, a dynamic literature metadata lake providing fine-grained, survey-oriented paper representations for topic-specific agentic construction.
2. A closed-loop manuscript construction methodology. It combines candidate-grounded taxonomy planning with reverse paper-to-section routing, hierarchical paragraph and claim-level citation planning, a scoped semantic review-and-repair loop, and deterministic validation, maintaining cross-level consistency from candidate literature to final manuscript assembly.
3. DAS-Bench, described as the first benchmark designed to evaluate publication-oriented academic survey generation. It contains 30 survey topics (21 core computer science topics and 9 non-CS topics), with DAS-Eval scoring manuscripts on four dimensions containing four criteria each (16 criteria total, each scored 1 to 5).
4. Empirical demonstration of the strongest overall performance among compared systems, including system-level results, per-mechanism ablations, repair-policy comparisons, a backbone sensitivity study, and blinded expert evaluation.

Main Findings

• Highest overall score among systems evaluated on all 30 topics: DAS reaches a Total Avg. of 4.34 across the 16 criteria, compared with 4.03 for Naive RAG, the strongest competitor. For reference, the Human row also reports 4.34.
• Best group averages in all four dimensions: DAS has the highest BSC, TSQ, HDQ, and MAR group averages among systems evaluated on all 30 topics, and is best or tied for best on 18 of the 21 measures reported in the main results table.
• Strengths cluster in specific criteria: the clearest DAS advantages appear in multi-reference synthesis, global organization, research insight, paragraph progression, local synthesis, and visual integration. The advantage is not uniform: Naive RAG obtains the highest taxonomy boundary and multi-level goal alignment scores, and AutoSurvey performs best on citation balance.
• Ordering preserved on the matched subset: the same ordering among generated systems is preserved on the shared 21-topic CS subset, and on the matched subset comparison.
• Human manuscripts are strong but not uniformly ahead: the Human reference row scores 4.34 overall, matching DAS, with MAR group average 5.00; on individual criteria such as citation balance (Human 4.07), DAS reports 3.37.
• Ablations identify routing and semantic review as the most consistent contributors: removing reverse paper-to-section routing decreases BSC, TSQ, and HDQ by 0.09, 0.12, and 0.11 respectively, while removing semantic review lowers both TSQ and HDQ by 0.16. Replacing structured paper representations with titles and abstracts reduces TSQ by 0.10 and HDQ by 0.07.
• Some mechanisms trade off across dimensions: removing taxonomy planning slightly increases BSC and HDQ but reduces TSQ (notably global organization and research insight); removing hierarchical drafting improves several discourse scores but lowers BSC, especially citation balance.
• Deterministic validation primarily buys reliability: Full DAS compiles all 30 manuscripts, whereas the variant without it compiles 27 and reduces HDQ from 4.28 to 3.98; the three failed outputs received only minimal syntax corrections for MAR rendering.
• Adaptive repair policy balances success and cost: Full DAS achieves the highest Review Pass Rate at 74.59% and the lowest review cost among policies performing semantic review, at 0.79M tokens per survey. Paragraph Replan Only yields a lower pass rate (53.69%) and higher cost (0.96M), Direct Edit Only 58.42% and 0.95M, and Section Replan Only 45.13% and 2.57M.
• Backbone capacity changes analytical quality but not the framework's operation: on five stratified topics, GPT-5.5 scores 4.54 total (0.16 above the default backbone), the default Qwen3.5-397B-A17B-FP8 scores 4.38, and Qwen3.5-35B-A3B-FP8 scores 3.96 (0.42 below default) while retaining complete manuscript artifacts. No value is extrapolated to all 30 topics.
• Blinded expert evaluation favors DAS: three domain experts preferred DAS to Naive RAG on 27 of 30 topics and to AutoSurvey on 19 of 21 shared CS topics, ranking DAS first on 18 of these 21 topics, with unanimous agreement on 63 of 72 pairwise comparisons.
• Cross-judge robustness is partial: re-evaluation with Kimi K2.6 yields a moderate correlation with Qwen3.5 (rho = 0.507, MAE = 0.630) while preserving the ordering DAS > Naive RAG > AutoSurvey; the judges agree on 48 of 63 CS comparisons, with lower local agreement on the non-CS subset.
• Capability comparison: in the systems table, DAS is the only listed system documented as having all of the following: precomputed paper representation, grounded taxonomy, literature routing, discourse planning, review loop, deterministic checking, visual integration, content-adaptive visualization, and PDF output. Comparison corpora listed are AutoSurvey 530K, SurveyForge 600K+20K, SurveyX 2.63M (unreleased) plus online, InteractiveSurvey online plus uploads, LiRA provided references, DeepSurvey online, and DAS 2M.

Methodology in Plain English

The system splits the work into two phases that other systems tend to blur together.

First, before any topic is known, DAS precomputes a structured profile of each paper and stores it in a persistent metadata lake. The authors collected roughly 2 million arXiv PDFs released between January 2020 and June 2026 across all subject categories from the arXiv corpus available through Google's Kaggle platform, keeping only the latest version of each paper; after deduplication and removal of corrupted or unparsable files, 1.53 million papers remained in the experimental resource. PDFs are parsed offline with MinerU and PDF-Extract-Kit-1.0 model assets, with a 600-second parsing budget per paper, and an LLM extractor (Qwen3.5-397B-A17B-FP8, thinking disabled, five concurrent workers, temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5, 4,096-token output limit, up to ten retries) converts each parsed paper into eight high-level field groups: bibliographic metadata, topical categorization, technical configuration, resource availability, methodological details, dataset usage, research rationale and findings, and empirical evaluation and limitations. Lexical and semantic indexes are built over this metadata, and the lake is updated through the same pipeline as new papers arrive.

Second, for a given topic, agents build the manuscript against an explicit, shared state that has four parts: a literature state (query plan and candidates), an organization state (taxonomy and paper-to-section routing), a writing state (paragraph plans, writing-point plans, evidence, drafts, review status), and a finalization state (figures, tables, bibliography, source files). A query planner expands the topic into lexical and semantic queries; a hybrid retriever runs BM25 plus dense search over the offline indexes and merges results with weighted reciprocal rank fusion to form the candidate set. A taxonomy planner then organizes those candidates into a rooted taxonomy whose nodes are labelled analytical, reflective, or navigational, with a deterministic structural check verifying the hierarchy. Because the candidate set is optimized for recall, a paper router evaluates each candidate against eligible taxonomy nodes and produces a sparse multi-label assignment, so each section has an explicit citation scope.

Drafting is hierarchical. For each taxonomy node, a paragraph planner either writes a concise overview (Route A, used for navigational nodes and nodes whose literature cannot support multi-paragraph technical discussion) or enters the full pipeline (Route B). For Route B, a claim planner turns each paragraph into an ordered sequence of writing points, each with an intended claim, a supporting citation group, and the technical details needed, and may issue a bounded request to the source document when a detail is missing. A drafter writes each paragraph, which must immediately pass a deterministic validator checking citation identifiers, unresolved placeholders, paragraph boundaries, section formatting, and LaTeX constraints.

A reviewer then examines each assembled subsection for technical relevance, alignment with the taxonomy-defined objective, argumentative progression, redundancy, and citation support for central claims, and chooses one of four actions: accept, revise paragraph j, replan paragraph j, or replan the entire subsection. The corresponding scoped repair regenerates only the affected writing state; regenerated paragraphs must pass deterministic validation again before recommitting, and the loop repeats until acceptance or the retry budget is exhausted, in which case the latest complete validated version is retained. Finally, visual generation roles produce content-adaptive figures and tables, and a finalizer resolves bibliographic records, inserts and validates cross-references, and assembles LaTeX, BibTeX, and asset files into the compiled manuscript.

For evaluation, closed-source systems use their native configurations, while reproducible systems keep their released workflows and use Qwen3.5-397B-FP8 as a common generation backbone; original retrieval resources are retained where available, otherwise systems receive a frozen 300-paper candidate set. Multimodal LLM judges score rendered manuscripts along BSC, TSQ, HDQ, and MAR, and domain experts independently rank method-blinded manuscripts. SurveyX exceeded the 12-hour limit and is reported only in a completion analysis; DeepSurvey is excluded because no public implementation was available.

Why This Matters

Automated survey generation is one of the few academic tasks where the output is itself a scholarly artifact, so reliability, traceability, and assembly correctness matter as much as fluent prose. This paper's central move is architectural: by making the manuscript a persistent, structured state that can be selectively reactivated, it separates reusable paper understanding from per-topic writing and makes repair scoped rather than wholesale. The ablation and repair-policy results give concrete evidence that this design choice, not just the model, drives quality and cost.

Real-world applications (framed as plausible uses; the paper does not report deployed applications):

• Literature onboarding and teaching material: generating structured, citation-supported overviews of unfamiliar fields for students and new researchers.
• Research group and laboratory knowledge management: maintaining living surveys of a fast-moving subfield that update as new arXiv papers enter the metadata lake.
• Pre-submission support for authors: drafting related-work and background sections with claim-level citation plans that can be checked by a human reviewer.
• Editorial and review workflows: providing structured summaries with visual comparisons and traceable references.

Industry relevance: the framework's cost profile is quantified in the paper (review and repair costing 0.79M tokens per survey under the full policy, and 0.00M when semantic review is removed), and the backbone study shows sensitivity to model scale across GPT-5.5 (4.54), Qwen3.5-397B-A17B-FP8 (4.38), and Qwen3.5-35B-A3B-FP8 (3.96). That makes the paper relevant to teams deciding how much model capacity they need for document-generation pipelines that combine retrieval, planning, validation, and formatting, and to anyone building agentic systems where retry cost must be bounded.

Future Directions

• Scaling and refreshing the substrate: the metadata lake covers arXiv papers from January 2020 to June 2026 with 1.53 million papers remaining after cleaning. Whether the same pipeline generalizes to other corpora, non-English literature, or other publication venues is not evaluated.
• Understanding the backbone ceiling: GPT-5.5 improved the total score by 0.16 over the default 397B backbone on five stratified topics, while the 35B model lost 0.42, and the paper explicitly does not extrapolate these five-topic results to all 30 topics. A larger, fully crossed backbone study is a natural next step.
• Making evaluation less judge-dependent: the cross-judge comparison shows only moderate correlation (rho = 0.507, MAE = 0.630) and weaker local agreement on the non-CS subset, so the authors' own results point to a need for more stable fine-grained scoring.
• Extending beyond the current comparison set: SurveyX could not complete within the 12-hour limit and DeepSurvey had no public implementation, so the relative standing of DAS against those systems remains unmeasured.

Target Audience

Researchers and practitioners in automated scientific writing, agentic LLM systems, and retrieval-augmented generation, plus evaluation researchers interested in rubric-based and expert-based assessment of long-form generated artifacts. It is also relevant to authors and publication venues who want to understand how far automated survey construction has progressed and where human oversight still matters. Readers should be comfortable with multi-agent pipelines, citation grounding, and LaTeX-based manuscript production.

Authors’ abstract

Academic surveys play a central role in organizing rapidly expanding scholarly literature, yet their construction requires extensive paper analysis, coherent knowledge organization, fine-grained citation support, and reliable manuscript assembly. Existing Deep Research and automated survey generation systems address parts of this process, but typically do not coordinate paper understanding, literature organization, evidence-grounded drafting, and manuscript validation through a shared, revisable state. We introduce DAS, a stateful agentic framework for generating publication-oriented academic surveys. Its key idea is to separate reusable paper analysis from topic-specific manuscript construction. DAS builds on DAS-2M, a dynamically updated metadata lake containing survey-oriented representations of approximately two million papers. Its agents maintain explicit literature, organization, writing, and finalization states through candidate-grounded taxonomy planning, reverse paper-to-section routing, and hierarchical claim and citation planning. Semantic review reactivates only the affected writing states for repair and reevaluation, forming a scoped closed loop with deterministic validation. We further introduce DAS-Bench, a 30-topic benchmark, together with DAS-Eval, which assesses scholarly citation quality, taxonomic synthesis, hierarchical discourse, and manuscript assembly reliability through 16 criteria. Among systems evaluated on all 30 topics, DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor, and the same ordering is preserved on the matched 21-topic CS subset. Blinded expert evaluation further prefers DAS to Naive RAG on 27 of 30 topics and to AutoSurvey on 19 of 21 shared CS topics. The project page is available at https://zhikaixu24.github.io/projects/DAS/.