TaxoAlign: Scholarly Taxonomy Generation Using Language Models

AI summary

Overview

Research area: Natural Language Processing, specifically scholarly document understanding, automatic taxonomy construction, and AI for Science.

Technical level: Intermediate. The paper assumes familiarity with large language models, instruction tuning (QLoRA), hierarchical tree structures, and evaluation metrics such as BLEU, ROUGE, and BERTScore.

Scope: The paper introduces a benchmark (CS-TaxoBench) and a three-phase LLM pipeline (TaxoAlign) for automatically generating research taxonomies that structurally and semantically resemble taxonomies written by human experts.

What This Paper Is About

Researchers who write survey papers organize a field into a hierarchical taxonomy: a root topic with sub-topics and finer-grained sub-sub-topics. Building this structure by hand is laborious, and existing automated survey-generation systems never check whether the outlines they produce actually match the structures human experts write. This paper defines the task of automated scholarly taxonomy generation, releases a benchmark of 460 human-written taxonomies plus an 80-taxonomy conference test set, and proposes TaxoAlign, a method whose generated trees are consistently closer to human-written trees than several baselines.

Key Contributions

1. CS-TaxoBench benchmark: 460 taxonomies extracted from human-written survey papers along with their reference papers, plus an additional test set of 80 taxonomies curated from conference survey papers. The benchmark contains 79,027 cited papers in total, of which 60,373 (76.40%) are available on Semantic Scholar.

2. The TaxoAlign method: A three-phase, topic-based, instruction-guided pipeline consisting of Knowledge Slice Creation, Taxonomy Verbalization, and Taxonomy Refinement.

3. A new evaluation framework: Two newly proposed metrics — the average degree score (structural similarity) and level-order traversal comparison (semantic similarity) — combined with Node Soft Recall, Node Entity Recall, and LLM-as-a-judge evaluation.

4. Empirical validation: Automated evaluation on CS-TaxoBench plus a human evaluation study with three domain-knowledgeable annotators, showing TaxoAlign surpasses baselines on nearly all metrics.

Main Findings

• TaxoAlign best matches human structure: On the original test set, the average degree score (Δ, where the ideal value is 1) was 1.6687 for TaxoAlign with LLaMa knowledge slices and 1.668 with Mistral knowledge slices. Baseline values were far higher — AutoSurvey 4.4659, STORM 6.151, Topic+Keyphrases 4.4517 (LLaMa keyphrases) and 4.91 (Mistral keyphrases), and the ablated TaxoAlign w/o Taxonomy Verbalization w/o Taxonomy Refinement ranged from 5.486 to 7.1965. The paper reports that other baseline methods tend to produce overly large trees (Δ > 2.9).

• Semantic and lexical alignment also favors TaxoAlign: On level-order traversal, TaxoAlign reached BLEU-2 of 0.0132 and 0.0051, ROUGE-L of 0.2975 and 0.2974, and BERTScore of 0.8501 and 0.8517. For comparison, STORM scored BLEU-2 0.0012, ROUGE-L 0.1349, BERTScore 0.8166, and Topic only scored BLEU-2 0.0052, ROUGE-L 0.2359, BERTScore 0.8376.

• Node Soft Recall and NER: TaxoAlign achieved NSR of 1.3244 and 1.3635, the highest among methods compared, and NER of 0.1986 and 0.1872. The ablated baseline sometimes scored higher on Node Entity Recall because it generates larger trees, which increases Noun Phrase matches. The paper notes BLEU-2, ROUGE-L, and BERTScore values are much lower than typically seen in machine translation or question answering, indicating substantial room for improvement.

• Ablation shows the value of the later stages: The TaxoAlign w/o Taxonomy Verbalization w/o Taxonomy Refinement baseline performed second best to the full method on most metrics, showing knowledge slices are important, but its higher Δ values reveal that Taxonomy Verbalization and Refinement narrow the gap to gold trees and improve node label quality.

• Conference test set results hold up: On the 80-taxonomy conference set, TaxoAlign reached Δ of 2.1924 (LLaMa) and 2.3617 (Mistral), versus 6.361 and 7.2083 for the ablated version, with BLEU-2 of 0.0058 and 0.013, ROUGE-L of 0.3091 and 0.3004, BERTScore of 0.8542 and 0.8522, NSR of 1.2129 and 1.2072, and NER of 0.2566 and 0.2716.

• Human evaluation agrees: Three annotators using a 5-point Likert scale on 20 randomly sampled test instances rated TaxoAlign 3.17 on structure and 2.62 on content, versus AutoSurvey at 2.17 and 2.25. Inter-annotator agreement was 0.61 and 0.73 (Krippendorff's alpha). LLM-as-a-judge scores correlated with human scores at a Spearman's rho of 0.527.

• Extraction accuracy was validated manually: On 10 manually annotated taxonomy trees, path-level precision, recall, and F1 between annotated and extracted taxonomies were 83.92%, 94.35%, and 88.83%. Errors arose from general section headers that were absent from annotated trees.

• Error patterns differ by pipeline stage: Direct generation from knowledge slices produces verbose taxonomies with irrelevant information and repeated nodes or sub-trees. Adding Taxonomy Verbalization brings structure closer to gold but leaves factual errors, hallucinated node labels, or trees that are too short. Adding Taxonomy Refinement yields better structural and semantic alignment, but layer-wise exact matches remain low.

Methodology in Plain English

Building the benchmark. The authors sourced survey papers from ACM Computing Surveys (impact factor 23.8) covering 2020 to 2024. Of 1,165 accepted papers, 325 were open-access and 285 had arXiv copies. Text was extracted with Docling; papers with noisy layouts or parsing errors were removed, leaving 499. Reference abstracts were retrieved from Semantic Scholar, and papers where fewer than 50% of references were available were dropped — 39 papers — leaving 460. Taxonomies were built from headings, subheadings, and sub-subheadings, with the paper title as the root. Headings containing terms like "Introduction", "related work", "conclusion", "result", or "future" were discarded, as were nodes without retrievable references. The 460 taxonomies were split into 400 training and 60 test instances, averaging about 131 reference papers each. A separate test set came from 2024 survey papers at IJCAI and the ACL* conferences (ACL, NAACL, EMNLP, EACL): 86 papers narrowed to 80, averaging about 71 reference papers each.

The three-phase pipeline. First, Knowledge Slice Creation uses an LLM to pull out the passages from each cited paper most relevant to the taxonomy topic, which both guides later stages and keeps the material within the model's context window. Second, Taxonomy Verbalization instruction-tunes an LLM on these extracted slices so the model learns to produce concise taxonomies grounded in the documents and, critically, to reproduce tree structure — something direct prompting fails at. Third, Taxonomy Refinement prompts a stronger reasoning LLM to check that each parent-child link is grounded in the document slices, replacing nodes when warranted, expanding trees that have too few nodes (fewer than five), and leaving the root unchanged.

Models and training. Mistral-7B-Instruct-v0.3 and Meta-Llama-3-8B-Instruct produce knowledge slices. Llama-3.1-Tülu-3-8B and SciLitLLM1.5-7B are instruction-tuned for verbalization; GPT-4o-mini does refinement. The ablations also probed QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and Sky-T1-32B. Tuning used QLoRA (4-bit NormalFloat, Double Quantization, Paged Optimizers) for 800 steps, input context window 16,384, output context window 1,024, learning rate 2e-4, batch size 1, on a single A100. Models were instructed to produce trees of at most depth three, generating at most 1,024 new tokens.

Evaluation design. The average degree score Δ is the ratio of the mean degree of the predicted tree to that of the gold tree, with 1 being ideal. Level-order traversal flattens both trees and computes BLEU-2, ROUGE-L, and BERTScore on the resulting lists, since standard text metrics cannot compare trees directly. Node Soft Recall uses soft cardinality with Sentence-BERT cosine similarity and a normalizing factor; Node Entity Recall measures the overlap of noun phrases using the FLAIR chunking model. LLM-as-a-judge uses GPT-4.1 to score generated trees from 1 to 5 on structural and semantic similarity against the gold tree.

Baselines. AutoSurvey (outline generation only, given reference papers), STORM (pre-writing stage, given reference papers), Topic only (prompting with the topic alone), Topic + Keyphrases (extracting keyphrases from each cited paper, then prompting), and TaxoAlign without the verbalization and refinement stages.

Why This Matters

Taxonomies let researchers and practitioners navigate a field hierarchically and are a core component of literature surveys. The paper argues that prior work on automated survey generation never compared generated survey structure against expert-written structure, and that no open-source data resource existed for scholarly taxonomy generation. CS-TaxoBench and TaxoAlign fill that gap and contribute to the broader agenda of AI for Science.

Real-world applications:

• Literature review and survey writing: Automatically drafting the organizing skeleton of a survey from a topic and its reference papers.
• Research discovery and navigation: Helping researchers and industry practitioners explore an unfamiliar field through a structured hierarchy.
• Biomedical and scientific information systems: The paper cites prior deployments of taxonomies in biomedical systems and information management.
• E-commerce and recommendation: The paper cites taxonomies used in e-commerce, personalized recommendation, query understanding, and knowledge graph construction.

Industry relevance: Organizations that maintain large internal document collections or product catalogs benefit from consistent hierarchical organization. The finding that LLM-as-a-judge scores correlate with human ratings at a Spearman's rho of 0.527 suggests automated evaluation can partially substitute for costly human review, though the paper's moderate BLEU-2, ROUGE-L, and BERTScore values signal that generated taxonomies are not yet a drop-in replacement for expert-authored ones.

Future Directions

• Broaden the benchmark: The authors note CS-TaxoBench was built from a single journal within a defined time frame for consistency; additional open-access journals and conference venues could be curated.

• Add reference retrieval: The work assumes a set of reference documents is already given. Retrieving those references from a large corpus given only a taxonomy topic is named as an important open task for end-to-end taxonomy construction.

• Improve alignment further: The paper states there is "a lot of scope for improvement" and specifically flags the low number of layer-wise exact matches even after refinement.

• Apply the framework more widely: The authors frame the results as encouraging groundwork for the community to build on.

Target Audience

• NLP and IR researchers working on text generation, document understanding, and structured knowledge extraction.
• Scientific literature and bibliometrics researchers interested in automated surveying, knowledge synthesis, and AI for Science.
• LLM practitioners concerned with long-context reasoning, instruction tuning, and domain adaptation for scientific text.
• Graduate students and survey authors who want to understand how machine-generated taxonomies compare with expert-authored ones and where current gaps remain.

Authors’ abstract

Taxonomies play a crucial role in helping researchers structure and navigate knowledge in a hierarchical manner. They also form an important part in the creation of comprehensive literature surveys. The existing approaches to automatic survey generation do not compare the structure of the generated surveys with those written by human experts. To address this gap, we present our own method for automated taxonomy creation that can bridge the gap between human-generated and automatically-created taxonomies. For this purpose, we create the CS-TaxoBench benchmark which consists of 460 taxonomies that have been extracted from human-written survey papers. We also include an additional test set of 80 taxonomies curated from conference survey papers. We propose TaxoAlign, a three-phase topic-based instruction-guided method for scholarly taxonomy generation. Additionally, we propose a stringent automated evaluation framework that measures the structural alignment and semantic coherence of automatically generated taxonomies in comparison to those created by human experts. We evaluate our method and various baselines on CS-TaxoBench, using both automated evaluation metrics and human evaluation studies. The results show that TaxoAlign consistently surpasses the baselines on nearly all metrics. The code and data can be found at https://github.com/AvishekLahiri/TaxoAlign.