AcademicEval: Live Long-Context LLM Benchmark

AI summary

Overview

Research area: Natural Language Processing — long-context LLM evaluation and benchmarking, with a focus on text generation rather than question answering.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking terminology, retrieval-augmented language models, in-context learning, and metrics such as BERTScore and ROUGE-L.

Scope: The paper introduces AcademicEval, a live benchmark built from arXiv papers that evaluates LLMs on four long-context academic writing tasks (Title, Abstract, Introduction, Related Work) with flexible, few-shot-extended context and periodic data updates to prevent label leakage.

What This Paper Is About

Existing long-context LLM benchmarks fix the context length of each problem, require costly human annotation (which the paper says limits benchmark size to about 2000 samples), and can be leaked into LLM training data because their samples are statically collected. AcademicEval addresses all three problems by turning real arXiv papers into automatically labeled academic writing tasks, where the paper's own Title, Abstract, Introduction, or Related Work section serves as the ground truth. The goal is a benchmark whose context length can be scaled up or down via few-shot demonstrations, and whose test set is continually refreshed with newly published papers.

Key Contributions

1. A live long-context generation benchmark. AcademicEval evaluates LLMs on four academic writing tasks with hierarchical abstraction levels — Title Writing, Abstract Writing, Introduction Writing, and Related Work Writing — and requires no manual annotation because paper sections act as expert-curated labels.

2. A co-author graph built from the arXiv API. The graph supplies few-shot demonstrations drawn from co-author papers, making context length flexible and scalable. The paper contrasts this with Random selection of papers under the same arXiv category.

3. Live evaluation via periodic data updates. An incremental update procedure (Node Update, Node and Edge Update, Graph Pruning) refreshes the benchmark with the latest arXiv papers and uses a chronological split, keeping the test set on the newest papers and ensuring no label leakage.

4. A holistic evaluation with two complementary lenses. The authors evaluate standard LLMs, long-context LLMs, and retrieval-augmented language models using automatic metrics (BERTScore, ROUGE-L) and an LLM-as-a-Judge protocol covering novelty, feasibility, consistency, factuality, and academic style.

Main Findings

• Long-context LLMs are not dominant. Under BERTScore and ROUGE-L, RALM augmented with LLaMA frequently attains the highest scores in multiple settings (Title-30K/31K-G, Intro-28K/28K-G, Related-53K/53K-G) despite using only an 8K input window. Hermes performs best among long-context LLMs overall, but the paper reports it is still slightly inferior to RALM with LLaMA.

• Tasks differ substantially in difficulty. Title Writing tends to score higher than Abstract Writing, which the authors attribute to titles needing only a roughly related theme while abstracts require more detailed description. Related Work Writing is the hardest setting for automatic metrics.

• Performance degrades as input length grows. Across all task settings and both automatic metrics, baselines often perform worse as context length increases (for example, Title Writing scores drop as the context extends from 10K to 31K tokens).

• Few-shot demonstrations have mixed effects. Integrating few-shot demonstrations is neutral or slightly negative in several settings under automatic metrics, with small but consistent gains for certain model–task pairs. Demonstrations drawn from co-author papers generally have a more positive impact than randomly selected ones.

• Automatic metrics and the LLM judge disagree. Under LLM-as-a-Judge win rates, retrieval is not always preferred. Qwen attains the highest win rate at Abs-9K (77.0) and remains strong at Abs-28K (72.7) and Abs-29K-G (71.0), and consistently leads Introduction Writing (Intro-8K: 79.3; Intro-28K: 70.3; Intro-28K-G: 70.9). Hermes reaches the highest Title-10K win rate (72.0), while Mixtral becomes top under Title-31K-G (62.7).

• Retrieval helps Related Work specifically. RALM models, especially LLaMA augmented with a retriever, achieve the top preferences at longer lengths (Related-53K: 90.7; Related-53K-G: 90.2), consistent with retrieval being beneficial when judged quality rewards citations, prior studies, and domain terminology.

• Dense retrievers beat sparse retrievers. In experiments on the Abs-9K setting with Gemma Instruct (7B) and LLaMA-3 Chat (70B), dense retrievers consistently outperform sparse ones, with Contriever achieving the best results.

• The performance plateau is driven by in-context learning limits, not memorization. A Title-only ablation on Abstract Writing (Abs-9K) removes most context; BERTScore and ROUGE-L drop by 5–7 points for both LLaMA and Hermes (LLaMA: 62.4/25.0 to 57.4/18.8; Hermes: 62.2/26.1 to 56.7/19.3), indicating genuine reliance on external context.

Methodology in Plain English

The researchers treat arXiv as both the data source and the label source. Using the arXiv API, they build a co-author graph by breadth-first search, where edges connect co-authors and each author node carries features such as published first-author papers. Authors who never published as first author are removed, and the graph is pruned to its maximum connected component. Paper PDFs are downloaded and split into sections with PyMuPDF by detecting section headings; for Related Work, the title and abstract of each cited paper are also pulled from the arXiv API to form a citation corpus.

From this data they define four leave-one-out style writing tasks. Title Writing takes the main body plus abstract and asks for a title; Abstract Writing takes the main body (with Conclusion removed) plus title; Introduction Writing takes the main body (with Introduction removed), title, and abstract; Related Work Writing takes the main body (with Related Work removed), title, abstract, and the citation corpus. The removed section is the ground truth.

Context length is made flexible by prepending few-shot demonstrations, either randomly selected papers from the same arXiv category or papers by co-authors, or both. Data is split chronologically so the test set always contains the newest papers. For live evaluation, the graph is updated incrementally: new first-author papers are added to node features, newly discovered co-authors are added to a prioritized candidate list and traversed by BFS, and outdated papers and inactive authors are pruned.

Evaluation uses BERTScore (with deberta-xlarge-mnli as backbone) and ROUGE-L, plus an LLM-as-a-Judge using Mixtral-8x22B-Instruct-v0.1 that reports win rate across five quality dimensions. All model calls go through the together.ai API with temperature fixed to 0. Each model's own tokenizer is downloaded from Hugging Face so inputs are truncated correctly. RALM runs use LangChain's RecursiveCharacterTextSplitter with chunk size 512 and chunk overlap 64, recalling up to 12 chunks.

Why This Matters

Impact on research. The paper argues that held-out future data is one of the most effective ways to keep open benchmarks honest, and it provides a concrete recipe for doing that with graph-structured, automatically labeled data. It also argues that existing long-context summarization benchmarks consider only one level of summarization, while AcademicEval spans multiple abstraction levels through its compression rate measure (input tokens divided by output tokens), which ranges from 6 for Intro-8K to 587 for Title-10K.

Real-world applications:

• Automated drafting assistance for researchers preparing titles, abstracts, introductions, or related work sections.
• Evaluation of long-context models for document-heavy enterprise workflows where inputs are far longer than a chat prompt.
• Benchmark infrastructure for retrieval systems, since the paper compares BM25, TF-IDF, DPR, Contriever, and Dragon within a single long-context generation setting.
• Contamination-resistant model evaluation for labs and regulators who need test sets that cannot be memorized from pretraining data.

Industry relevance. The finding that retrieval-augmented 8K models can match or beat 32K-context models on several settings has direct cost implications: practitioners may not need the longest context windows, and retrieval quality (dense over sparse) may matter more. The divergence between overlap-based metrics and judged quality also warns teams not to tune solely against BERTScore or ROUGE-L.

Future Directions

• Disentangling in-context learning from in-weights learning. The paper explicitly invites the benchmarking community to design datasets and update schedules that better separate the two effects, since the observed plateau partly reflects limited ability to use ultra-long inputs.

• Scaling few-shot demonstrations further. The authors note that context length can be scaled by adding more demonstrations to provide more informative references; the full -M settings (Title-50K-M, Abs-48K-M, Intro-48K-M, Related-72K-M) were collected but not evaluated because the context was too long for the selected baselines.

• Improving long-context utilization. The results show current LLMs cannot exploit long few-shot demonstrations well, leaving open the question of how to make in-context learning effective at tens of thousands of tokens.

• Broadening the task suite and update cadence. The paper leaves open how update cycles (monthly or quarterly are suggested) and graph pruning thresholds should be tuned to balance freshness, connectivity, and graph size.

Target Audience

Researchers and engineers working on long-context LLMs, benchmark designers concerned with data contamination and label leakage, retrieval-augmented generation practitioners, and anyone studying in-context learning. The paper is also relevant to teams evaluating whether long-context models or retrieval pipelines are the better fit for long-document generation tasks.

Authors’ abstract

Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval