Leveraging LLMs for Semi-Automatic Corpus Filtration in Systematic Literature Reviews

AI summary

Overview

Research area: Human-AI collaboration, large language models, and visual analytics applied to academic knowledge work — specifically the corpus retrieval and title/abstract screening stages of systematic literature reviews (SLRs).

Technical level: Intermediate. The conceptual pipeline is easy to grasp, but the evaluation relies on classification metrics (accuracy, precision, recall, F1, TP/FP/TN/FN counts) and multi-model consensus design.

Scope in one sentence: The paper proposes and evaluates a human-supervised pipeline in which multiple LLMs independently classify candidate papers and a conservative consensus scheme decides inclusion, tested on a ground-truth corpus of 8,323 papers from a real survey.

Note on completeness: The supplied content is truncated at the end of Section 5.3 (Prompt Optimization). The paper's stated discussion section on challenges and future trajectories is listed as a contribution in the introduction, but its content is not present in the provided text, so it is not summarized here.

What This Paper Is About

Systematic literature reviews are the gold standard for structured research synthesis, but the screening stage — deciding which of thousands of keyword-retrieved papers are actually relevant — is slow, repetitive, and cognitively demanding. This paper asks whether a supervised pipeline of multiple large language models, voting through a consensus scheme, can filter a paper corpus while losing fewer relevant papers than a single human annotator would. The authors implement the pipeline in an open-source visual analytics tool, LLMSurver, and benchmark model generations from mid-2024 and fall 2025 on real ground-truth data.

Key Contributions

1. A conceptual pipeline for semi-automatic paper corpus filtration that uses multiple LLM agents voting in a consensus scheme, built on the PRISMA framework and keeping the human in control of database selection, search, preprocessing, model choice, prompts, and consensus design.
2. LLMSurver, an open-source web application (MIT License) implementing the pipeline, with a paper table, prompt editor, LLM registration and execution controls, a consensus and statistics view, and comparative visual plots; deployable at no cost and usable without a backend.
3. A thorough evaluation on a large SLR dataset (8.3k papers) covering five state-of-the-art models from mid-2024 and 13 modern models from fall 2025, eight of which are openly available, plus an analysis of how alternative prompts change results for one open model.
4. A detailed discussion of current challenges, emerging potentials, and future trajectories of AI-driven literature filtration (listed as a contribution; the discussion text itself is not included in the supplied content).

Main Findings

• The consensus scheme is what makes the pipeline work. A paper is discarded only if all participating LLMs vote for rejection, and included if at least one recommends inclusion. This prioritizes recall, which the authors treat as the most important property because losing a relevant paper is the costliest error.

• Mid-2024 models: high accuracy, weak precision. All five models exceeded 90% accuracy. GPT-4o led on accuracy (99.30%), precision (61.54%), and F1 (73.39%). Llama-3 (8B) was the most conservative, with 774 false positives but only 2 false negatives (97.73% recall, 10.00% precision).

• Consensus in mid-2024 lost only one relevant paper. Both Consensus (All) and Consensus (Best — Gemini 1.5 Flash, Claude 3.5 Sonnet, GPT-4o) reached recall of 98.86% with 87 true positives and 1 false negative. A manual review found that single case ambiguous even for the human evaluators.

• The three-model consensus cut workload substantially. Consensus (Best) produced 167 false positives versus 862 for Consensus (All) — reducing the number of papers requiring manual filtering by 695 while using three models instead of five.

• Errors were usually attributable to single models. For false inclusions, in most cases only one model — most often Llama-3 8B — was responsible. For incorrectly excluded relevant papers, Gemini 1.5 Flash was most often the culprit. Overlaps among multiple models were rare.

• Fall 2025 models improved markedly. All 13 models achieved accuracies ranging from approximately 95% to 99%, with improved precision and F1. Llama 3.1 8B reduced the mid-2024 Llama-3 8B's 774 false positives to 98.

• Qwen3 (235B) had the highest individual scores — accuracy 99.45%, precision 70.59%, F1 75.79% — but at the cost of 16 falsely discarded papers.

• Consensus (All) in fall 2025 retained every relevant paper (recall 100%, 0 false negatives) but produced 653 false positives, increasing manual review load.

• The best fall-2025 configuration was a three-model consensus of GPT 5, Claude Sonnet 4.5, and Llama 3.3 (70B): recall 100%, 0 false negatives, only 166 false positives, accuracy 98.01%, precision 34.65%, F1 51.46%.

• Consensus among only small open 8B models was the weakest. It was the only fall-2025 scheme to exclude relevant papers (6 false negatives), with recall 93.18% and F1 27.52% — attributed to limited reasoning capability or the need for better prompts.

• Open models are now sufficient. A combination of open models retained all relevant papers while including slightly more than 300 irrelevant ones (Consensus Open Large: 0 false negatives, 382 false positives).

• Prompt changes trade precision for recall dramatically. For Llama 3.1 8B, the original prompt gave the best accuracy (98.63%), precision (42.35%), and F1 (55.81%) with 16 false negatives. Prompts P1–P7 raised recall — up to 100% for P7 — but at heavy cost: P3, P4, and P5 dropped accuracy to 79.68%, 79.71%, and 78.49%, with precision of 4.90%, 4.90%, and 4.64%. P7 achieved zero false negatives but accuracy of 52.47%, precision 2.18%, and 3,956 false positives.

• The failure mode was over-strict inclusion rules, not extraction errors. When examining Llama 3.1 8B's reasoning, the model correctly identified network visualization in immersive contexts but judged the applications too specific or not centered enough on network analysis, leading to exclusions.

• One model had output-format problems. DeepSeek R1 0528 8B produced incomplete or irrelevant answers in 34 cases; these papers were included, since avoiding loss of relevant items is the priority.

Methodology in Plain English

The authors start from a real survey — "Visual Network Analysis in Immersive Environments: A Survey" — and the corpus that was gathered for it. Using PRISMA, they searched paper titles and abstracts in the ACM Digital Library, IEEE Xplore, and Eurographics, unified metadata, removed duplicates, and excluded non-paper entries, arriving at 8,323 candidate papers. Multiple researchers spent several weeks manually screening titles and abstracts, producing ground truth: 88 relevant and 8,235 irrelevant papers.

The pipeline then replaces that manual screening with LLM agents. Each paper's title and abstract are fed to each participating model along with a customized prompt that defines the model's role, the topic, inclusion and exclusion criteria, and a required answer format of INCLUDE or DISCARD plus a two-sentence justification. Every model classifies the entire corpus, producing potentially differing votes per paper. A consensus scheme harmonizes them: include if any model says include, discard only if all models agree to discard.

A human stays in the loop throughout — designing and refining prompts, sampling and inspecting outputs, examining the models' stated reasoning, and choosing which models and which consensus scheme to use. This is supported by LLMSurver, a containerized single-page React application that runs locally in the browser (only LLM API calls leave the environment), accepts BibTeX files, code, or DOI numbers as input, stores intermediate results, exports to CSV, and visualizes classification distributions and model agreement levels to highlight outliers.

Three evaluations were run: five mid-2024 models (two open, three commercial); 13 fall-2025 models (eight open, five commercial); and a prompt-variation study using seven alternative prompts on Llama 3.1 8B, which had produced more balanced results than the other open 8B models. The alternative prompts reversed the uncertainty rule, added task context, removed explicit criteria, or were generated by GPT 5 — and the resulting behavior is captured in the trade-offs above.

Why This Matters

Impact on research: Manual title-and-abstract screening is a bottleneck that scales badly. The paper cites Wallace et al.'s figure that an experienced reviewer screens about two papers per minute, meaning roughly 8,000 papers take around 66 person-hours — about one and a half full workweeks of uninterrupted effort, and in practice several months once fatigue, dual verification, and competing commitments are factored in. Showing that a supervised multi-model pipeline can match or beat single-annotator error rates while running at machine speed makes large, fast-moving research domains tractable for systematic review.

Real-world applications:

• Academic SLR production in fast-growing or semantically ambiguous fields where keyword searches return large volumes of irrelevant results, as in the immersive-analytics survey used here.
• Research groups and libraries with data-privacy constraints, since open and small models can be deployed locally on institutional servers or personal computers without sending data to external paid providers.
• Institutionally hosted review services and tooling, through LLMSurver itself — a self-hostable, browser-local application that can be adapted with custom consensus strategies and decision visualizations.
• Cost-constrained research settings, where the finding that modern open models suffice means reviews do not depend on commercial API budgets.

Industry relevance: The same retrieve-then-filter pattern appears in legal document review, patent landscaping, due diligence, and enterprise knowledge management, where large candidate sets must be screened against inclusion criteria. The paper's design lessons — use multiple diverse models, make the consensus conservative toward recall, keep a human supervising prompts and model selection, and measure precision/recall explicitly — transfer directly to production document triage systems. The local-execution architecture also matters for organizations that cannot export sensitive corpora.

Future Directions

• Closing the small-model recall gap through prompts. The paper's own result is that prompt relaxation can eliminate false negatives (P7 reached 100% recall) but destroys precision. A prompt design that achieves high recall without thousands of false positives remains unresolved.
• Better consensus schemes. The authors note that strategies such as using an additional LLM to analyze individual decisions and rationales are possible, but they only evaluate all-model and selected-subset voting. Whether rationale-aware consensus beats simple voting is untested in the reported content.
• Model selection as an optimization problem. The best combinations were found by inspecting results (GPT 5, Claude Sonnet 4.5, Llama 3.3 70B for fall 2025; Gemini 1.5 Flash, Claude 3.5 Sonnet, GPT-4o for mid-2024). The paper does not report an automated method for identifying the smallest sufficient model subset.
• Recovering false exclusions downstream. The authors argue that a small number of false removals may be acceptable because they can typically be recovered in a later snowballing step — but that recovery is not quantified here.

Target Audience

Researchers and practitioners who conduct or support systematic literature reviews — especially in computer science and adjacent fields where keyword searches are noisy. It is also relevant to visualization and visual analytics researchers interested in human-in-the-loop interfaces for model supervision, to research software engineers who want a deployable open-source tool for corpus screening, and to anyone designing human-AI document triage workflows where recall matters more than precision. Readers should be comfortable with standard classification metrics and with the general capabilities and limitations of large language models.

Authors’ abstract

The creation of systematic literature reviews (SLR) is critical for analyzing the landscape of a research field and guiding future research directions. However, retrieving and filtering the literature corpus for an SLR is highly time-consuming and requires extensive manual effort, as keyword-based searches in digital libraries often return numerous irrelevant publications. In this work, we propose a pipeline leveraging multiple large language models (LLMs), classifying papers based on descriptive prompts and deciding jointly using a consensus scheme. The entire process is human-supervised and interactively controlled via our open-source visual analytics web interface, LLMSurver, which enables real-time inspection and modification of model outputs. We evaluate our approach using ground-truth data from a recent SLR comprising over 8,000 candidate papers, benchmarking both open and commercial state-of-the-art LLMs from mid-2024 and fall 2025. Results demonstrate that our pipeline significantly reduces manual effort while achieving lower error rates than single human annotators. Furthermore, modern open-source models prove sufficient for this task, making the method accessible and cost-effective. Overall, our work demonstrates how responsible human-AI collaboration can accelerate and enhance systematic literature reviews within academic workflows.