A Human-in-the-Loop Corpus for LLM-Based Simplification of Scientific Summaries

AI summary

Overview

Research area: Natural language processing, specifically automatic text simplification of scientific writing, human-in-the-loop corpus construction, and evaluation of large language model (LLM) outputs.

Technical level: Intermediate. The workflow is easy to follow, but the paper assumes familiarity with simplification metrics (BLEU, BERTScore, SARI) and readability formulas (Flesch-Kincaid Grade Level and Reading Ease).

Scope: The paper builds and releases a corpus of scientific summaries paired with GPT-4o-mini simplifications, reader difficulty annotations from cross-disciplinary STEM participants, and expert-edited reference simplifications, together with automatic evaluation results for both simplification conditions.

What This Paper Is About

Interdisciplinary research is growing, but scientific papers are written for specialists in one field: domain terminology, compressed arguments, and field-specific conventions make them hard for researchers in neighboring fields to read. The authors study whether LLMs can make scientific summaries more accessible to non-specialists, and where human expertise is still required. To answer this, they build a human-in-the-loop workflow on top of the SciSummNet corpus that combines LLM-generated simplifications, feedback from STEM readers outside computer science, and expert post-editing into a released corpus and evaluation setting.

Key Contributions

1. A human-in-the-loop methodology for scientific summary simplification that centers comprehension by cross-disciplinary STEM readers while aiming to maintain technical accuracy.
2. A curated corpus derived from SciSummNet, containing (a) baseline LLM outputs from GPT-4o-mini, (b) sentence- and phrase-level difficulty annotations from readers outside the source domain, and (c) expert-edited reference simplifications.
3. An evaluation showing where LLMs improve surface readability and where expert post-editing is crucial for claim calibration and terminology fidelity, using automatic metrics plus qualitative analysis.
4. A publicly released resource — the corpus, annotation artifacts, and code are available at https://github.com/faerber-lab/scientific-text-simplification-corpus.

Main Findings

• Readers preferred GPT simplifications for Understanding and Simplicity: across the completed Phase 1 evaluations, participants favored the LLM-generated summaries in 73 of 92 judgments for Understanding and 70 of 92 judgments for Simplicity. Each dimension comprised 92 comparative judgments.
• Naturalness judgments were more balanced: 42 of the 92 Naturalness judgments indicated no difference, while 14 favored the original summary, suggesting the LLM outputs occasionally contained stylistic inconsistencies or overly informal phrasing.
• Difficulty annotations clustered around specific phenomena: participants most frequently flagged multi-noun compounds, unexplained abbreviations, and domain-specific terminology.
• Two recurring tensions emerged from reader comments: (1) precision versus approachability — replacing terms such as latent variables, PCFG, or NP-hard with generic paraphrases can improve readability but reduce technical precision; and (2) style and naturalness — the LLM often improved local clarity but sometimes introduced awkward or overly informal formulations, whereas the original summaries generally retained a more conventional academic style.
• Expert editing covered a subset of the data: Phase 2 had 92 candidate summaries, of which 47 were completed and included as expert-edited references.
• Automatic metrics favored the GPT outputs: on the 47 aligned items, mean sentence-level BLEU was 0.949 for GPT-Simplified versus 0.349 for Expert-Edited, and mean BERTScore F1 (using distilbert-base-uncased) was 0.9907 versus 0.9031.
• The reference-free SARI variant was higher for GPT: 46.76 for GPT-Simplified versus 33.72 for Expert-Edited. Because the calculation uses empty references and does not include the standard reference-based keep component, the authors describe it as a descriptive indication of performed edit operations, not an overall quality judgment.
• Readability formulas favored the GPT outputs at the surface level: Flesch-Kincaid Grade Level was 12.10 for GPT-Simplified versus 13.00 for Expert-Edited, and Flesch-Kincaid Reading Ease was 48.54 versus 40.89.
• Source similarity does not equal semantic quality: the GPT outputs remained substantially closer to the wording of the original summaries, but the authors note this does not by itself demonstrate higher semantic quality, since deliberate expert rewordings may preserve meaning while reducing lexical overlap.
• Expert edits followed a hybrid post-editing strategy: editors preserved domain-critical terminology such as computational lexicon and syntactic information while selectively adopting clearer LLM-generated formulations for general explanatory content.
• All differences are descriptive: no inferential significance tests were conducted, and the expert-edited summaries were not independently evaluated by target readers.

Methodology in Plain English

The authors start from SciSummNet, a corpus of 1,000 highly cited ACL papers with abstracts, citation contexts, and 150-word expert-written summaries. Those summaries capture central contributions but often keep dense terminology and complex sentence structures, so they serve as the source texts.

Baseline simplification. Each summary is simplified by GPT-4o-mini in a zero-shot setting, meaning the model receives instructions only, with no input-output demonstrations. The system prompt asks the model to simplify for readers without expertise in the field and for the general public, to identify complex words or key phrases, and to preserve the paragraph format. The user prompt supplies the paper title and source summary. The exact prompt is included in the released repository.

Phase 1 — reader evaluation. STEM participants from disciplines outside computer science, fluent in English, were shown an original SciSummNet summary and its GPT-simplified version in randomized order. They selected sentences they found difficult to understand, made three-way comparative judgments (prefer original, prefer GPT-simplified, or no difference) for Understanding, Naturalness, and Simplicity, and highlighted specific words or phrases they considered complex. Across completed evaluations this produced 92 comparative judgments per dimension.

Phase 2 — expert editing. Computer science graduate students or researchers with strong English proficiency received, for each item, the original summary, the GPT-simplified version, and the Phase 1 user highlights. They produced expert-edited reference simplifications following four guided scenarios: (1) both summaries contain difficult spans — refine and merge for clarity and fidelity; (2) only the original was flagged — verify and edit GPT simplifications where necessary; (3) only the GPT version was flagged — correct over-simplifications or awkward phrasing; (4) no flags — polish structure and verify meaning retention. Editing guidelines emphasized retaining technical terminology when required for accuracy, preserving the strength and scope of scientific claims, and using shorter, well-structured sentences where possible.

Evaluation. For the 47 items with all three versions available, the authors compared each original summary against its GPT output and against its expert-edited version. They report sentence-level BLEU averaged across the 47 items, BERTScore F1 with distilbert-base-uncased, a reference-free SARI variant computed with empty references, and Flesch-Kincaid Grade Level and Reading Ease computed over the concatenated outputs.

Why This Matters

Impact on research. Most existing simplification benchmarks are drawn from news articles or Wikipedia, giving limited coverage of scientific discourse and cross-disciplinary audiences. This work supplies both a dataset and an evaluation protocol for scientific text simplification, and its results reinforce prior criticism of relying on source-overlap metrics alone: higher similarity to the source does not necessarily imply greater accessibility or better preservation of scientific meaning.

Real-world applications.

• Helping researchers in neighboring fields read and reuse findings from papers outside their own discipline.
• Producing plain-language summaries of technical work for science communication, education, and public-facing material.
• Serving as a drafting aid for reviewers, editors, and science writers who need accessible versions that still preserve technical accuracy.
• Supporting policy and educational contexts where distorted or overstated scientific claims carry particular risk — the authors explicitly caution that simplified texts should be used with care in these settings.

Industry relevance. The findings suggest LLM-generated simplifications are most useful as initial drafts rather than directly publishable scientific texts. That positions the workflow as a quality-control pipeline: automated drafting plus cross-disciplinary reader feedback plus expert post-editing. For companies building summarization or documentation tools, the corpus supports training and benchmarking systems that must balance readability against terminology fidelity and appropriate claim strength.

Future Directions

• Extend the number and disciplinary coverage of the expert-edited summaries, which currently exist for only 47 of the 1,000 source summaries and are limited to English-language computer science texts.
• Evaluate expert-edited summaries with independent target readers, since they were not independently assessed by the intended audience in this study.
• Compare alternative models and prompting strategies systematically — the initial simplifications used GPT-4o-mini with a single fixed zero-shot prompt.
• Develop evaluation methods that jointly capture accessibility, factual fidelity, terminology preservation, and scientific claim calibration, since BLEU, BERTScore, SARI, and Flesch-Kincaid capture only selected aspects of similarity and readability.
• Broaden participant sampling beyond cross-disciplinary STEM readers and computer science experts to include social science or humanities researchers, educators, and policy professionals, and examine the reliability of difficulty annotations more systematically.

Target Audience

Researchers working on text simplification, scientific summarization, and LLM evaluation will find the corpus and protocol directly useful, as will NLP practitioners building accessibility or documentation tools. It is also relevant to science communication specialists, editors, and educators who need to understand the trade-off between readability and scientific precision. The paper is most valuable to readers with some familiarity with automatic evaluation metrics and readability formulas, though the overall workflow is described clearly enough for newcomers interested in human-in-the-loop data collection.

Authors’ abstract

Interdisciplinary research is accelerating, yet scientific papers remain difficult to understand outside their home fields. We study large language model (LLM)-based simplification of scientific texts and present a human-in-the-loop workflow that transforms expert summaries into more accessible versions for non-specialists. Using SciSummNet as the source corpus, we first generate baseline simplifications with GPT-4o-mini. In Phase 1, readers from STEM fields outside computer science identify difficult sentences and phrases and compare the original and GPT-simplified summaries in terms of comprehensibility, naturalness, and simplicity. In Phase 2, computer science experts use this feedback to create expert-edited reference simplifications. We release the resulting corpus together with human judgments and automatic evaluation results. The Phase 1 judgments show a clear preference for the GPT-generated summaries in terms of comprehensibility and simplicity, while qualitative analysis of the Phase 2 edits highlights the importance of preserving domain-specific terminology and the strength of scientific claims. The resulting resource supports the training and benchmarking of simplification systems for cross-disciplinary scientific communication.