PCoA: A New Benchmark for Medical Aspect-Based Summarization With Phrase-Level Context Attribution

AI summary

Overview

• Research area: Natural Language Processing — medical document summarization, factual attribution, and evaluation benchmark design.
• Technical level: Intermediate (assumes familiarity with summarization metrics, entailment models, and LLM prompting).
• Scope: The paper introduces PCoA, an expert-annotated benchmark of 1,799 medical aspect-based summaries with sentence-level citations and phrase-level attribution, plus a decoupled evaluation framework and a comparison of three attribution strategies across four LLMs.

What This Paper Is About

Large language models can write fluent summaries of clinical research, but in medicine a summary is only useful if a reader can verify it against the original text. Existing attribution methods typically point to whole documents or paragraphs, forcing users to search through long passages to check a single claim. This paper builds a benchmark where every aspect-specific summary of a randomized controlled trial (RCT) is linked to the exact supporting sentences and the specific phrases within them, and provides metrics to score summaries, citations, and phrases separately.

Key Contributions

1. PCoA dataset: 1,799 aspect-based summaries drawn from 152 melanoma RCT abstracts, each annotated by medical students with cited contextual sentences and aligned contributory phrases across 16 clinical aspects. Human evaluation scored completeness and conciseness above 4.6 on a 5-point scale, with inter-annotator agreement of 97.4% within-one and 92.1% exact match.
2. Decoupled evaluation framework: A fine-grained scheme that independently measures claim-level recall/precision (summary quality), sentence-level recall/precision (citation quality), and phrase-level recall/precision (attribution quality), rather than a single blended score.
3. LLM benchmarking: Systematic zero-shot evaluation of LLaMA3.1-70B, Mistral-Large-2411, DeepSeek-V3-0324, and GPT-4o under an intrinsic attribution setting, showing where current models succeed and fail.
4. Strategy comparison: Empirical evidence that prior attribution (identifying relevant sentences and phrases before summarizing) outperforms both intrinsic and post-hoc attribution on citation and phrase metrics.

Main Findings

• Overall model performance is modest and imbalanced: On the intrinsic setting, DeepSeek-V3 led summary evaluation (C-F1 0.659), DeepSeek-V3 and GPT-4o tied for best citation evaluation (S-F1 0.672), and Mistral-Large led phrase evaluation (P-F1 0.574). All models scored notably higher on recall than precision, suggesting they over-generate redundant or unsupported content.
• Phrase attribution is the hardest component: Precision on contributory phrases stayed near 0.45–0.52, well below sentence-level and claim-level scores, indicating LLMs struggle to isolate the exact supporting phrases.
• Performance varies sharply by medical aspect: DeepSeek-V3 scored highest on Funding (C-F1 0.942), Blinding (0.697), and Registration (0.860); it scored lowest on Findings (C-F1 0.382), Comparator (0.393), and Outcomes (0.546), where information is denser and harder to extract.
• More context complexity means worse performance: Claim recall falls as the number of reference subclaims grows, and both citation recall and precision decline as the number of cited sentences increases.
• Annotation quality is high: Supportive-sentence ratios exceed 88% for nearly all aspects (except Funding), subclaim attribution rates are at least 81% for most aspects, and phrase-to-summary ROUGE-1 averages at least 86%.
• Prior attribution wins: Compared to intrinsic (S-F1 0.67, P-F1 0.54) and post-hoc (S-F1 0.58, P-F1 0.48), prior attribution achieved S-F1 0.70 and P-F1 0.61 while keeping claim performance comparable (C-F1 0.63 vs. 0.66 and 0.62).
• Predicted attributions are weaker than reference attributions: Supportive citation rates for Outcomes, Findings, and Funding fell to 65% or below in model outputs, and phrase-summary ROUGE-1 dropped below 63% for Outcomes and Findings.
• Aspect coverage in source articles is uneven: Nearly all 152 articles reported Participants, Intervention, Outcomes, and Findings, but fewer than 70 reported Secondary Endpoints, Blinding, or Funding.
• Format compliance is not guaranteed: Mistral-Large produced only 1,609 of 1,799 template-conforming responses; LLaMA3.1 deviated on 3 responses.

Methodology in Plain English

The researchers started from a narrow, high-value slice of the medical literature: abstracts of randomized controlled trials on melanoma, published in the last ten years in top-quartile journals. They retrieved 607 candidates from PubMed and kept 152 after screening.

Two trained medical students then labeled every article in three passes. First, they tagged each sentence with the clinical aspects it addresses (aspects such as Participants, Intervention, Comparator, Outcomes, and Findings, following an expanded version of the PICO framework). Second, they wrote a short summary for each aspect using only the tagged sentences. Third, they marked the specific phrases inside those sentences that the summary actually depends on.

To score model outputs, the authors built a pipeline that breaks both reference and generated summaries into atomic factual claims, then uses an entailment model (TRUE) to check whether one summary's claims are supported by the other — yielding claim recall and precision. For citations, a cited sentence counts as correct only if it both appears in the reference citation set and independently entails at least one claim of the generated summary, a stricter standard than prior work that treats a group of citations as valid if they collectively support the summary. For phrases, a phrase counts as correct only if it is in the reference phrase set, appears verbatim or as a variant in the generated summary, and comes from one of the cited sentences. The pipeline uses Mistral-Large-2411 for claim decomposition and NLTK for tokenization.

Finally, the authors evaluated four models zero-shot through commercial APIs under three workflows: intrinsic (citations generated alongside the summary), post-hoc (summary first, citations retrieved afterward), and prior (relevant sentences and phrases retrieved first, summary written from that reduced input). Total compute cost was roughly $23.6.

Why This Matters

Impact on research. Aspect-based summarization and attribution have largely been evaluated with coarse-grained clicks or document-level citations. PCoA shows that phrase-level attribution is a substantially harder task, and it supplies a reproducible benchmark with human-verified ground truth so that future systems can be compared on equal footing. The decoupled metrics also expose a failure mode hidden by aggregate scores: a system can write a good summary while citing the wrong evidence.

Real-world applications.

• Clinical decision support: clinicians can verify a machine-written claim about a trial's outcomes in seconds by jumping to the exact supporting phrase instead of re-reading an abstract.
• Systematic reviews and evidence synthesis: reviewers comparing many trials on the same aspect gain structured, traceable extractions rather than untraceable prose.
• Patient-facing plain-language summaries: attributed summaries let patients and caregivers see where a statement comes from, which matters when the content informs treatment choices.
• Regulatory and journal workflows: publishers and regulators can audit AI-generated summaries against source text at the phrase level.

Industry relevance. Any organization deploying LLMs in regulated or evidence-heavy domains (health tech, pharma, clinical documentation, legal and financial research) needs attribution that is finer than paragraph-level. The finding that prior attribution beats post-hoc is directly actionable for product architecture: retrieve first, then generate, rather than generate and explain afterward.

Future Directions

• Make phrase matching semantic, not lexical. The current phrase metrics rely on surface matching (ROUGE-1 style), which penalizes correct paraphrases and inflates scores for verbatim copies; embeddings or entailment-based phrase matching would fix this.
• Reduce dependence on a single decomposition model. Claim decomposition quality drives the entire evaluation, and the authors rely on Mistral-Large-2411 with manual cleanup of roughly 10 anomalous outputs per 1,799. Cross-model or ensemble decomposition would test robustness.
• Broaden the domain beyond melanoma RCT abstracts. The current dataset is deliberately narrow; extending to full-text articles, other diseases, and other study designs would test whether aspect and attribution difficulty patterns generalize.
• Improve attribution on hard aspects. Outcomes, Findings, and Comparator remain weak across models; targeted retrieval, fine-tuning, or structured extraction for dense, multi-value aspects is an open problem.
• Explore attribution-aware training. Since prior attribution helps even with zero-shot prompting, fine-tuning models to retrieve-then-summarize could push citation and phrase precision further.

Target Audience

Researchers and practitioners working on summarization, retrieval-augmented generation, or factuality and attribution in LLMs, particularly those in biomedical NLP. It is also relevant to clinical informatics teams evaluating whether LLM-generated summaries are safe to deploy, and to benchmark designers interested in decoupled, component-level evaluation frameworks for grounded generation.

Authors’ abstract

Verifying system-generated summaries remains challenging, as effective verification requires precise attribution to the source context, which is especially crucial in high-stakes medical domains. To address this challenge, we introduce PCoA, an expert-annotated benchmark for medical aspect-based summarization with phrase-level context attribution. PCoA aligns each aspect-based summary with its supporting contextual sentences and contributory phrases within them. We further propose a fine-grained, decoupled evaluation framework that independently assesses the quality of generated summaries, citations, and contributory phrases. Through extensive experiments, we validate the quality and consistency of the PCoA dataset and benchmark several large language models on the proposed task. Experimental results demonstrate that PCoA provides a reliable benchmark for evaluating system-generated summaries with phrase-level context attribution. Furthermore, comparative experiments show that explicitly identifying relevant sentences and contributory phrases before summarization can improve overall quality. The data and code are available at https://github.com/chubohao/PCoA.