Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning

AI summary

Overview

Research area: Natural Language Processing; specifically scientific figure captioning, multimodal generation, evaluation methodology, and human-AI co-writing.

Technical level: Intermediate. The paper is a retrospective and position piece rather than a technical systems paper, so it is readable by a general audience, but it assumes familiarity with concepts such as domain-specific language models, reference-based metrics (BLEU, ROUGE), and the LLM-as-judge paradigm.

Scope in one sentence: A five-year retrospective (2021–2025) of the SciCap project that summarizes the datasets, evaluations, challenges, and writing-assistant systems the team built, and lays out five unsolved problems in scientific figure captioning.

What This Paper Is About

Scientific figure captions are often vague or under-informative, and writing a good one is hard because the needed information is scattered across the paper, not contained in the image alone. Before SciCap, no large-scale, real-world dataset of scientific figures paired with their captions existed, which made it impossible to build and measure captioning technology. This paper reviews how the authors closed that data gap, what they learned about generating and evaluating captions as large language models arrived, and what remains unsolved.

Key Contributions

1. A retrospective account of the SciCap project. The paper documents how a seed-funded idea at Penn State grew into a multi-institution collaboration with Adobe Research, spanning datasets, benchmarks, annual challenges, and interactive writing tools.

2. A synthesis of methodological lessons across generation and evaluation. It consolidates findings on text-only contextual summarization, LLM-as-judge evaluation, and the persistent trade-offs between automatic and human evaluation.

3. A body of released artifacts summarized in one place. These include the original SciCap dataset, the SciCap Challenge dataset, the LaMP-Cap personalized captioning dataset, and the SciCapenter writing assistant.

4. A position statement with five named unsolved challenges. The authors argue that many of the hardest remaining problems are fundamentally human-centered, not purely modeling problems.

Main Findings

• The original dataset was large and real. The team gathered 295,028 papers published between 2010 and 2020, extracted over 2.17 million figures, identified 416,804 graph plots (19.2%), and after filtering out compound figures produced 133,543 single-panel figures.

• Early baselines failed badly. In the 2021 SciCap release paper, baseline vision-to-language models produced captions that were low-quality, inaccurate, or outright gibberish.

• Captioning behaves like contextual summarization, not visual description. Fine-tuning the Pegasus summarization model on only the figure-mentioning paragraphs, with no images, worked remarkably well. Roughly 75% of caption words could be traced directly to those paragraphs or the figure's OCR.

• Many author-written captions are mediocre. PhD students rated over 50% of author-written captions as unhelpful.

• LLM judges align with experts, not novices. Published at EMNLP 2023, the study found LLM judgments aligned well with PhD students' ratings, while undergraduate ratings diverged: less-experienced readers cared about clear take-away messages, experts prioritized numerical precision and technical detail. The work deliberately excluded LLM-generated captions from the evaluation pool, which avoided the self-preference bias later reported by Panickssery et al. (2025).

• Multimodal LLMs beat human captions under editor review. In the TACL study built on the 2023 Challenge submissions, GPT-4V consistently produced captions preferred by professional editors over all other approaches, including human-written captions, across domains, figure types, and even papers published after the models' claimed training cut-off.

• Challenge design changed results repeatedly. In 2023, rankings barely changed between a full-test-set ROUGE-2-Normalized award and an award on a curated subset of higher-quality captions. In 2024, using human evaluation on a sample of 200 figures, teams outperformed original authors in the long-caption track while authors performed better in the short-caption track. In 2025, teams could mimic the author's style per automatic evaluation, but human evaluators consistently preferred GPT-4o's vanilla captions over personalized ones.

• Personalization helps, but only in moderation. On the LaMP-Cap dataset across 110,828 target figures, adding multimodal profiles consistently helped four tested LLMs generate captions closer to the original author-written ones, and human judges preferred captions generated with one profile figure over both no-profile and all-profile conditions.

• Writers start from the AI draft. In a study where 18 researchers rewrote captions from their own recently published papers, writers typically began by copying or adapting AI-generated captions, preferred longer and detail-rich drafts, and relied heavily on textual–visual alignment of suggestions, but struggled when figures were conceptually complex or when suggestions conflicted with disciplinary norms.

• Missing context is common. 18.81% of figures in the original SciCap dataset lacked any identified mentions.

Methodology in Plain English

The researchers worked in four loosely sequential phases.

Build the data. They scraped arXiv papers from a defined publication window, extracted every figure, filtered for graph plots and then for single-panel figures, and preserved each figure's caption, OCR text, figure type, and the paragraphs in which the paper explicitly mentioned that figure.

Test whether captions need images at all. Rather than assuming a multimodal model was required, they reframed captioning as summarization and trained a text-only model on the figure-mentioning paragraphs. The strong performance of this simple setup reframed the whole task.

Rethink evaluation. Because reference-based metrics like BLEU and ROUGE did not correlate reliably with human judgment, and because expert human evaluation is expensive and hard to staff, they compared LLM ratings against ratings from PhD students and trained undergraduates.

Run public challenges and study writers. They invited teams to generate captions on an expanded hidden-test dataset, then evaluated submissions with a mix of automatic metrics and human judgment. Separately, they built and studied an interactive writing assistant, and ran a controlled study in which real authors rewrote their own captions with LLM drafts as starting points.

What this paper itself does. This is a position and retrospective paper. It does not report a new model, a new benchmark result, or a new experiment; the numbers cited above are drawn from the team's earlier publications and challenge reports.

Why This Matters

The paper's central argument is that a task which looks small and well-defined from the outside—write a caption for a figure—turns out to expose deep problems in generation, evaluation, and human-AI workflow design. It shows that automatic evaluation for this task remains unreliable, that expert judgment is hard to scale, and that stylistic personalization can trade off against factual clarity. It also documents a rare case where AI-generated captions were preferred by professional editors over human-written ones, which raises practical questions about caption authorship and quality control in publishing.

Real-world applications:

• Assistive writing tools for researchers. SciCapenter-style systems that show the figure, its mentioning paragraphs, and multiple AI drafts alongside quality ratings and contextual guidance.
• Pre-submission quality improvement in scientific publishing. Flagging or improving captions before a paper goes out, since readers currently have little control over caption quality.
• Standalone visuals without accompanying text. Figures circulating on social media, in news, or in advertisements, where no figure-mentioning paragraphs exist.
• Audience-adapted scientific communication. Producing different caption versions for experts, students, and practitioners based on measured differences in what each group values.

Industry relevance: The work was conducted in collaboration with Adobe Research, indicating direct commercial interest in document intelligence and authoring tools. The results are relevant to LLM and multimodal model providers seeking evaluation methodology that is cheaper than expert human review, and to scientific publishers and preprint platforms that want to improve figure accessibility at scale. Two of the paper's named challenges—evaluation at scale and personalization without extensive user profiling—map onto general problems in deploying generative AI in professional workflows.

Future Directions

1. Captioning with incomplete or missing context. Nearly a fifth of figures in the original SciCap dataset had no identified mentions, and standalone figures have none by definition. Generating meaningful captions from the image alone remains unsolved.

2. Audience-aware captioning. Reader expertise demonstrably shapes caption preference, but building systems that adapt to audience without relying on extensive user profiling is still open.

3. Resolving the personalization trade-off. Both LaMP-Cap experiments and the 2025 Challenge show that strong stylistic imitation can reduce factual accuracy or clarity. A broader question the authors pose is whether personalized text generation is needed at all for writing support, or whether generic generation is sufficient for most authors to build on.

4. Studying the writing process, not just the output. Most caption research evaluates from the reader's perspective. The authors note that recruiting authors who are actively drafting their own papers is difficult, that proxy studies only approximate the real task, and that the Sloan-funded deployment of SciCapenter offers an opportunity for ecologically valid observation, though such studies remain expensive.

5. Bridging captioning and figure understanding. Figure parsing, chart reasoning, and figure QA research has evolved largely in isolation from user-facing captioning, and many captioning projects—including the authors' own—did not fully evaluate faithfulness or correctness against visual content. Joining these threads could reduce duplicated effort and yield systems that both understand and explain scientific visuals.

Target Audience

Researchers and practitioners in NLP, multimodal machine learning, and human-computer interaction who work on text generation, evaluation methodology, or writing-assistance systems will find the methodological lessons most directly useful. It is also valuable for scientific publishers, research-tool builders, and teams deploying LLMs in authoring workflows, since it documents real trade-offs in evaluation cost, personalization, and caption quality. Scientists who write papers will recognize the described problems with caption authoring. Because the paper is a retrospective rather than a technical report, newcomers to scientific figure captioning can read it as an entry point to the subfield's datasets, benchmarks, and open problems without needing to reproduce any experiments.

Authors’ abstract

Between 2021 and 2025, the SciCap project grew from a small seed-funded idea at The Pennsylvania State University (Penn State) into one of the central efforts shaping the scientific figure-captioning landscape. Supported by a Penn State seed grant, Adobe, and the Alfred P. Sloan Foundation, what began as our attempt to test whether domain-specific training, which was successful in text models like SciBERT, could also work for figure captions expanded into a multi-institution collaboration. Over these five years, we curated, released, and continually updated a large collection of figure-caption pairs from arXiv papers, conducted extensive automatic and human evaluations on both generated and author-written captions, navigated the rapid rise of large language models (LLMs), launched annual challenges, and built interactive systems that help scientists write better captions. In this piece, we look back at the first five years of SciCap and summarize the key technical and methodological lessons we learned. We then outline five major unsolved challenges and propose directions for the next phase of research in scientific figure captioning.