PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback

AI summary

Overview

Research area: Natural language processing and multimodal generative AI, specifically automatic generation and iterative refinement of scientific diagrams from paper content.

Technical level: Advanced. The paper assumes familiarity with vision-language models, multi-agent pipelines, LLM-as-judge evaluation, and benchmark construction.

Scope: The paper introduces a multi-turn scientific diagram refinement benchmark (MTPaperBananaBench) with a simulated user, diagnoses two failure modes of existing systems (quality drift and forgetting), and proposes a multi-agent refinement system (PaperBanana-Interact) that mitigates them.

What This Paper Is About

Recent systems can draft a scientific figure from a paper's content, but a single generation pass rarely matches an author's visual and communicative preferences. The paper documents that gap with a study of 14 participants, all of whom asked for further revisions after seeing an initial draft, and then builds a benchmark and a system for the multi-turn refinement workflow that authors actually follow. The goal is a diagram generator that improves across turns of natural-language feedback instead of degrading.

Key Contributions

1. MTPaperBananaBench, described as the first multi-turn diagram generation benchmark, containing 292 diagrams annotated with 3,518 user requirements across content, organization, and visual representation categories and three severity levels, paired with a user simulator that reveals hidden requirements turn by turn.
2. A systematic multi-turn evaluation of generative-model refiners (NanoBananaPro, GPT-Image-2) and agentic refiners (PaperBanana-DirectRefine) on top of several single-turn generators (NanoBananaPro, GPT-Image-2, AutoFigure, AutoFigure-Edit, PaperBanana, Crafter), which surfaces two shared failure modes: quality drift and forgetting.
3. PaperBanana-Interact, a multi-agent refinement system combining a summarizer that compresses multi-image interaction history into textual memory, a critic-refiner-visualizer internal loop, and a multi-objective critic that checks the current request, all prior requests, source-context faithfulness, and presentation quality.
4. An ablation study isolating the contributions of the multi-objective critic design, the history representation (summarized memory, no history, or full history), and the internal iteration budget.

Main Findings

• All users wanted revisions: In a formative study with N = 14 participants, every participant requested further revisions after viewing an initial draft, and 86% rated the refined diagram as more satisfactory. Average satisfaction rose from 3.2 to 4.2 on a 1–5 scale, with 12 of 14 participants rating the refined diagram higher, by 0.5 to 2 points.
• Revisions emerge from seeing drafts: Nine participants reported that the initial draft shaped or changed their vision of how the diagram should be presented, and this pattern appeared in 64% of sessions. Participants stopped after a median of 5 interaction turns, within a maximum study duration of 30 minutes.
• Baseline failures observed in the study: Interview feedback cited reduced factual faithfulness (29%), degraded aesthetic quality (43%), and failure to preserve previously satisfied requests (29%).
• Quality drift: Generative-model refiners degraded diagram quality relative to their own single-turn output. The paper reports quality moving 50.3 → 19.0 for NanoBananaPro and 50.3 → 47.1 for PaperBanana-DirectRefine. The drift was amplified in the k = 3 setting, where feedback contains multiple requirements per turn.
• Forgetting: Across all baselines, 14.1–22.9% of previously satisfied requests became unsatisfied in later turns. Forgetting rates ranged from 14 to 23% under both k = 1 and k = 3 simulator settings.
• Single-turn capability is not the bottleneck: All refiners could follow a request reasonably well in the turn it was raised, with per-turn requirement satisfaction rates (Req^pt) ranging from 50 to 80 across all systems. The paper reports single-turn satisfaction rates of 68.6% for PaperBanana-DirectRefine and 56.2% for NanoBananaPro.
• PaperBanana-Interact reduces both failure modes: The paper reports quality-score gains of 11.9–18.6 points (phrased as "percent" in the contributions section) over baselines and forgetting reductions of 3.7–6.2 points/percent. In the k = 1 setting, refining PaperBanana outputs gave quality 61.2, requirement satisfaction 58.0, per-turn satisfaction 77.2, and forgetting 12.6, versus 47.1 / 52.6 / 68.6 / 18.8 for PaperBanana-DirectRefine and 19.0 / 45.2 / 56.2 / 19.0 for NanoBananaPro.
• Forgetting is reduced, not eliminated: PaperBanana-Interact still shows a forgetting rate of 10–13%.
• Human preference validation: In a pairwise evaluation by a human annotator over 150 samples, PaperBanana-Interact outputs were preferred over NanoBananaPro in 81.3% of cases and over PaperBanana-DirectRefine in 76.7% of cases.
• Critic design matters: Replacing the multi-objective critic with a naive unstructured critic lowered per-turn satisfaction by 4.9 points and quality by 2.3 points, while slightly increasing forgetting.
• Summarized memory beats full history: Removing history entirely raised forgetting from 12.6 to 14.7; supplying the full history slightly lowered forgetting but degraded per-turn requirement satisfaction (77.2 → 73.5) and final quality while consuming 37.1% more input tokens.
• Iteration budget dominates: Performance declined monotonically as the internal iteration budget τ_max decreased from 10 to 1, with τ_max = 1 (a single refinement pass) dropping quality to 43.5 and raising forgetting to 16.8.
• Annotation reliability: Manual review removed 0.5% and revised 24.7% of candidate requirements. Inter-annotator agreement on a subset of 207 requirements was Cohen's κ = 0.767, and VLM-judge agreement with a human on 200 requirements was Cohen's κ = 0.812.

Methodology in Plain English

The authors first ran an interview-style study to confirm that figure creation is iterative in practice. They then turned that workflow into a benchmark: for 292 diagrams from the existing PaperBananaBench dataset, Gemini-3.1-Pro proposed candidate user requirements grounded in each source paper and its human-drawn reference figure, and three human annotators reviewed, revised, or removed them. Each diagram has requirements spanning content, organization, and visual representation, with severity labels.

Because collecting real human feedback at scale is expensive and hard to reproduce, the authors replaced the human with a simulator. At each turn, an LLM-as-judge checks the current diagram against the full hidden requirement list, collects the failed requirements, and turns the first k of them (k = 1 or k = 3) into natural-language feedback using the judge's rationales. The interaction runs for up to T_max = 5 turns, stopping early if all requirements are satisfied. Final diagrams are scored both on how many requirements are satisfied and on overall quality, where quality is measured by comparing against the human reference diagram across faithfulness, conciseness, readability, and aesthetics. Two extra diagnostics capture multi-turn behavior specifically: per-turn requirement satisfaction and forgetting rate.

The proposed system, PaperBanana-Interact, wraps this in three ideas. A summarizer compresses the whole multi-image history into a short textual memory that all agents share, avoiding repeated image processing. An internal loop repeatedly alternates between a critic that inspects the current image and a refiner that updates the image-generation prompt, with a visualizer rendering each new revision, stopping when the critic is satisfied or after 10 iterations. Crucially, the critic is multi-objective: it checks the current request, all earlier requests, faithfulness to the source paper, and presentation quality, emitting structured critiques for each. Baselines refine either the image directly with a generative model or the prompt directly with a vision-language model.

Why This Matters

Impact on research: The paper reframes scientific figure generation as an interactive, multi-turn problem and shows that strong single-turn performance does not translate into strong multi-turn performance. It supplies a reproducible benchmark and user simulator so that the quality-drift and forgetting failures can be measured without new human studies each time, and it names those failures explicitly so future systems can be designed and ablated against them.

Real-world applications (as grounded in the paper's framing):

• Drafting and iterating on the main methodology figure of a paper before submission.
• Redesigning an existing published figure for a keynote presentation or journal extension, which the study explicitly asked participants to simulate.
• Assisting researchers without design training in producing readable, well-organized diagrams from their own method descriptions.
• Supporting assistive use in academic writing workflows, subject to the paper's stated requirement that generated images be disclosed in publications.

Industry relevance: The work targets a common bottleneck in scientific publishing and technical communication, where figure quality affects how a method is understood and reviewed. It also demonstrates a general recipe for multi-agent refinement with long multi-image histories and multiple competing constraints, which is relevant to any product that iterates on generated visual content under user feedback.

Future Directions

• Reducing the residual 10–13% forgetting rate, since the system mitigates but does not eliminate the problem.
• Closing the gap between multi-turn gains and single-turn behavior, particularly under multi-requirement turns (k = 3), where the authors observed more modest gains and amplified quality drift.
• Using the benchmark to evaluate other refiners, since AutoFigure and AutoFigure-Edit were excluded from multi-turn evaluation due to low performance.
• Improving the history representation, as full history slightly reduced forgetting while degrading per-turn satisfaction and consuming 37.1% more input tokens; the cost/benefit of memory designs is left open.
• Extending beyond the current study setup, which used a maximum of 5 refinement turns and a single simulated-user backbone; the paper does not report results for longer interaction budgets or real human interaction at benchmark scale.

Target Audience

Researchers and practitioners working on multimodal generation, agentic refinement systems, or human-AI interaction for creative and scientific authoring. It is most useful to those building diagram- or figure-generation tools, those designing benchmarks with simulated users, and NLP or HCI researchers studying multi-turn feedback loops. Readers need comfort with vision-language models and multi-agent pipelines to follow the method and ablation sections in detail.

Authors’ abstract

Recent efforts have aimed to automate scientific diagram generation from paper content (Lin et al., 2026; Zhu et al., 2026a). However, fully satisfying an author's visual and communicative preferences in a single turn is challenging: in our formative user study (N = 14), all participants requested further revisions after viewing an initial draft, and 86% of them rated the refined diagrams as more satisfactory. Despite the clear demand, the multi-turn workflow remains largely underexplored. To bridge this gap, we present MTPaperBananaBench, a benchmark for multi-turn diagram generation containing 292 images annotated with 3,518 user requirements. To reduce expensive human studies and enable scalable benchmarking, we construct a user simulator that, at each turn, identifies unsatisfied requirements and converts k of them into natural language feedback. Evaluating both requirement satisfaction and overall diagram quality reveals two key failure modes shared across baseline multiturn systems: (1) quality drift, where diagram quality progressively declines over turns, and (2) forgetting, where previously implemented features are lost in subsequent turns. To address these issues, we introduce PaperBanana-Interact, a multi-agent system that refines diagrams via an internal critique-and-refine loop. PaperBanana-Interact consistently improves rather than degrades diagram quality across turns, outperforming baselines by 11.9-18.6 points in quality score and reducing forgetting by 3.7-6.2 points.