SciForma: Structure-Faithful Generation of Scientific Diagrams

AI summary

Overview

Research area: Computer vision and generative modeling, specifically image-based generation of scientific methodology diagrams, with a focus on post-training alignment objectives. Technical level: Advanced — the paper assumes familiarity with diffusion/flow-matching models, DPO-style preference optimization, and multi-way ranking objectives, though the central argument about "conjunctive" correctness is stated plainly. One-sentence scope: The paper proposes SciForma, a data-plus-training framework built on three independently verifiable structural axes (Component, Arrow, Text) that aims to make open scientific-diagram generators structure-faithful enough to rival proprietary systems.

What This Paper Is About

Scientific methodology diagrams (process pipelines, architecture overviews) must render every component, arrow, and text label correctly, because a single reversed arrow or unreadable equation can invalidate the whole figure. Open-source models fail at this: supervised fine-tuning learns plausible layouts but cannot guarantee structural correctness, and scalar-reward post-training collapses distinct structural dimensions into one number, hiding which dimension failed. The paper's goal is to make diagram quality explicitly decomposable and individually enforceable at training time.

Key Contributions

1. A decomposable evaluation framework. The paper formalizes scientific diagrams into a "structural inventory" of three independently verifiable axes — Component (entity identity and spatial layout), Arrow (information topology), and Text (literal annotations) — extracted as a JSON checklist from a prompt and reference image, with per-axis scoring and critical (weight 1.0) versus moderate (weight 0.5) error severity.

2. SciFormaData-700K and SciFormaBench-2K. A dataset built from 726K methodology diagrams collected from 593K arXiv LaTeX sources (2015–2025), yielding 656K generation pairs and 70K editing triplets, plus a 2,000-sample logic-verified benchmark with human-verified inventories.

3. Multi-Dimensional Conjunctive Preference Optimization (M-DPO). A post-training method that constructs dimension-anchored preference pairs (one shared winner, axis-specific losers) and optimizes a multi-way Bradley–Terry objective whose gradient adaptively concentrates on the most deficient axis.

4. SciForma-9B and an inference-time refinement loop. A two-stage SFT model (SciForma-Base) post-trained with M-DPO into SciForma-9B, plus a verification-gated iterative editing pipeline that localizes and inpaints structural defects and rolls back edits that reduce global consistency.

Main Findings

• SciForma-Base lifts its initialization substantially. Two-stage SFT raises FLUX.2-klein-base-9B from 33.87 to 67.59 on SciFormaBench-2K, bringing it close to GPT-Image-1.5 (68.96).

• M-DPO surpasses GPT-Image-1.5. SciForma-9B reaches 69.51 overall, exceeding GPT-Image-1.5 (68.96) but remaining below GPT-Image-2 (85.62) and Nano Banana Pro (81.34). The largest M-DPO gains land on the two weakest axes: Arrow (+1.82) and Text (+3.16).

• Iterative refinement adds further gains. SciForma-9B + Edit reaches 72.40 (+2.89 over SciForma-9B), with Arrow again gaining most.

• AIBench results beat human-drawn originals by a small margin. SciForma-9B scores 70.29 versus 70.09 for the original images, and leads GPT-Image-1.5 (61.62) by 8.67 points. The largest margin is on Topology (+6.19 over originals: 64.17 vs. 57.98), while originals lead on Component (82.65 vs. 77.53), which the authors attribute to the 1024 px rendering limit on dense diagrams.

• M-DPO breaks the SFT plateau. From the same 90K SFT checkpoint, M-DPO achieves +1.92 average in 4K steps, versus +0.37 for 30K additional SFT steps — Text +3.16 vs. +0.46 and Arrow +1.82 vs. +0.01.

• Scalar-reward methods do not help. GDRO reaches 67.49 (−0.10 versus SFT) and GRPO 66.30 (−1.29), with GRPO degrading every axis (C −0.34, A −1.66, T −1.91).

• The conjunctive objective matters more than the multi-axis framing. Progressively: scalar DPO +0.83, DPO (Pareto) +0.96, M-DPO (mean) +1.72, full M-DPO +1.92. Peer methods CaPO (67.68, Text −1.34) and MCDPO (67.21, Text −1.85) both regress the Text axis below the SFT baseline.

• Agentic integration trails the strongest proprietary pairing. On PaperBanana (292 samples, Gemini-3-Pro judge, pairwise win rates against human drafts), PaperBanana + SciForma reaches 30.7% overall versus 60.2% for PaperBanana + Nano Banana Pro and 19.0% for PaperBanana + GPT-Image-1.5. Faithfulness is 34.5% versus 45.8% for Nano Banana Pro; Conciseness is the main bottleneck at 37.7% versus 80.7%.

• Automated scores correlate with human judgment. In a user study with 36 AI-specialized graduate students evaluating 30 samples, the Pearson correlation between per-sample SciFormaBench-2K scores and user preferences is r = 0.76.

Methodology in Plain English

The authors begin from a standards-inspired observation: in diagrams like UML activity diagrams and BPMN process models, meaning is carried by typed entities, directed relations, and text labels. They adopt this as a three-axis grammar and have a vision-language model extract, for any prompt-image pair, a ground-truth checklist of component coordinates, arrow source-target pairs, and literal text strings. Each discrepancy against that checklist is scored as critical or moderate, and each axis gets its own 0-to-1 score.

Training proceeds in stages. First, two rounds of supervised fine-tuning adapt a FLUX.2-klein-base-9B backbone: one stage on 656K generation pairs at roughly 768 px, then a joint stage on 244K high-quality pairs plus 70K editing triplets at roughly 1024 px, where source and target latents are concatenated with a temporal offset so one model handles both generation and localized editing.

The novel part is post-training. Instead of scoring candidates with one overall number, the system samples 12 candidates per prompt and scores each along the three axes. The best overall candidate becomes a shared "winner"; for each axis, the system picks a loser that is weak on that axis but competitive elsewhere. The loss requires the winner to beat every axis-specific loser simultaneously, which mathematically reduces to ordinary DPO when there is only one axis. A side effect is that the gradient weight on each axis is a softmax over performance deficits, so the most broken axis automatically receives the strongest learning signal.

At inference, the same checklist powers a loop: a critic model localizes defects, the model inpaints a semantic-block bounding box snapped to a 16×16 VAE grid, candidate edits are accepted only if the local score strictly improves, and any round that lowers the global score is rolled back.

Why This Matters

Impact on research: the paper reframes diagram quality from a continuous visual-matching problem into explicit, per-axis structural verification, and argues that decomposing the supervision signal along the same axes used to define validity is the real bottleneck — not model capacity. It also supplies a benchmark (SciFormaBench-2K, 2,000 samples) whose distinguishing features relative to ProImage-Bench (1,660), GENFIG1 (2,450), SciFlow-Bench (500), AIBench (300), and PaperBanana (292) are element-level verification, independent per-axis scores, difficulty tiers, and error severity.

Real-world applications:

• Automated figure drafting for papers, theses, and technical reports, reducing manual TikZ or SVG authoring.
• Generating architecture and pipeline diagrams for software documentation, engineering specs, and patents.
• Producing flowchart-like visuals for education, textbooks, and explainer content where a wrong arrow changes the meaning.
• Acting as the visualizer component inside agentic research-assistant pipelines, as demonstrated with PaperBanana.

Industry relevance: the results position an open 9B model as competitive with GPT-Image-1.5 on structural fidelity while remaining below GPT-Image-2 and Nano Banana Pro, and the framework's explicit checklists give an auditable failure report rather than an opaque quality score — valuable where diagram errors carry correctness consequences. The authors also advocate human verification and in-the-loop editing to prevent misuse in paper fabrication.

Future Directions

• Higher-resolution training. The 1024 px output resolution is named as a limitation that hurts dense diagrams with many small modules, which the authors link to the Component gap against human-drawn originals on AIBench (82.65 vs. 77.53).

• Open-source evaluators. The evaluation-and-editing pipeline currently depends on a proprietary VLM (GPT-5.4 for scoring; Qwen3-VL-8B-Instruct used for M-DPO labeling and an appendix cross-check), so replacing it with open models is flagged as an opportunity.

• Human-in-the-loop agentic frameworks. In agentic use, structural inventories are auto-generated rather than human-verified, risking drift from user intent.

• Conciseness and generalization of M-DPO. PaperBanana shows a large Conciseness gap (37.7% vs. 80.7% for Nano Banana Pro), and the authors invite the community to extend M-DPO's conjunctive design to other per-axis tasks.

Target Audience

Researchers and engineers working on controllable image generation, diffusion/flow-matching post-training, and structured visual synthesis will find the M-DPO formulation and the axis-decomposed ablations most useful. Benchmark and evaluation designers will benefit from the structural-inventory scoring scheme and its comparison against five existing diagram benchmarks. Practitioners building agentic paper-writing or documentation tools will care about the PaperBanana integration and the inference-time refinement loop. The paper is written at an advanced level: readers should already understand DPO, Bradley–Terry preference models, and flow-matching losses to follow Section 4.5 in detail.

Authors’ abstract

Structural fidelity is essential to scientific methodology diagrams. To communicate research logic, these diagrams must faithfully render components, directional relations, and textual annotations. Since a single error, such as a reversed arrow or an unreadable equation, can invalidate the entire figure, structural fidelity is inherently conjunctive: correctness on one axis cannot compensate for failure on another. Current open-source models fail to satisfy this criterion. Supervised fine-tuning (SFT) learns plausible layouts but cannot reliably ensure structural correctness, while scalar reward-based post-training obscures which structural dimension has failed. To address this, we introduce SciForma, a framework for the structure faithful generation of scientific methodology diagrams. Specifically, SciForma decomposes diagram quality into three structural axes: Component, Arrow, and Text, guided by a structural inventory. Built on this foundation, we curate SciFormaData-700K for structured training and SciFormaBench-2K for logic-verified evaluation. To close the gap left by SFT, we develop Multi-Dimensional Conjunctive Preference Optimization (M-DPO), which enforces simultaneous correctness across all axes and adaptively routes gradients to the most deficient dimension in post-training. The same structural inventory also enables iterative editing at inference time to correct residual errors. This combination allows SciForma-9B to exceed all open-source baselines and GPT-Image-1.5 on both SciFormaBench-2K and AIBench, bringing open scientific diagram generation close to proprietary-level structural fidelity. Our code and data will be available at: https://github.com/microsoft/SciForma.