RAP: A Dataset for Paper-to-Poster Generation via Declarative Markup from Human-Crafted Exemplars

AI summary

Overview

Research area: Computer vision and multimodal document generation — specifically "visual coding," where a model must read a full scientific paper and emit renderable HTML/CSS that reproduces a conference poster.

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, layout detection (LayoutLMv3), OCR (PaddleOCR), retrieval-augmented generation, and reinforcement learning with verifiable rewards (GRPO).

Scope (one sentence): The paper introduces RAP, a 25,435-pair dataset of papers and human-crafted posters that are automatically "derendered" into editable, slot-typed HTML/CSS, plus a held-out 150-paper benchmark and two applications — a training-free retrieval pipeline (PosterRAG) and an SFT+RL post-trained model.

What This Paper Is About

Making a conference poster from a paper compresses an entire paper onto one page, exercising summarization, visual design, and code implementation at once. Existing benchmarks for this task are small (for example, Paper2Poster has 100 paper-poster pairs, P2PEval 121) and annotated only with layout boxes, so they must be scored by VLM-as-judge or human checklists rather than by objectively rendering the output and comparing it to the human poster. The authors build a large corpus in which every poster has been converted into renderable HTML/CSS with typed, refillable content slots, so quality can be measured by render-and-compare — and so that human-crafted designs can be reused both as a retrieval library and as training supervision.

Key Contributions

1. A large-scale dataset and benchmark. RAP contains 25,435 verified paper-poster pairs collected from NeurIPS (11,256), ICLR (8,504), and ICML (5,675) across 2022–2026 via OpenReview, annotated with a three-level taxonomy of 15 domains, 20 tasks, and 39 methods, plus a distilled introduction summary. A held-out benchmark of 150 test papers (10 per domain across the 15 domains) keeps its human posters for evaluation.

2. A fully automated derendering pipeline. Raw poster images are converted into styled, renderable HTML/CSS in which every element is an addressable, typed, and refillable slot — producing paper–poster–markup triplets rather than layout boxes alone.

3. PosterRAG, a training-free retrieval-augmented pipeline. A three-level retrieval cascade (categorical filtering, narrative matching, aesthetic scoring) selects a human-crafted template; a mapping agent refills its slots programmatically while leaving layout and styling untouched; a VLM reviewer iteratively repairs visual defects.

4. An SFT+RL recipe for a dedicated paper-to-poster model. Supervised fine-tuning on paper–markup pairs followed by rendering-aware reinforcement learning with a composite rule-based plus VLM-based reward, applied to Qwen3.8-27B.

Main Findings

• Derendering reconstruction is faithful: the full pipeline reaches SSIM 0.7896, DISTS 0.0663, class-aware layout mIoU 0.6897, class-agnostic mIoU 0.7246, FID 16.90, and FDiD 61.71 (backbone: Claude-Opus-4.6). The conclusion rounds these to SSIM 0.79 and layout mIoU 0.69.

• Layout detection dominates the pipeline: adding layout detection to the base pipeline raises mIoU from 0.0652 to 0.5236 and lowers FID/FDiD from 53.63/977.20 to 18.32/93.48; adding OCR then raises mIoU to 0.6808, and the design caption adds smaller consistent gains (final FID/FDiD 16.90/61.71).

• Backbone choice matters: Gemini-3.1-Pro leads on SSIM (0.8076) and DISTS (0.0623), while Claude-Opus-4.6 leads on layout accuracy (mIoU 0.6897) and distributional fidelity (FID 16.90, FDiD 61.71); GPT-4o (FID 41.00) and Qwen-3.5 (FID 28.86) trail.

• HTML guidance beats image guidance by a wide margin: PosterRAG with HTML Guidance reaches FID/FDiD 23.75/306.79 versus 77.11/1267.84 for image guidance, and DINOv2-CosDist 0.4443 versus 0.0877.

• PosterRAG leads all baselines on all five metrics: LPIPS 0.6471, DINOv2-CosDist 0.4443, DISTS 0.3490, FID 23.75, FDiD 306.79 — with FDiD 18.7% lower than P2P (377.25), the best baseline on that metric; Paper2Poster scores FID 81.59 / FDiD 1156.73 and PosterGen 133.83 / 682.27.

• Post-training improves over the base model on every metric: Qwen3.8-27B-SFT reaches LPIPS 0.6599, DINOv2-CosDist 0.2784, DISTS 0.2892, FID 28.00, FDiD 326.24, up from the base model's 0.6284 / 0.2579 / 0.2680 / 32.44 / 465.00.

• RL further improves rendering fidelity: Qwen3.8-27B-RL lowers FID/FDiD to 23.42/306.94 and raises LPIPS to 0.6925 and DISTS to 0.2944, with a small drop in DINOv2-CosDist to 0.2735; it holds the highest LPIPS and DISTS among compared models and the second-lowest FID/FDiD behind Claude-Opus-5 (21.56/289.49).

• Dataset composition skews toward theory: Optimization & Learning Theory is the largest domain at 3,487 papers (13.7%), while Fairness & Social Impact is smallest at 722 (2.8%); poster counts grow from 3,992 in 2022 to 6,982 in 2025, with 3,265 for 2026 (ICLR only at time of writing).

Methodology in Plain English

Collecting data. The authors pulled accepted papers and author-uploaded poster files from OpenReview for three top machine learning conferences, normalized the poster formats, and matched each poster to its source paper. Each paper was tagged with domain, task, and method labels, and its introduction was distilled into a short structured summary (motivation, problem formulation, key insight, claimed contributions).

Turning posters into editable code. A four-stage pipeline reconstructs each poster as HTML. First, a fine-tuned LayoutLMv3 model finds every visual element and sorts it as text-like, image-like, or decorative. Second, PaddleOCR reads the text in each text region, and font size is estimated as (bounding-box height / number of lines) × 0.8, a conservative factor for line spacing that can be used directly as CSS font-size. Third, a multimodal model writes a structured "design caption" describing the page's color palette, typography, and decorative motifs, plus per-region composition, reading flow, density, and emphasis. Fourth, an HTML scaffold is built with absolutely positioned divs: images and decorations are cropped and embedded, text blocks become empty slots pre-annotated with their measured font sizes, and an MLLM fills each slot with content read from the poster and writes the CSS decoration.

Generating a poster without training (PosterRAG). An input paper is parsed and labeled. A first retrieval step keeps only templates whose domain, task, and method labels match. A second step re-ranks them by comparing how the input paper's introduction frames its contribution against each template's source-paper summary (for example, "gap-and-fill" versus "observation-driven" narratives). A third step scores candidate posters on three 1–5 scales — content density, visualization ratio, and scientific aesthetics — picking the one closest to neutral unless the user states preferences. The chosen template's slots are then filled by an LLM that decides, for each slot, whether to replace text, replace an image, insert a pre-rendered asset, keep, or clear it, with content trimmed to the slot's estimated capacity. Crucially, filling happens programmatically — text slots get new innerHTML, image slots new src — so all original CSS, typography, and decoration are preserved. A reviewer agent then renders the HTML, screenshots it, and fixes visual defects iteratively.

Training a dedicated model. Paper text from the method, experiments, and conclusion sections is paired with the paper's figures, tables, and equations, matched to the reference poster's slots through visual feature matching and geometric verification. A natural-language user instruction (design description plus a layout specification that does not reveal exact poster text or HTML) is also constructed. The model is first supervised-fine-tuned on these triplets, then optimized with GRPO: multiple HTML candidates are sampled, rendered, and scored by a validity gate times a weighted sum of a rule-based reward (0.4) — checking text clipping, overlap, out-of-canvas elements, density, and asset usage from the rendered DOM — and a VLM-based reward (0.6) — checking quality and similarity to the reference poster.

Evaluation. Diversity is measured with pairwise LPIPS, DINOv2-CosDist, and DISTS across generated posters; human-likeness is measured with FID and FDiD (FID with DINOv2 features instead of InceptionV3) against the RAP-25K collection. All LLM-based agent calls in all compared methods use Claude-Opus4.6 for fairness.

Why This Matters

The paper reframes poster generation from "generate an image and hope a judge likes it" to a verifiable coding task: the output is HTML, and it can be rendered and compared against the human original. That makes measurement reproducible at the scale of the whole corpus rather than on a handful of curated examples, and it converts human designs from evaluation yardsticks into a reusable design library and a source of training supervision.

Real-world applications:

• Automating conference poster preparation for researchers, with layouts inherited from real accepted posters rather than invented from scratch.
• Generating slide decks, lab-meeting summaries, or grant one-pagers from long technical documents, since the same parse–retrieve–refill pattern applies to any document-to-fixed-page-format task.
• Producing editable, brand-consistent marketing or report collateral from structured content, where preserving a designed template matters more than novelty.
• Building training data and evaluation harnesses for document-understanding and code-generation models, because each sample carries a verified reference output.

Industry relevance: The derendering pipeline is a general technique for converting raster documents into editable, structured markup — directly useful to design tools, document-conversion services, and any product that needs to preserve visual fidelity while swapping content. The retrieval-then-refill approach also shows a practical alternative to fine-tuning: inherit professional design, adapt only the content.

Future Directions

• Extending coverage and freshness. The dataset currently spans 2022–2026 with 2026 containing ICLR data only; expanding to later venues and to non-ML fields would test whether the taxonomy and templates generalize.
• Reporting fuller evaluation of the reward design. The RL reward-component ablations are stated to be in Appendix B.6 and are not detailed in the main content, leaving open which reward terms drive the gains.
• Improving the weakest dimensions. RL raises LPIPS and DISTS but slightly lowers DINOv2-CosDist (0.2784 to 0.2735), and FID/FDiD still trail Claude-Opus-5 — questions remain about closing that gap and about what semantic diversity is being lost.
• Moving beyond layout-box-free but structure-limited templates. Because PosterRAG never alters layout or styling, it cannot invent designs for papers that no retrieved template fits; how to blend retrieval with controlled generation is left open.

Target Audience

Researchers in multimodal LLMs, document AI, and code-generation benchmarks; practitioners building document-to-format conversion or design-automation systems; and conference organizers or authors interested in automating poster preparation. Readers should be comfortable with retrieval-augmented generation and reinforcement learning from verifiable rewards; the dataset and demo lower the barrier for those who mainly want to use the resource rather than reproduce the methods.

Authors’ abstract

Making a conference poster is visual coding in miniature: an entire paper must be compressed onto a single page, which exercises summarization, visual design, and code implementation at once, using nothing beyond the paper itself. Yet benchmarks for this task in the context of visual coding remain scarce: existing resources are small and annotated only with layout boxes, so evaluation must fall back on VLM-as-judge or human checklists rather than objective render-and-compare, and the generation methods they support design layouts from scratch, leaving human-crafted posters unusable as either a retrievable design library or as verified supervision. We introduce RAP (Reconstruct A Poster), a dataset and benchmark for paper-to-poster generation as a visual coding task. RAP contains 25K+ verified paper-poster pairs from various machine learning conferences held between 2022 and 2026, annotated with a three-level content taxonomy, plus a held-out benchmark of 150 test papers each paired with its human poster. A fully automated derendering pipeline turns each poster into styled HTML/CSS of typed, refillable slots, so every sample is a paper-poster-markup triplet verifiable by rendering against the source poster. The corpus serves both inference and training: PosterRAG, a training-free multi-agent pipeline, retrieves an exemplar and refills its slots without touching layout or styling, while an SFT+RL recipe trains a dedicated model from the same triplets. An online demo is available at https://reconstruct-a-poster.github.io/.