SlidesGen-Bench: Evaluating Slides Generation via Computational and Quantitative Metrics

AI summary

Overview

Research area: Evaluation methodology for automated presentation (slide deck) generation — spanning natural language processing, multimodal (vision-language) systems, and computational aesthetics.

Technical level: Intermediate. Readers need general familiarity with LLM/VLM evaluation, but the paper's formulas and protocols are explained in accessible, self-contained terms.

Scope: SlidesGen-Bench is a structured benchmark that scores heterogeneous slide-generation systems on content fidelity, visual aesthetics, and editability using quantitative, verifiable pipelines and a 1.5k-example human-preference dataset.

What This Paper Is About

Automated slide generation now happens through very different architectures — template-filling PPTX tools, HTML/CSS code generators, and image-synthesis models — and existing evaluation methods struggle to compare them on a common scale. Reference-based metrics need ground-truth files that open-ended generation does not have, while LLM-as-a-Judge grading is prone to biases such as verbosity bias and reasoning failures. The authors build SlidesGen-Bench to produce comparable, quantitative, and human-validated scores across all three paradigms within one framework.

Key Contributions

1. A three-branch structured benchmark. SlidesGen-Bench evaluates content fidelity, visual aesthetics, and editability. Content and aesthetics are measured on rendered slide images so that template-based, code-driven, and image-centric systems become directly comparable; editability is measured by format-conditional structural parsing of the generated files.

2. Slides-Align1.5k, a human-preference dataset. A ground-truth testbed covering slides from nine mainstream generation systems across seven scenarios, annotated by CS Ph.D. students, art undergraduates, and office clerks.

3. Computational rather than monolithic metrics. Instead of a single LLM prompt, content is assessed via a structured VLM extraction plus a QuizBank question-answering pipeline, and aesthetics is grounded in cognitive-load and visual-ergonomics formulas rather than a subjective judgment call.

4. Reliability validation. Human-alignment studies plus robustness analyses — bootstrap confidence intervals, oracle-context QA, style-transfer validation, and a backend-sensitivity check — support the reported scores.

Main Findings

• Our metrics track human preference better than LLM judges. On Slides-Align1.5k, SlidesGen-Bench reaches an Average Spearman correlation of 0.71 with a Standard Deviation of 0.16 and an Average Identical ratio of 32.6%. LLM-as-Judge Rating scores 0.57 / 0.23 / 20.7, LLM-as-Judge Arena scores 0.52 / 0.27 / 17.3, and PPTAgent's PPT-Eval scores 0.53 / 0.26 / 17.8. Human annotators themselves reach 0.85 / 0.12 / 45.3. The LLM judges were powered by gemini-3-flash-preview with prompts mirroring the paper's aesthetic criteria.

• Content quality is high on average but uneven by topic. Zhipu is the top performer with an overall average of 88.29%, and it also achieves the highest accuracy on both High and Medium difficulty questions. Skywork-Banana leads on "Brand" (92.14) and "Report" (90.00) generation, and Kimi-Standard reaches 96.47% on "Personal" topics. Business topics are the primary bottleneck, with the text reporting the lowest average accuracy of 61.61% (the Business Plan column average in Table 1 is 57.93%).

• Content comparisons are statistically and configurationally stable. 95% bootstrap confidence intervals are typically ±2–5%, and the top-ranked system's lower bound remains at or above the runner-up's upper bound. An all-question comparison of VLM backends yields nearly identical accuracies (DeepSeek-v4-Pro 60.8% vs. MiniMax-M2.7 61.1%), and an oracle-context QA test in which the evaluator answers the same MCQs from ground-truth source evidence reaches 100%.

• A "Structural Barrier" separates visual polish from editability. Under the Presentation Editability Intelligence (PEI) framework, most systems (Gamma, Skywork, Kimi-Standard, Kimi-Smart, Zhipu) plateau at Level 2, producing slides as isolated canvases without a global master, a pattern the authors call "Structural Amnesia." Quark is the only system to breach the "Toy-to-Tool" threshold at L3. AutoPresent* achieves L4 by natively generating programmable chart objects, and PPTAgent* reaches L5 using raw templates as structural priors — though its inherited animations and transitions cannot be parametrically modified. NotebookLM is the only system evaluated at L0.

• The Visual-Editability matrix has an empty quadrant. Commercial systems dominate the left hemisphere (L0–L2), prioritizing visual polish over structural depth. The lower-right quadrant (Q4) holds Quark (L3), AutoPresent* (L4), and PPTAgent* (L5). The upper-right quadrant (Q1), where a system would excel at both visual quality and structural depth, remains vacant.

• Aesthetics ranking favors image-centric generation. Skywork-Banana leads the aggregate Aesthetics score at 27.28 versus 26.58 for the runner-up Kimi-Banana, and also leads Engagement at 8.30 versus 6.41. Systems such as Quark and AutoPresent* show flatter narrative pacing, yielding lower Rhythm scores.

• Aesthetic components are complementary. In ablation, Usability alone is the strongest heuristic (Average Spearman 0.574), ahead of Visual HRV (0.618 — actually the strongest single component), Engagement (0.224), and Harmony (0.312). Combining Harmony, Engagement, and Usability yields 0.667, while the full four-metric method reaches 0.710 with the lowest standard deviation (0.160) and the highest Identical ratio (32.6%). Metric parameters were optimized on an independent verification set of 1000 human-generated slide pairs.

• Content errors are dominated by omission, then by mismatch. Of 13,023 total samples, 2499 were incorrect (19.1%). Missing Content accounts for 66.3% of errors, and Content Value Mismatch 31.7%. Top-performing systems such as Zhipu and Kimi-Banana reduce omission but show higher rates of value mismatch (49–52%), shifting the challenge from recall to factual consistency.

• Evaluation robustness to style. Over the 1,435 Slides-Align1.5k aesthetics records, style buckets were standard corporate/mixed (66.8%), minimalist-light (14.8%), image-rich marketing (8.6%), diagrammatic/scientific (6.3%), data-heavy report (2.6%), and dark-mode minimalist (0.8%). No systematic dark-mode penalty appears (dark decks receive slightly better human mean ranks, 3.84 vs. 4.27), and removing any single style bucket changes the composite human-alignment score by less than 0.05. Boundary cases such as image-rich marketing decks show larger rank errors.

Methodology in Plain English

The core trick is to make unlike things comparable by projecting them into shared spaces. Every system's output is rendered into images so that content and aesthetics can be judged visually regardless of whether the underlying artifact was a PPTX file, HTML, or a generated picture. Separately, the generated files themselves are parsed structurally (Office Open XML schemas via python-pptx, DOM trees for HTML) to measure editability.

For content, the authors do not just ask an LLM "is this good?" They first build a reference QuizBank from the source documents using a three-phase multi-agent pipeline: a Forensic Analyst agent drafts key points by scanning 100k+ tokens of raw PDF context; a Critic phase runs cyclic hallucination checks, quote verification, and gap expansion; and an Exam Setter converts the refined evidence into a standardized probe of exactly 10 multiple-choice questions per document — 5 Concept and 5 Data. All questions were manually verified so they cannot be answered from general world knowledge alone. At test time, a VLM parses the slide images into Markdown, then answers the questions using only that slide-derived context. This decoupling of extraction from reasoning isolates whether a wrong answer came from a visual extraction failure (only 1.2% of errors) or a comprehension shortfall.

For aesthetics, four formula-based dimensions are computed. Harmony fits per-slide hue distributions to HSV hue templates and aggregates deck mean while penalizing cross-slide standard deviation. Engagement adapts the Hasler and Süsstrunk colorfulness metric using opponent channels, plus a pacing score based on the standard deviation of slide colorfulness. Usability measures figure-ground text contrast within regions detected by a layout-analysis model, mapping the 21:1 contrast ratio to 1.0 via logarithmic normalization. Visual Rhythm introduces "Visual Heart Rate Variability," combining per-slide clutter (Subband Entropy) with temporal variability (length-normalized RMSSD over decks with at least 5 slides

Authors’ abstract

The rapid evolution of Large Language Models (LLMs) has fostered diverse paradigms for automated slide generation, ranging from code-driven layouts to image-centric synthesis. However, evaluating these heterogeneous systems remains challenging, as existing protocols often struggle to provide comparable scores across architectures or rely on uncalibrated judgments. In this paper, we introduce SlidesGen-Bench, a benchmark designed to evaluate slide generation through a lens of three core principles: universality, quantification, and reliability. First, to establish a unified evaluation framework, we ground our analysis in the visual domain, treating terminal outputs as renderings to remain agnostic to the underlying generation method. Second, we propose a computational approach that quantitatively assesses slides across three distinct dimensions - Content, Aesthetics, and Editability - offering reproducible metrics where prior works relied on subjective or reference-dependent proxies. Finally, to ensure high correlation with human preference, we construct the Slides-Align1.5k dataset, a human preference aligned dataset covering slides from nine mainstream generation systems across seven scenarios. Our experiments demonstrate that SlidesGen-Bench achieves a higher degree of alignment with human judgment than existing evaluation pipelines. Our code and data are available at https://github.com/YunqiaoYang/SlidesGen-Bench.