VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT

AI summary

Overview

Research area: Computer vision and multimodal document understanding — specifically, evaluating vision-language models (VLMs) as automatic critics for PowerPoint presentation slides.

Technical level: Intermediate. The framing is accessible, but the paper introduces a dense set of evaluation metrics (Hungarian matching, POA, MACE, Kendall's tau), so some familiarity with model benchmarking helps.

Scope: The paper introduces VLM-SlideEval, a slide-native evaluation framework that probes five VLMs on element-level extraction, sensitivity to controlled geometry/text/style perturbations, and deck-level narrative reordering.

What This Paper Is About

Presentation slides carry structured information across education, science, and business, and VLMs are increasingly used as automated judges of slide quality inside agentic generation pipelines. It is unclear, however, how well these models actually read slides — whether they can locate elements precisely from pixels, whether their quality judgments stay stable when a slide is degraded, and whether they understand how slides in a deck relate to one another. The paper builds a verifiable ground-truth pipeline and a perturbation suite to measure exactly those three abilities.

Key Contributions

1. A curated dataset with verifiable ground truth. The authors sample 100 English-dominant (at least 70 percent by langid) .pptx decks from Zenodo10K, excluding legacy .ppt, totaling 1,948 slides under a CC-BY 4.0 license. Ground-truth element geometry, content, and style are extracted from PowerPoint XML plus a post-layout COM (Component Object Model) API query, producing a unified schema with a fixed 960×540 px coordinate frame and characters stored in normalized #RRGGBB form.

2. A matching-based extraction protocol. Element predictions are aligned to ground truth with Hungarian matching using a blended cost combining 1−IoU, normalized center distance, relative size drift, and content distance, followed by an acceptance threshold gate. Each slide is run with N=3 repetitions at low temperature, and metrics are reported per-run and pooled.

3. A controlled perturbation suite. From the same 100 decks, 234 slides with at least 3 visible text elements were manually selected as seeds and degraded along three axes — geometry, text, and style — parameterized by a single severity value s in {0, 0.1, …, 1.0}, generating 7,722 original-plus-perturbed slides in total, with up to 50 slides subsampled per severity per axis for evaluation.

4. A deck-level narrative probe. VLMs are asked to reorder shuffled slides within a deck, scored with Kendall's tau, Spearman's rho, and Exact Match, extending evaluation from single-slide parsing to cross-slide comprehension.

Main Findings

• Parseability collapses with slide complexity for older models. GPT-4.1 succeeds on roughly 93 percent of simple slides with 8 or fewer elements, dropping to 72.1 percent for (8–16], 32.8 percent for (16–32], and 18.2 percent for slides with 32 or more elements. GPT-4o follows the same trend but declines earlier: about 88.0 percent for 8 or fewer, 57.6 percent for (8–16], 45.8 percent for (16–32], then a small noisy uptick to 66.7 percent at 32 or more elements (N=66). o3 and the GPT-5 variants remain effectively at ceiling across all bins (99.5 percent and above).

• Newer models lead on element extraction, but pixel-accurate style remains weak. o3 achieves the highest end-to-end Matching F1 (0.72), followed by GPT-5 (0.71–0.72), GPT-4.1 (0.59), and GPT-4o (0.44); parsed-only to end-to-end performance drops by roughly Δ ≈ 0.12 for both GPT-4.1 and GPT-4o. Text Content F1 is best for o3 at 0.78, then GPT-5 at 0.76, then GPT-4.1/GPT-4o at 0.69/0.63. On geometry (1−IoU, lower is better), o3 is best at 0.55, GPT-5 at 0.56, GPT-4.1 at 0.57, and GPT-4o worst at 0.65. Font Family Accuracy is low across the board (0.17–0.42), with GPT-5-high highest at 0.42 and GPT-4o lowest at 0.17.

• Coverage differs sharply between models. End-to-end coverage is limited, especially for GPT-4o (0.33) and GPT-4.1 (0.54) versus the remaining models (0.74–0.78).

• The two evaluation scales are largely monotone reparameterizations. An isotonic link maps 5-point scores to 100-point scores with R² in [0.85, 0.89] across models (p=0.001) on the normalized degradation scale y* in [0,1]; GPT-4.1 is tightest at RMSE = 0.075, with GPT-5-high close at 0.083.

• A consistency-fidelity trade-off appears for text but not for geometry or style. Moving from the 5-point to the 100-point scale yields no material fidelity gain for geometry and style — bootstrap confidence intervals overlap — but reduces consistency (POA_adj falls from [0.87, 0.95] to [0.62, 0.73] for geometry and from [0.88, 0.98] to [0.63, 0.81] for style). For text, the finer scale substantially raises fidelity (GPT-5-high from 0.51 to 0.75; GPT-5-minimal from 0.52 to 0.76) while lowering POA_adj (1.00 to [0.88, 0.92]).

• Models diverge most on text quality. Even the most convergent text pair (GPT-5-high vs. GPT-5-minimal) reaches only ρ̄ ≈ 0.55, whereas geometry and style pairs frequently exceed 0.80–0.90. The most divergent geometry pair (GPT-4o vs. o3) still agrees more (ρ̄ ≈ 0.78) than the most convergent text pair.

• Perturbation severity behaved as designed. The manipulation check showed high adjacent POA (about 0.95 on the 5-point scale, about 0.80 on the 100-point scale) with moderate calibration to the identity y* = s (overall MACE ≈ 0.34).

• Narrative ordering is barely above chance. Across models, Kendall's tau falls in [0.04, 0.12], Spearman's rho in [0.05, 0.13], and Exact Match in [0.10, 0.17]. GPT-4.1 delivered the strongest performance, a [0.04, 0.07] point improvement over GPT-5-minimal, but all remain far below the theoretical upper bound of 1.0.

Methodology in Plain English

The authors start by collecting 100 real PowerPoint decks and building a trustworthy answer key. They read the underlying PowerPoint XML for each slide, then query the PowerPoint COM API after a layout pass so that font metrics and tight text bounds reflect what a viewer would actually see rather than what the container says. Everything lands in one schema: positions and sizes in pixels on a fixed 960×540 canvas, font sizes and stroke widths in points, colors as normalized hex, and text alignment as one of left, center, right, justify, or distributed.

Each slide is then rasterized to a PNG and handed to a VLM with a prompt that fixes the coordinate system and enumerates the required JSON output fields (size, background, texts, rects, lines, images, tables). The API's JSON schema mode plus Pydantic validation enforces structure; anything that fails validation is counted as a parse failure rather than scored as a bad answer. To compare predictions against the answer key, the authors treat it as an assignment problem: they build a cost matrix where each candidate pairing costs a weighted combination of box overlap error, center distance, size drift, and text dissimilarity, then solve it with the Hungarian algorithm and discard matches that are too costly.

For robustness testing, they take 234 slides that the schema preserves well and degrade them along three axes using one severity knob from 0 to 1. Geometry perturbations include translation drawn from a Gaussian whose spread grows with severity, log-normal scaling, occasional extreme resizing, random repositioning, and collapse to a skinny shape. Text perturbations inject character-level noise (substitution, deletion, insertion, and adjacent swaps with weights 0.50, 0.20, 0.15, 0.15), optionally drop whole text boxes, or insert irrelevant ones. Style perturbations swap font families, jitter sizes, toggle bold/italic/underline, and inject or shift colors. They verify that higher severity really does mean worse slides by checking that scores move monotonically and calibrate against the identity line.

Finally, they ask models to rate perturbed slides for "text quality", "layout geometry", and "style" on either a 1–5 or a 1–100 scale, and separately ask them to restore the original order of a shuffled deck. All comparisons use seeded randomness and bootstrap 95 percent confidence intervals where appropriate.

Why This Matters

The paper argues that VLMs are already being used as critics in agentic slide-generation pipelines — to select candidates, drive iterative refinement, and supply reward signals — so their reliability as slide judges has direct practical consequences. The results suggest current models are good enough for coarse single-slide content understanding but not for pixel-accurate style judgments or cross-slide narrative reasoning, and that a coarse rating scale can be more stable than a fine one for geometry and style.

Real-world applications:

• Automated quality gates in slide-generation tools, where an evaluator decides whether a generated deck is good enough to ship.
• Accessibility and structure auditing of existing decks, recovering which elements play which roles (title, subtitle, body).
• Iterative refinement loops in which a model revises a slide and a separate critic estimates whether the revision improved it.
• Document and UI parsing pipelines that need element-level layout understanding from rasterized images rather than source files.

Industry relevance: The work comes from PowerPoint AI at Microsoft, and its framing is explicitly about evaluating the critics that sit inside model-forward production pipelines. The finding that text quality is the axis on which models disagree most — with even the most convergent pair reaching only ρ̄ ≈ 0.55 — is a direct caution for anyone using a single VLM as a text-quality judge. The fidelity-versus-consistency results also give concrete guidance: use a coarse scale for geometry and style, but expect finer text scales to buy sensitivity at the cost of internal stability.

Future Directions

• Broaden the corpus. The authors list the reliance on public PPTX files as a limitation; a wider, more diverse set of decks would test whether the observed trends hold.
• Richer narrative probes. Deck reordering produced Kendall's tau in [0.04, 0.12], and the authors call for stronger narrative evaluations beyond shuffled-slide recovery.
• A fuller parsing schema. The simplified schema used for slide parsing is named as a limitation, implying more expressive schemas could capture more slide structure.
• Stronger verifiable checks and judge calibration. The paper calls for calibrated, critic-in-the-loop evaluators with verifiable signals to gate selection and steer refinement, and lists judge calibration among the open directions.

Target Audience

This paper is most useful to researchers and engineers building or evaluating multimodal document-understanding systems, particularly those working on slide generation, agentic content pipelines, or VLM-as-judge setups. It also serves benchmark designers who need a template for coupling pixel-level ground truth to controlled perturbations, and product teams deciding how much to trust a VLM's rating of visual content. Readers looking for a leaderboard of absolute model quality will find the comparative results, but the paper's distinctive value is in its methodology for measuring robustness and consistency rather than raw accuracy alone.

Authors’ abstract

Vision-language models (VLMs) are increasingly used to evaluate multimodal content, including presentation slides, yet their slide-specific understanding remains underexplored {despite their growing role as critics in agentic, model-forward pipelines}. We introduce VLM-SlideEval, an evaluation framework that probes VLMs along three axes: (1) element-level extraction from slide images aligned to ground truth; (2) robustness to controlled perturbations in geometry, style, and text; and (3) higher-level comprehension, such as recovering a deck's narrative order from shuffled slides. Using publicly available decks from Zenodo (https://huggingface.co/datasets/Forceless/Zenodo10K/viewer/default/pptx), we standardize ground-truth element metadata from PowerPoint XML and live renderings into a unified, verifiable schema. Empirically, VLMs underperform on pixel-accurate extraction and show non-trivial agreement, fidelity, and consistency under controlled perturbations, while performing better on single-slide content understanding; however, they do not reliably capture narrative structure across slides. These results highlight the limits of current VLMs for slide evaluation and motivate calibrated, critic-in-the-loop evaluators that drive iterative refinement and selection in agentic pipelines.