PPTArena: A Benchmark for PowerPoint Editing

AI summary

Overview

Research area: Agentic multimodal AI and benchmark design for document editing, positioned at the intersection of computer vision, vision-language model (VLM) agents, and productivity software automation.

Technical level: Intermediate. The paper is readable without deep expertise, but assumes familiarity with vision-language models, benchmark evaluation protocols, and the notion of structured document formats (e.g., Office Open XML).

Scope: PPTArena is a benchmark of 100 real PowerPoint decks with over 1,300 human-curated editing tasks, paired with a structure-aware agent (PPTPilot) and a dual VLM-judge scoring protocol that separately measures instruction following and visual quality.

What This Paper Is About

Most PowerPoint tools and benchmarks either treat slides as flat images/PDFs, which discards the underlying structure of fonts, shapes, themes, and slide masters, or they focus on generating new slides from text rather than editing existing ones. The authors argue that real decks are almost always revised rather than built from scratch, so the practical question is whether today's multimodal agents can edit an existing deck precisely and attractively. PPTArena is their answer: a benchmark of in-place deck edits with element-level ground truth and style-aware rubrics, plus PPTPilot, an agent that plans edits over deck semantics and executes them with a hybrid of code and direct XML patching.

Key Contributions

1. PPTArena, a benchmark for agentic PowerPoint editing that operates on deck-native structure rather than rasterized slides. It contains 100 decks, 2,125 slides, and 1,300+ human-curated edits, organized into a taxonomy of five parent categories that decompose into 16 concrete edit types.

2. A dual-judge, rubric-grounded evaluation protocol. Two separate VLM judges score each edit from 0 (Failure) to 5 (Perfect): an Instruction Following (IF) judge that sees only structured data diffs, and a Visual Quality (VQ) judge that sees only rendered slide images. Each edit carries a per-sample style target generated with GPT-5 and then exhaustively human-verified.

3. PPTPilot, a structure-aware pilot agent that decomposes instructions into semantic operations, routes each query between programmatic tools (e.g., python-pptx) and deterministic XML patching, and runs an iterative plan-edit-verify loop.

4. A comprehensive empirical study across proprietary PPT agents, extended-thinking VLM configurations, an open-weight model, and PPTPilot ablations, including cross-judge robustness checks and blind human expert ratings.

Main Findings

• PPTArena is markedly harder than prior PowerPoint benchmarks. PPTArena averages 13.4 operations per edit and 8.3 slides per edit, with 32% of edits involving cross-slide dependencies and 28% requiring visual-textual reasoning. By comparison, PPTC-R averages 2.9 operations, 1.3 slides, 21% cross-slide, and no text-visual dependencies; T2US averages 1.2 operations, 1.2 slides, 5% cross-slide, and 1.3% text-visual dependencies.

• Task difficulty concentrates in the less visual categories. Structure and Interactivity edits require the longest programs (17.8-18.9 operations) across the most slides (10-11), producing the highest cross-slide rates (56-70%) and high-diff rates (71-75%). Content, Layout, and Styling cases still keep roughly a third of cases cross-slide and 41-52% high-diff.

• PPTPilot leads on the full benchmark under the GPT-5.2 judge. It reaches an instruction-following score of 2.57 and a visual-quality score of 2.69 across 100 decks and 1,340 edits, versus ChatGPT at 2.07/2.22 and Gemini CLI at 1.21/1.98.

• The leaderboard is stable under a different judge. Re-scored by an independent Gemini 3.1 Pro judge, PPTPilot still leads with IF 2.45 and VQ 2.74, ahead of Gemini CLI (1.92/2.15) and ChatGPT (1.97/2.03). On the 25-deck, 206-edit subset, PPTPilot scores 1.87/1.91 versus Gemini CLI 1.78/1.71, ChatGPT Agent 1.68/1.60, ChatGPT 1.12/1.56, MiniMax Agent 1.04/0.84, and Kimi-K2.6 1.68/1.00.

• One-shot generation-driven pipelines collapse. PPTAgent and Paper2Poster both fail on all tasks under the rubric, with 0.00 IF and 0.00 VQ.

• Both execution paths are necessary. Forcing a single route degrades PPTPilot: XML-only reaches 0.95 IF (it struggles with deck-wide operations) and python-pptx-only reaches 2.06 IF (it lacks fine-grained structural control). Hybrid routing recovers to 2.36 IF, and adding the iterative refine-and-verify loop lifts the system to 2.84 IF / 3.21 VQ, with most corrections landing by the second pass.

• The dual-judge protocol is what keeps scores calibrated. A single VLM judge over all signals is over-lenient (IF 2.31 / VQ 4.26, a 33% IF/VQ gap), and a dual judge without structured diffs inflates scores further (3.76/4.54). The dual judge with diffs yields 2.36/2.40.

• Automated judges track human perception. On a stratified 10% subset rated by 25 human experts, Pearson correlation is 0.72 (IF) and 0.81 (VQ); Spearman is 0.65 and 0.80; Kendall is 0.52 and 0.71. Running each judge five times per example gives majority agreement of 78% (IF) and 95% (VQ), with standard deviations of 15.4% and 9%.

• Human raters confirm the automated ranking, with wider gaps. Blind human scores give PPTPilot 3.86/3.81, ChatGPT Agent 3.17/2.82, and MiniMax Agent 1.75/1.85. PPTPilot leads ChatGPT Agent by 0.69 IF and 0.99 VQ under direct human preference.

• PPTPilot is fast in wall-clock terms. End-to-end latency per task is 1.5 minutes for a single pass and 3 minutes for a 3-iteration loop, versus 3 minutes for MiniMax Agent and 4-30 minutes for ChatGPT Agent.

• The router favors code. Across PPTArena, PPTPilot dispatches 66% of edits to the programmatic path and 34% to the direct XML path.

• Human alignment holds page-wise as well. An independent Gemini 3.1 Pro judge scoring 105 matched pairs one page at a time agrees within one point on 8 of 10 cases (mean VQ 1.88 page-wise vs. 2.00 batched, Pearson r = 0.917).

• No system solves the benchmark. All evaluated agents, including PPTPilot, still fail on hard, visually dependent, long-horizon tasks.

Methodology in Plain English

Building the benchmark. The authors web-scraped over 18,000 PowerPoint files from sources including SlidesCarnival, Zenodo, and SlideShare, and converted them into structured JSON traces capturing layout, styling, and content metadata. Automated filtering retained decks with diverse multimodal assets, and these were combined with a curated internal corpus contributed by literature analysts, biology researchers, and art and design students. From more than 500 hand-reviewed candidates, including 25 decks created from scratch by the authors, they selected 100 decks spanning professional, academic, multi-lingual, and art/design genres. Each edit bundles an initial deck, a fully specified target deck, structured textual instructions, and a rubric covering layout, typography, color, and content.

Designing the task taxonomy. Drawing on established presentation-design principles, they defined five parent categories decomposed into 16 concrete edit types: Content (text and typography, shapes, images, tables, charts, SmartArt, audio/video), Layout (alignment, distribution, grid, grouping, z-order, plus slide layouts and placeholders), Styling (themes, backgrounds, master-level edits), Interactivity (animations, transitions, hyperlinks), and Structure (slide/section/order management, slide numbers, headers/footers, notes, comments, accessibility). Many tasks straddle multiple buckets.

Scoring with rubrics rather than string or pixel comparisons. Correctness is not a simple diff. For each sample, GPT-5 generates a style target from the JSON summaries and screenshots of both the ground-truth and original decks, and every style target is then manually verified for correctness and faithfulness. Two separate VLM judges score the result: the IF judge sees only structured data diffs between original, predicted, and ground-truth slides, which forces it to concentrate on content-level correctness; the VQ judge sees only rendered screenshots of predicted and ground-truth slides and focuses purely on aesthetics. For multi-slide edits, SSIM screening forwards only slides with salient changes to the visual judge so it is not overwhelmed by context.

How PPTPilot works. The agent follows a dual-path design. A fast router (GPT-5 nano or Gemini-3.0 flash) inspects the prompt together with the presentation's structure, screenshots, and contents, and dispatches the task either to a programmatic path using python-pptx, which is effective for repetitive, well-defined operations like find-and-replace or translation, or to a direct XML path that reads, parses, and rewrites raw OOXML files for fine-grained control over positions, theme slots, and master-level changes. The edit calls themselves use GPT-5.2. After an edit is proposed, the XML is rendered to a temporary PPT file, a verifier model assesses the output against the original instructions, and feedback drives another pass in a plan-edit-verify loop.

Why This Matters

Impact on research. The paper shifts PowerPoint evaluation away from rasterized renderings and text-to-slide generation and toward deck-native structure with causal, element-level ground truth. It raises the complexity bar substantially compared with prior benchmarks (13.4 operations per edit versus 2.9 and 1.2), and its dual-judge, rubric-grounded protocol offers a template for evaluating open-ended agentic tasks where a single judge or a simple diff is unreliable. The reported 1,300+ tasks also exceed the scale of recent multimodal agent benchmarks the authors cite, such as OSWorld (369 tasks) and SWE-bench Multimodal (517 visual issues).

Real-world applications:

• Rebranding decks at scale, for example flipping theme color roles or swapping master styles consistently across every slide.
• Adapting existing decks for new audiences through translation while preserving charts, structure, and layouts.
• Accessibility remediation passes covering alt text, reading order, and contrast.
• Multi-step layout repair and bulk normalization, such as aligning logos to a grid, standardizing image dimensions, or moving content between slide text boxes and speaker notes.

Industry relevance. The benchmark explicitly measures how agents behave when modifying real corporate, academic, and creative decks rather than generating templates, which is the actual workflow in consulting, research, and design. The result that an open, simple architecture (PPTPilot) beats proprietary products like ChatGPT Agent and MiniMax Agent by more than 10 percentage points on compound, layout-sensitive, and cross-slide edits is directly relevant to teams building productivity agents. The finding that API-driven execution outperforms GUI approaches for fine-grained control also carries practical guidance for product design.

Future Directions

• Conversational refinement for under-specified intent, moving beyond a single natural-language instruction toward multi-turn clarification of what the user actually wants.
• Cross-application workflows, including live charts and document-to-deck synthesis, which extend beyond editing within a single PowerPoint file.
• Hyper-specialized domains beyond the current taxonomy of 16 edit types, which the authors acknowledge is not exhaustive.
• Robustness on edits coupling visual, spatial, and cross-slide reasoning. The paper reports that agents remain brittle specifically on these tasks, and that PPTPilot can over-edit master XML when a change should be slide-local, while GUI-based agents sometimes hold an edge on Structure edits because PowerPoint natively handles masters, footers, and section dividers.

Target Audience

Researchers and engineers working on multimodal agents, computer-use agents, and document or presentation automation will benefit most, particularly those building or evaluating systems that must modify existing files rather than generate new content. Benchmark designers and evaluation researchers will find the dual-judge, style-target protocol and the human-alignment study useful for open-ended tasks where diffs and pixel similarity fail. Product teams building productivity assistants for enterprise or academic workflows, and practitioners who need to understand what current vision-language agents can and cannot reliably do to real decks, are also a natural audience.

Authors’ abstract

We introduce PPTArena, a benchmark for PowerPoint editing that evaluates how agents modify real slides from natural-language instructions. Unlike benchmarks that rely on image-PDF renderings or text-to-slide generation, PPTArena features 100 decks with over 1,300 human-curated edits across 2,125 slides, spanning text, charts, animations, and professional master styles. Each edit pairs a ground-truth deck with a target rubric and is scored by two Vision-Language Model (VLM) judges: one rates instruction following from structural diffs, the other visual quality from slide images. On top of this benchmark, we present PPTPilot, a structure-aware agent that plans semantic edit sequences, routes between programmatic tools and deterministic XML operations, and verifies each result in an iterative plan-edit-check loop. PPTPilot outperforms strong VLM-based agents by more than 10 percentage points on compound, layout-sensitive, and cross-slide edits, with large gains in visual fidelity and deck-wide consistency. Despite this, all agents still struggle on long-horizon, document-scale tasks, underscoring how hard reliable PowerPoint editing remains. We publicly release our code at https://github.com/michaelofengend/PPTArena .