From Dead Pixels to Editable Slides: Infographic Reconstruction into Native Google Slides via Vision-Language Region Understanding

AI summary

Overview

• Research area: Computer vision and multimodal document understanding, specifically visual-to-structured generation (derendering) applied to presentation authoring.
• Technical level: Intermediate. The paper combines a system description with light mathematics (coordinate mapping, a piecewise-linear font calibration) and a quantitative benchmark, but no new model architecture is trained.
• Scope in one sentence: The paper describes and evaluates Images2Slides, an API-based pipeline that turns a static infographic image into a native, editable Google Slides slide by extracting regions with a vision-language model and rebuilding them through the Google Slides API.

What This Paper Is About

Infographics communicate information through a mix of text, icons, and charts, but once exported as raster images their content is locked into pixels, which the authors call "dead pixels." Updating a label, translating text, or reusing a design then requires manual redesign work. The paper's goal is to recover editability: reconstruct the infographic as a Google Slides slide where text becomes real editable text objects and the original layout is preserved as closely as possible.

Key Contributions

1. An end-to-end, API-based reconstruction system. Images2Slides takes a PNG or JPG infographic, extracts regions with a vision-language model (VLM), maps pixel geometry into slide coordinates, and generates the slide through the Google Slides batch update API.
2. A model-agnostic region schema with deterministic postprocessing. A lightweight, typed JSON region schema plus validation and postprocessing lets multiple VLM backends be swapped behind a common interface, including Google Gemini, OpenAI GPT, Anthropic Claude, and OpenRouter-hosted models such as Qwen3-VL.
3. Practical reconstruction techniques. Deterministic element IDs (for example TXT_<id>, IMG_<id>) for retries, a piecewise-linear font size calibration to improve small-text readability, collision-aware text box width expansion, 10 px crop padding, and optional VLM-guided background synthesis.
4. A controlled quantitative evaluation. A round-trip benchmark of 29 programmatically generated infographic slides with known ground-truth region coordinates, reporting text accuracy (CER/WER), layout fidelity (IoU and center offset), and editability recovery rates.

Main Findings

• Element recovery is high overall: across the 29-run benchmark, overall element recovery rate is 0.989 ± 0.057, with text at 0.985 ± 0.083 and images at 1.000 ± 0.000.
• Text transcription error is low but variable: mean CER is 0.033 ± 0.149 and mean WER is 0.037 ± 0.167, with the larger character recovery rate reported as 0.969 ± 0.124 for text regions. The high variance indicates a minority of runs still incur noticeable recognition errors, typically on stylized or low-contrast text.
• Layout fidelity is stronger for images than for text: mean IoU is 0.364 ± 0.161 for text regions versus 0.644 ± 0.131 for image regions; median IoU is 0.388 ± 0.177 and 0.639 ± 0.130 respectively. The authors attribute the text gap to font metrics, line breaking, and text box width producing low IoU even when transcription is correct.
• Center displacement is much larger for text: mean center offset is 53.7 ± 26.9 px for text regions versus 11.0 ± 16.3 px for image regions.
• Threshold-based fidelity: Table 3 reports the fraction of IoU ≥ 0.5 as 0.376 ± 0.292 (text), 0.871 ± 0.332 (image), and 0.513 ± 0.237 (overall); the fraction of IoU ≥ 0.75 as 0.031 ± 0.092 (text), 0.181 ± 0.362 (image), and 0.071 ± 0.110 (overall). Section 8.3 separately states that, aggregated globally across all matched elements, 37.6% of text boxes and 85.1% of image boxes reach IoU ≥ 0.5, while 2.8% of text boxes and 15.8% of image boxes reach IoU ≥ 0.75.
• Runtime: mean VLM extraction time is 54.999 ± 18.337 seconds; mean Slides API time is 5.792 ± 0.682 seconds.
• Recurring failure modes: text over-segmentation (one paragraph split into several regions, raising false positives and lowering IoU), text under-segmentation (multiple logical blocks merged, harming editability), font metric mismatch, low-contrast or stylized text, and non-uniform backgrounds. Table 4 lists a mitigation for each, such as merging adjacent text boxes, OCR line clustering, font calibration with width expansion, upscaling or OCR fallback, and VLM-guided background sampling.
• The system does not fully vectorize: complex vector primitives such as custom shapes, gradients, and rich charts are not reconstructed as native slide objects; non-text regions are handled as cropped raster assets placed as image elements.

Methodology in Plain English

The authors avoid manual labeling by building a round-trip benchmark. For each run, they sample a grid or panel layout template, prompt a text VLM to produce an infographic concept and a region JSON specification that conforms to their schema, and generate the referenced images with a text-to-image model. The ground-truth slide is created using the same slide generation functions as Images2Slides, so region boxes correspond to known pixel coordinates in the exported PNG. That slide is rasterized into a 1600 × 900 infographic image and fed back into Images2Slides, which produces a predicted region file and a reconstructed editable slide. The pipeline itself works in six stages: VLM analysis into a raw JSON layout; validation and postprocessing into a clean typed layout object; asset preparation that crops image regions and uploads them to HTTPS-accessible URLs; geometry mapping that preserves aspect ratio, centers the infographic, and converts pixel boxes to slide points; construction of a single Slides API request batch; and execution via presentations.batchUpdate.

Because the VLM tends to underestimate font sizes, the system derives a base font size in points from the model estimate times the image-to-slide scale factor, then applies a piecewise-linear calibration, f' = f + max(0, (14 − f) × 0.294), which increases 5.5 pt small body text to 8 pt while leaving text at 14 pt and above unchanged. To reduce overflow from larger fonts, text box width can be expanded in proportion to the font scaling factor, capped at the slide boundary and at the left edge of any horizontally neighboring region whose vertical span overlaps. Image regions are cropped with 10 px padding on the right and bottom edges, stored under content-addressed names derived from a hash of the image bytes for deduplication, and placed as image elements. Non-uniform backgrounds are handled optionally with a --synthesize-background flag: a VLM returns a background_sample box with a mode label, and the pipeline either fills the slide with the patch's average color (solid) or tiles the patch (tile).

Why This Matters

Impact on research. The paper connects document-style visual understanding (layout analysis, document VQA, chart derendering, screenshot parsing) to a concrete authoring target: the Google Slides object model. It contributes a controlled round-trip benchmark design that sidesteps the difficulty of manually labeling ground-truth element geometry in raster-to-editable evaluations, and it surfaces engineering issues such as typography calibration and non-uniform backgrounds that are often left implicit in derendering work.

Real-world applications (as motivated or enabled by the paper's framing):

• Updating or correcting labels and numbers in an existing infographic without redesigning it.
• Localizing infographics into other languages by translating editable text objects rather than recreating artwork.
• Repurposing infographic designs for new audiences or channels while keeping the original layout.
• Building slide decks from existing image-based marketing or report graphics.

Industry relevance. Presentation automation, marketing asset management, and document-conversion products all face the same raster-content problem. The paper's model-agnostic schema and VLM backend-swapping design matter for deployment decisions, and the reported wall-clock times (about 55 s for VLM extraction per run versus about 5.8 s for Slides API calls) point to where latency actually lives. The operational constraint that the Slides API requires publicly accessible image URLs also matters for anyone automating this end to end.

Future Directions

• Chart-to-native regeneration: extending beyond raster image placement toward native chart objects, following chart derendering lines of work the paper cites.
• Improving region localization and ordering fidelity, since reconstruction quality is described as dependent on the region specification, and layout/structure learning methods are named as motivation.
• Logging input/output token counts from provider response metadata, where available, to enable post-hoc cost estimation, which the paper lists as future work.
• Re-running the backend-agnostic benchmarking harness with alternative VLM providers, since the reported numbers come from a single backend (Google Gemini) and serve as a concrete reference point rather than a comparison.
• Addressing the listed failure modes directly, including text over- and under-segmentation, font metric mismatch, low-contrast or stylized text, and non-uniform backgrounds.

Target Audience

Researchers and practitioners in computer vision, document layout analysis, and multimodal document understanding who are interested in visual-to-structured generation and derendering. It is also relevant to engineers building presentation automation, document conversion, or asset-management products, and to readers looking for a concrete example of how VLM region extraction can be wired into a production-style API pipeline with a quantitative benchmark behind it.

Authors’ abstract

Infographics are widely used to communicate information with a combination of text, icons, and data visualizations, but once exported as images their content is locked into pixels, making updates, localization, and reuse expensive. We describe \textsc{Images2Slides}, an API-based pipeline that converts a static infographic (PNG/JPG) into a native, editable Google Slides slide by extracting a region-level specification with a vision-language model (VLM), mapping pixel geometry into slide coordinates, and recreating elements using the Google Slides batch update API. The system is model-agnostic and supports multiple VLM backends via a common JSON region schema and deterministic postprocessing. On a controlled benchmark of 29 programmatically generated infographic slides with known ground-truth regions, \textsc{Images2Slides} achieves an overall element recovery rate of $0.989\pm0.057$ (text: $0.985\pm0.083$, images: $1.000\pm0.000$), with mean text transcription error $\mathrm{CER}=0.033\pm0.149$ and mean layout fidelity $\mathrm{IoU}=0.364\pm0.161$ for text regions and $0.644\pm0.131$ for image regions. We also highlight practical engineering challenges in reconstruction, including text size calibration and non-uniform backgrounds, and describe failure modes that guide future work.