Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation

AI summary

Overview

Research area: Computer Vision / multimodal generative modeling, specifically automatic graphic layout generation (document, slide, and poster design).

Technical level: Intermediate. The paper assumes familiarity with vision-language models, layout generation metrics (IoU), and vision transformers, but the core idea and results are explainable without deep technical background.

Scope: The paper introduces the "sketch-to-layout" task, in which a rough hand-drawn sketch plus the actual image and text assets are used to generate a complete, aesthetically coherent layout, and it provides a method, a synthetic training-data pipeline, and benchmark results on three public datasets.

What This Paper Is About

Automated layout generation systems can arrange text and images on a canvas, but when users want to control the result they usually have to specify tedious constraints: exact element sizes, precise positional relationships, grid guidelines, or long textual descriptions. This paper argues that a quick hand-drawn sketch is a far more natural and faster way for a person to express layout intent, and it builds a system that turns such a sketch, together with the real content assets, into a finished layout. The central challenge the authors tackle is that no large dataset of human-drawn sketch/layout pairs exists, so they invent a way to synthesize sketch training data at scale.

Key Contributions

1. Establishing sketch-to-layout as a research direction. The authors empirically demonstrate that sketches outperform existing guidance methods (asset types, types plus sizes, spatial relations, and textual descriptions of sketches) on the performance-versus-effort trade-off, making sketches the most efficient constraint modality tested.

2. A scalable synthetic sketch generation pipeline. Rather than paying annotators to sketch thousands of full layouts, the authors collect only 237 hand-drawn primitives for training (236 for validation) and compose them into complete synthetic sketches by matching each asset to similar primitives using attributes such as width, aspect ratio, and font size.

3. A released dataset of roughly 200,000 synthetic sketches. Publicly available sketches are released for PubLayNet (175k), DocLayNet (33k), and SlideVQA (27k), plus human-collected test sets, addressing a previously data-scarce problem.

4. A content-aware multimodal model and a new metric. A fine-tuned PaLIGemma 3B model takes the sketch, image assets, text assets, and a textual prompt, and outputs a structured layout representation. The authors also introduce the Content Ordering Score (COS), a Levenshtein-distance-based metric that measures whether the generated layout preserves the intended reading order and narrative flow.

Main Findings

• Sketches dominate the time-performance trade-off. In a few-shot study with Gemini 1.5 Pro, sketch-based guidance achieved the highest mIoU while requiring the least time to produce, beating asset-type, type-and-size, spatial-relation, and textual-description-of-sketch guidance.

• The fine-tuned model substantially outperforms constraint-based baselines. On PubLayNet, DocLayNet, and SlidesVQA, the content-aware FT-PaliGemma model reaches mIoU of 0.76, 0.59, and 0.75 respectively, versus 0.32, 0.25, and 0.46 for the strongest baselines — an improvement the abstract reports as more than 40% in Maximum IoU.

• Synthetic sketches are a valid proxy for human sketches. The model performs comparably on synthetic and human-drawn test sketches (for example, PubLayNet IoU 0.68 synthetic versus 0.62 human; SlidesVQA IoU 0.64 versus 0.66), showing negligible distribution shift and validating the synthetic training approach.

• Content-awareness improves results. Providing the actual image and text assets beats a sketch-only variant across coverage levels, most clearly on PubLayNet (mIoU 0.76 versus 0.67) and SlidesVQA (0.75 versus 0.71), with DocLayNet near parity (0.59 versus 0.60).

• Partial sketches degrade gracefully. Varying sketch coverage from 0% to 100% shows that higher coverage monotonically improves both IoU and COS, confirming the sketch acts as a genuine constraint rather than decoration.

• The model does not rely on obvious shortcuts, but has a known limitation. Replacing text with gibberish hurts IoU and COS, indicating the model does not simply use text length as a shortcut. However, replacing images with Gaussian noise does not consistently hurt performance, which the authors attribute to most layouts containing only a single image and to PaLIGemma not being pretrained on multiple unrelated images.

• Synthetic data collection is dramatically cheaper. Collecting all sketch primitives took 50 minutes total, whereas full human sketching would have required an estimated 2,336, 292, and 63 human hours for PubLayNet, DocLayNet, and SlideVQA.

Methodology in Plain English

The researchers treat layout generation as a code-generation problem. A layout is written as a protocol buffer string describing each asset's position and properties. This representation can be converted to SVG for rendering, makes it possible to verify that the model is not inventing assets that were never supplied, and keeps the output editable by existing design tools.

The model is PaLIGemma 3B, an open-source vision-language model whose language backbone, Gemma, was pretrained on code. During training, the vision encoder is frozen, and the model is shown: a hand-drawn ink sketch of the layout structure, the actual image assets, a textual prompt giving the canvas dimensions, asset names, and text content, and the target layout as a protocol buffer string. To prevent the model from learning a spurious mapping between input order and output position, the order of assets in both the prompt and the vision encoder is randomized.

The key methodological innovation is how training sketches are produced. Since gathering human sketches for every layout is prohibitively expensive, the authors collect a small library of drawing primitives: crossed-out rectangles for images and stacks of horizontal lines for text blocks, drawn by 10 annotators on tablet devices. To build a training example, each asset in a real layout is matched to its k nearest primitives based on standardized size and aspect ratio (for images) or bounding box width, height, and font size (for text), and one is selected at random. These matched primitives are rescaled and combined into a full synthetic sketch that mirrors the original layout's structure.

Evaluation uses IoU, maximum IoU (which finds the best possible pairing between generated and reference assets purely by position), alignment, overlap, and the new Content Ordering Score. Because the user explicitly guides the model through the sketch, IoU becomes an appropriate measure here even though it is weak for unconstrained layout generation. Baselines include LayoutPrompter variants under three constraint types and a sketch-guided few-shot Gemini, all upgraded to a stronger model than the original LayoutPrompter setup to make the comparison conservative.

Why This Matters

Impact on research. This work opens a previously unexplored problem formulation and, crucially, removes the data bottleneck that would otherwise block it. The released 200k-sketch dataset and the human-annotated test sets give the community a foundation to build on, and the synthetic-primitive-composition technique is transferable to any domain where constrained generation lacks paired training data. The Content Ordering Score also offers a way to evaluate whether a model understands content semantics rather than only geometry.

Real-world applications:

• Automated document and report design, where a user sketches a rough page structure and the system fills it with actual paragraphs and figures in a coherent reading order.
• Presentation slide creation, generating consistent decks from a sketched layout and a set of assets.
• Marketing and advertising asset production, letting non-designers outline a poster structure in seconds rather than specifying dimensions and alignment rules.
• Direct manipulation and editing interfaces inside existing document editors, since the structured output format is human-readable and editable.

Industry relevance. The approach substantially lowers the expertise needed to control generative design tools, replacing complex constraint languages with an interaction everyone already knows. Because the output is a structured, verifiable representation rather than pixels, it integrates with existing production pipelines and can be checked for hallucinated assets. The fact that a 3B open-source model outperforms few-shot prompting of a much larger proprietary model also matters commercially, since it implies deployable on-premise or on-device inference.

Future Directions

• Closing the multi-image gap. The model does not meaningfully benefit from image content, largely because most layouts contain a single image. Extending to layouts with several images would require training on richer multi-image data or a model pretrained to reason over multiple uncorrelated images.

• Scaling to larger and more capable models. The authors explicitly call for training larger models on this task to reach production-level quality, and for applying the pipeline to broader asset types and domains beyond documents, slides, and posters.

• Richer sketch primitives. More expressive primitives could convey additional design intent, such as styling hints, emphasis, or relationships that the current image-block and text-line vocabulary cannot express.

• Robustness to real design behavior. Partial-sketch results are promising, but how the system behaves with messy, stylized, or iterative human sketching, and how it should handle sketches that deliberately conflict with the provided assets, remains open.

Target Audience

This paper is most valuable to researchers and practitioners in generative design, document understanding, and multimodal learning — particularly those working on controllable generation, layout synthesis, or vision-language model fine-tuning. It is also useful for data-centric ML researchers interested in synthetic data pipelines that substitute for expensive human annotation, and for product engineers building design automation tools who need to understand both the capability and the current limitations, especially around multi-image content. Readers should have some familiarity with layout generation metrics and vision-language architectures to get full value from the experimental sections.

Authors’ abstract

Graphic layout generation is a growing research area focusing on generating aesthetically pleasing layouts ranging from poster designs to documents. While recent research has explored ways to incorporate user constraints to guide the layout generation, these constraints often require complex specifications which reduce usability. We introduce an innovative approach exploiting user-provided sketches as intuitive constraints and we demonstrate empirically the effectiveness of this new guidance method, establishing the sketch-to-layout problem as a promising research direction, which is currently under-explored. To tackle the sketch-to-layout problem, we propose a multimodal transformer-based solution using the sketch and the content assets as inputs to produce high quality layouts. Since collecting sketch training data from human annotators to train our model is very costly, we introduce a novel and efficient method to synthetically generate training sketches at scale. We train and evaluate our model on three publicly available datasets: PubLayNet, DocLayNet and SlidesVQA, demonstrating that it outperforms state-of-the-art constraint-based methods, while offering a more intuitive design experience. In order to facilitate future sketch-to-layout research, we release O(200k) synthetically-generated sketches for the public datasets above. The datasets are available at https://github.com/google-deepmind/sketch_to_layout.