Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs

AI summary

Overview

Research area: Computer vision and multimodal large language models applied to UI-to-code generation, specifically the underexplored domain of app widgets.

Technical level: Intermediate — readers should be comfortable with multimodal LLMs, front-end code generation, and standard vision metrics (SSIM, LPIPS, CLIP), though the paper's core ideas are explained clearly enough for a motivated beginner.

Scope: The paper formalizes the Widget-to-Code task, contributes the first image-only widget benchmark with fine-grained visual metrics, benchmarks existing MLLM and UI2Code methods, and proposes a modular baseline combining a perceptual decomposition agent with a DSL-based generation and rendering infrastructure.

What This Paper Is About

Automatically converting a visual user interface into executable code (UI2Code) works reasonably well for web pages and mobile screens, because those formats come with paired HTML/CSS or view-hierarchy data and have rich structural context. App widgets, however, are compact, context-free micro-interfaces built from dense iconography, embedded charts, and vivid styling under tight spatial constraints — and because widget source code is proprietary, no (image, code) training pairs exist. This paper defines the Widget2Code problem, builds the first image-only widget dataset and evaluation suite, shows that current methods fail on widgets, and proposes a baseline system that substantially improves visual fidelity.

Key Contributions

1. Task formalization and dataset. The authors define Widget2Code as a distinct visual code generation task and release the first image-only widget benchmark: 2,825 curated high-quality widgets (1,000 held out for testing) drawn from Figma, Dribbble, Refero, and manual device screenshots.

2. Fine-grained, visual-only evaluation metrics. Inspired by Apple's widget Human Interface Guidelines, they propose metrics across five dimensions — Layout, Legibility, Style, Perceptual, and Geometry — that assess rendered output rather than relying on ground-truth markup (which does not exist for widgets).

3. Systematic benchmarking. They evaluate both general-purpose MLLMs (GPT-4o, Gemini 2.5-Pro, Seed1.6-Thinking, Qwen3-VL) and specialized UI2Code methods (Design2Code, DCGen, LatCoder, UICopilot, ScreenCoder, UI-UG, WebSight-VLM-8B), showing that specialized methods degrade sharply on widgets while general MLLMs still produce unreliable, visually inconsistent code.

4. A modular baseline and infrastructure. They introduce a Perceptual Agent (component extraction, icon retrieval, reusable visualization templates, color extraction) and WidgetFactory, an end-to-end system comprising a widget-specific domain-specific language (WidgetDSL), a multi-target compiler (React, HTML/CSS/JS), and an adaptive rendering module that eliminates overflow while preserving aspect ratio.

Main Findings

• Specialized UI2Code models fail badly on widgets. Methods tuned for web or mobile layouts scored far below general-purpose MLLMs across nearly every metric — for example, ScreenCoder scored 25.35 on Contrast and 0.101 on SSIM, and Design2Code scored 17.50 on Text Jaccard and 49.68 on Area ratio.

• General MLLMs are better but still unreliable. Gemini 2.5-Pro led the general group on several metrics (Content 62.74, Area 79.66, SSIM 0.701, CLIP 0.844), yet all models produced content overflow, structural misalignment, missing icons, distorted graphs, and color drift.

• No existing method reproduces widget dimensions. Even when explicitly prompted to match the input size, all baselines failed the Geometry metric. The proposed baseline achieves a perfect 100.00 by construction.

• The proposed baseline outperforms everything on fidelity metrics. It leads on Margin (72.15), Content (66.08), Area (82.24), Text (70.60), Contrast (66.20), LocCon (64.06), Palette (58.09), Vibrancy (51.38), Polarity (63.28), and SSIM (0.721).

• Every module contributes measurably. Ablations from a Qwen3-VL base show incremental gains: WidgetFactory lifts Area from 69.53 to 82.46 and CLIP to 100; component templates raise Text to 67.99; color analysis boosts Palette from 42.61 to 57.56 and Vibrancy to 50.71; layout grounding improves Margin to 71.49; and icon retrieval (the full model) pushes Text to 70.60 and LocCon to 64.06.

• Icon generation is unreliable, so retrieval replaces it. Directly prompting MLLMs to draw icons causes semantic hallucinations and failures on thin-stroke or abstract glyphs; a two-stage retrieval over 50,000 SVG icons (visual coarse filter via SigLIP embeddings, then caption-based re-ranking to top-5) preserves both semantics and appearance.

• Verbose code is a root cause of failure. The authors attribute many geometry errors to MLLMs emitting long, unstructured code and ignoring the rendering stage, which motivates the compact DSL intermediate representation.

Methodology in Plain English

The pipeline has three stages.

Data curation. Since no paired widget data exists, the authors crawl design platforms and take device screenshots, then run a multi-stage image-processing pipeline. They convert images to grayscale, apply bilateral filtering to preserve sharp widget edges while blurring stylized backgrounds, use CLAHE to cope with low-contrast boundaries, and binarize with adaptive Gaussian thresholding. Because no single edge detector handles all widget presentations, they run nine detectors in parallel — multi-scale Canny at three blur levels, Laplacian, Sobel, morphological gradient, and Canny on the HSV saturation and value channels to catch isoluminant edges — and union their outputs with a bitwise OR to maximize recall. Template matching against rounded-rectangle geometries then filters out non-widget contours such as text blocks. After deduplication using CLIP features and manual review, 2,825 clean widgets remain.

Perceptual Agent. Rather than asking a model to go straight from pixels to code, the system first decomposes the input. An MLLM detects and categorizes each visual element (icons, buttons, checkboxes, text, charts) and returns a cropped region, bounding box, textual description, and category. Icons are not generated — they are retrieved from a 50k SVG library by combined visual and caption similarity. Non-icon components (buttons, text blocks, chart types) are matched to reusable DSL templates, which the MLLM then populates with the input's style and data. A color module converts the image to a perceptually uniform space, filters transparent pixels, and runs K-means to extract the dominant palette with proportions, which then constrains generation.

WidgetFactory. The model does not emit HTML directly. Instead it writes WidgetDSL, a compact, human-readable language representing a widget as a tree of parameterized components with geometry, color, and style attributes. DSL generation is constraint-guided prompt chaining: the base grammar prompt is progressively enriched with bounding boxes, the extracted color palette, component specifications, icon candidates, and inferred component types. A compiler then deterministically translates DSL through an abstract syntax tree into HTML/CSS/JavaScript or React, guaranteeing reproducibility. Finally, an adaptive rendering module computes the target aspect ratio and binary-searches over widths, using browser feedback about content overflow to find the smallest feasible layout, resizing afterward to exactly match the original dimensions.

Why This Matters

Impact on research. The paper opens a new sub-area of UI2Code and provides the missing infrastructure — data, metrics, and a reproducible baseline — that the field needs. It also demonstrates a useful general lesson: for domains where source code is proprietary and unpaired, retrieval-augmented components plus a constrained intermediate language outperform end-to-end generation from a strong general MLLM.

Real-world applications:

• Widget and lock-screen design tools, letting designers convert mockups into platform-ready components without hand-coding each one.
• Rapid prototyping in Figma-style workflows, cutting the manual design-to-front-end translation step that is currently slow and error-prone.
• Cross-platform migration, since the same DSL compiles to React, HTML/CSS, and other targets from a single source representation.
• Synthetic data generation, as the authors note WidgetFactory can act as a data engine to produce training pairs that do not otherwise exist.

Industry relevance. Design-system teams, front-end platform groups, and low-code/no-code vendors all face the widget-reconstruction problem. The DSL-plus-compiler architecture is directly deployable engineering rather than just a research prototype, and the deterministic compilation guarantees reproducibility that raw MLLM code output cannot provide.

Future Directions

• Stronger fine-grained visual grounding. Dense layouts and micro-visualizations (sparklines, pie charts, small bar plots) remain the hardest elements; better detection and structural parsing of these is the clearest next step.

• Better intermediate representations. The authors explicitly call for improved IRs to further reduce hallucination in chart data and styling, suggesting WidgetDSL is a starting point rather than an endpoint.

• Interaction and dynamic behavior. Extending beyond static appearance to interactive or state-dependent widget behaviors is noted as future work, and would require the DSL to express events and runtime logic.

• Scaling and generalization. With only 1,000 test widgets and a closed icon library of 50k SVGs, open questions remain about how the retrieval and template strategies scale to long-tail iconography and unconstrained widget designs.

Target Audience

This paper is most valuable to researchers working on multimodal code generation, UI2Code, and design automation, as well as to practitioners building design tools or low-code platforms who need to convert visual designs into front-end code. It also serves as a useful case study for anyone interested in how to structure MLLM pipelines around retrieval, domain-specific languages, and rendering feedback when direct generation is unreliable.

Authors’ abstract

User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and reusable visualization modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML/CSS). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research.