SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement

AI summary

Overview

Research area: Computer Vision (text-guided vector sketch generation, agentic LLM systems, diffusion-based vector graphics)

Technical level: Advanced — assumes familiarity with large language/multimodal models, differentiable vector rendering, Bézier curve representations, and diffusion distillation sampling.

Scope (one sentence): The paper introduces SketchDirector, an agentic framework that turns a scene-level text prompt into an editable multi-concept vector sketch by decomposing the scene into Bézier-curve object assets, arranging them through a closed-loop Plan–Execute–Verify–Correct reasoning process with self-evolving memory, and then refining details with diffusion priors — alongside a new dataset, MCSketch8K.

What This Paper Is About

Most sketch generation methods rely on image guidance and work at the level of single objects or simplified scenes, so text-guided complex scene sketching remains underexplored. Existing scene-level methods treat the whole sketch as one indivisible image or treat objects as passive, pre-defined components, which blocks layout planning, per-object editing and reuse.

The goal of SketchDirector is to mimic how human artists actually work — drafting individual elements, iteratively rearranging them, and then polishing detail — in order to produce scene sketches whose objects are independently editable and whose layouts are reasoned rather than baked in.

Key Contributions

1. An agentic scene-sketching framework. SketchDirector automatically generates text-guided complex scene sketches with flexible object-level control and structure-preserving detail refinement, representing every asset as a set of adjustable Bézier curves.

2. Closed-loop agentic layout reasoning. Scene composition is formulated as an iterative Plan–Execute–Verify–Correct process over a compact action library (move(), scale(), rotate(), layer(), delete(), modify()), with verification combining geometric checks and vision-language-model judgement, plus a hierarchical self-evolving memory (short-term failure memory and long-term experience memory).

3. Structure-preserving detail refinement. Curve contour resampling converts object outlines into a fixed number of cubic Bézier primitives, followed by progressive joint local–global optimization using Rectified-Flow Distillation Sampling (RFDS) to add detail without destroying layout or object geometry.

4. The MCSketch8K dataset. A multi-concept scene sketch dataset with 1,000 scene sketches, over 4,000 disentangled vector object assets across 800 object categories, annotated with compositional layouts and object-level manipulation operations.

Main Findings

• Best scores across all five metrics. Evaluated on 30 prompts sampled from MCSketch8K with a four-VLM judge panel (Qwen3.7-Plus, Claude-Opus-4-7, Gemini3.1-Pro, OpenAI GPT-5.5), SketchDirector reaches 0.916 Sem., 0.772 Compo., 0.821 Compl., 0.836 Aes. and 0.821 Size, versus 0.911 / 0.730 / 0.811 / 0.813 / 0.820 for the strongest baseline, OpenAI GPT-5.5, and 0.742 / 0.598 / 0.589 / 0.714 / 0.693 for SketchAgent.

• Refinement adds consistent gains. SketchDirector+Refine improves to 0.923 Sem., 0.788 Compo., 0.843 Compl., 0.869 Aes. and 0.842 Size, showing that detail refinement enhances quality while the paper reports the contour organization of the layout is preserved.

• Optimization-based baselines score far lower. VectorFusion records 0.466 / 0.252 / 0.338 / 0.417 / 0.412, DiffSketcher 0.567 / 0.374 / 0.381 / 0.567 / 0.537, and SVGDreamer 0.690 / 0.546 / 0.620 / 0.650 / 0.695.

• Reported runtime is shorter than most baselines. The paper lists approximately 6 minutes for SketchDirector and 21 minutes for SketchDirector+Refine, versus approximately 13 minutes (DiffSketcher), 17 minutes (VectorFusion), 24 minutes (SVGDreamer), and approximately 20 seconds for both SketchAgent and OpenAI GPT-5.5.

• Human preference agrees with the automated judges. In a user study with 68 participants across 15 prompts and 5,100 responses per criterion, participants consistently favored SketchDirector over both LLM-based and optimization-based baselines on object completeness, layout alignment and visual aesthetics.

• GPT-5.5 is the strongest backbone. Among four tested backbones, OpenAI GPT-5.5 gets the highest scores (0.926 Sem., 0.795 Comp., 0.811 Compl., 0.871 Aes., 0.785 Size), the fewest reasoning iterations (1.656) and the highest verification pass rate (0.803); Qwen3.7-Plus is weakest (0.629 / 0.477 / 0.526 / 0.552 / 0.507, 10.743 iterations, 0.436 verification).

• Every agentic component matters. Removing planning drops performance to 0.676 Sem. / 0.503 Comp. with 12.375 iterations and a 0.357 verification rate; removing geometric verification yields 0.698 / 0.550 / 0.570 / 0.660 / 0.589; removing semantic verification yields 0.603 / 0.512 / 0.645 / 0.558 / 0.547; removing short-term memory yields 0.737 / 0.658 / 0.629 / 0.654 / 0.681; removing long-term memory yields 0.781 / 0.679 / 0.697 / 0.762 / 0.729 — all below the full method's 0.839 / 0.765 / 0.732 / 0.768 / 0.742 with 1.375 iterations and 0.873 verification.

• Curve-based resampling beats point-based resampling. Deriving Bézier primitives directly from existing contours preserves shape and structural detail, whereas point-based sampling lets control points drift and distorts characteristic object geometry.

• Progressive optimization avoids conflicts. Dropping local supervision distorts individual objects, dropping global supervision weakens scene coherence, and optimizing both simultaneously from the start causes unstable updates and reduced object fidelity.

Methodology in Plain English

1. Break the prompt into objects. An LLM reads the scene description and lists the drawable objects, skipping abstract elements like mist that cannot be drawn as visible shapes. Each object gets an ID, a short description, and a relative depth order.

2. Draft each object as a vector. SketchAgent generates an ordered sequence of points for each object outline; a least-squares fit turns those points into cubic Bézier curves on a separate canvas, and a white background mask is added so objects can be assembled like collage pieces.

3. Plan the layout as a program, not a picture. The agent expresses placement decisions as a sequence of small executable operations from a six-command library, snapping objects to a 64×64 grid where each cell spans 12×12 pixels in a 780×780 rendered canvas. Because operations are explicit, the composition trajectory is traceable and can be debugged.

4. Check the result two ways. Geometric verification rejects objects that leave the canvas or whose scale factor falls outside [0.1, 1.5]. Visual-semantic verification asks a VLM to score the rendered canvas and an object collage from five perspectives (semantic alignment, composition, completeness, size, aesthetics) and combines them into one weighted score that must pass a threshold.

5. Learn from failures and successes. Failed attempts are logged in a short-term failure memory with the program, renderings, attributes, feedback and failure category, then discarded once corrected. Successful layouts go into a long-term experience memory indexed by object set and retrieved via Jaccard similarity over object names (the paper's example: {tree, squirrel, bird} vs. {tree, squirrel, monkey} gives 0.50).

6. Polish the details without moving anything. Object contours are resampled into exactly N_target cubic Bézier segments using de Casteljau subdivision with a largest-remainder allocation of segment budget, keeping the geometry intact. Control points are then optimized with RFDS supervision, first per object and then on the accumulated canvas, so text conditions expand progressively as more objects join the scene.

Why This Matters

The work reframes sketch generation as composition and program synthesis rather than one-shot image synthesis, which makes the output editable at the object level — you can move a tree without regenerating the forest. It also provides the first dataset (per the paper's framing) pairing vector assets with compositional layouts and manipulation annotations, giving future work structured supervision for decomposition, arrangement reasoning and controllable editing.

Real-world applications:

• Animation and storyboarding — quickly producing and revising object-level scene drafts where individual elements can be repositioned or reused across shots.
• Concept art and pre-production — generating multiple layout variations from one prompt before committing to a design direction.
• Interactive design tools — exposing the move()/scale()/layer() primitives as user-facing controls in vector graphic editors.
• Scalable vector asset creation — producing reusable, individually editable SVG-style objects rather than flat raster sketches.

Industry relevance: the framework's runtime profile and object-level editability map to tooling needs in design software, game asset pipelines and content creation platforms, where partial revision matters more than one-shot generation quality.

Future Directions

• Relational and interaction modeling. The paper reports failures on complex physical and semantic relations, such as interactions between a woman and a kettle, or a castle located on a hill.
• Instance-aware constraints. Refinement can hallucinate redundant objects — the paper's example is a duplicated sheep — so constraining object identity and count is an open problem.
• Cheaper verification and optimization. Multimodal verification and iterative optimization carry real computational cost, especially with multiple refinement rounds; more efficient strategies are explicitly left to future work.
• Broader backbone and generalization studies. The backbone ablation shows large gaps between models, raising the question of how to make the reasoning loop robust when the underlying model is weaker (Qwen3.7-Plus needed 10.743 iterations at a 0.436 verification rate).

Target Audience

Researchers and practitioners in computer vision and graphics working on vector sketch generation, text-to-SVG systems, and diffusion-based vector optimization; LLM-agent researchers interested in closed-loop visual planning with verification and memory; and designers or technical artists who want object-level, iteratively editable scene generation rather than one-shot image output. Readers without grounding in agentic LLM pipelines and differentiable rendering will find the methodology dense.

Authors’ abstract

Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the \textit{MCSketch8K} dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement. The project page is: \href{https://yiki77.github.io/SketchDirector/}{https://yiki77.github.io/SketchDirector/}.