SketchForge: Training-Free Scene-to-Vector Sketching via Decoupled Structure and Texture Optimization

AI summary

Overview

• Research area: Computer vision / computer graphics — stroke-based vector sketch synthesis and image vectorization (SVG generation), with a stated CCS classification in Computing methodologies (computer vision, computer graphics) and Applied computing (media arts).
• Technical level: Advanced. The method assumes familiarity with differentiable vector rasterization (DiffVG), CLIP/DINO-style feature spaces, DDIM inversion, and AdaIN latent-statistic alignment.
• Scope: A training-free, per-image optimization pipeline that converts one full-color raster illustration into an editable, resolution-independent SVG sketch by optimizing two decoupled sets of Bézier strokes (structure and texture) and compositing them with multiply blending.

What This Paper Is About

Turning a painted raster picture into a clean, editable vector sketch is hard: raster-space rendering pipelines cannot be edited at the stroke level, while methods that generate sketches "from scratch" struggle to stay faithful to an arbitrary input image. The authors argue that a further problem is objective entanglement — contour strokes need precise geometric alignment, whereas texture strokes need freedom to capture tonal appearance — and that optimizing everything toward one loss compromises both. SketchForge addresses this by splitting the strokes into a structure branch and a texture branch, optimizing each against its own target, and fusing the results into a single SVG.

Key Contributions

1. A training-free raster-to-vector sketching framework that produces editable, resolution-independent SVGs through per-instance test-time optimization of Bézier curves, requiring no additional training, fine-tuning, or pretrained diffusion backbone.
2. A decoupled structure–texture formulation that pairs a patch-based multi-scale contour objective for structural strokes with a global appearance-reconstruction objective for free-form texture strokes, composited via multiply-blend layering.
3. A contour- and attention-guided initialization strategy that seeds strokes at geometrically and semantically informative locations (contour warping plus DINO class-token attention) instead of uniform random placement, improving optimization over random initialization.
4. An empirical comparison against CLIPasso, CLIPascene, SwiftSketch, ControlSketch, and an adapted DiffSketcher on 64 Stable Diffusion–generated illustrations, reporting higher Aesthetic Score, DINO similarity, and Edge F1 than the baselines, plus ablations isolating initialization and layer fusion.

Main Findings

• Best overall on the reported metrics: SketchForge reaches an Aesthetic Score of 4.8407, DINO similarity of 0.4507, DreamSim distance of 0.5552, and Edge F1 of 0.9780 on 64 content images. DINO and DreamSim are computed against the content image I_cnt; Edge F1 against the edge map I_edge.
• Comparison to the closest baseline: The adapted DiffSketcher is the strongest competitor, with Aesthetic 4.8020, DINO 0.3760, DreamSim 0.5351, and Edge F1 0.8325 — a lower DreamSim distance than SketchForge's 0.5552 but lower scores on the other three metrics.
• Other baselines: CLIPasso (4.1690 / 0.2064 / 0.7256 / 0.5262), CLIPascene (4.3904 / 0.2022 / 0.7024 / 0.5797), SwiftSketch (3.9784 / 0.1566 / 0.8305 / 0.3139), and ControlSketch (4.1493 / 0.1470 / 0.7960 / 0.3299) for Aesthetic / DINO / DreamSim / Edge F1 respectively.
• Qualitative ordering: The authors report that ControlSketch and SwiftSketch produce sparse, abstract strokes, CLIPasso yields spatially disordered strokes, CLIPascene over-abstracts fine detail, and adapted DiffSketcher under-represents local texture, while SketchForge most faithfully preserves object boundaries and structural detail.
• Guided initialization helps structure optimization: Against I_edge, Edge F1 rises from 0.7731 (random) to 0.8841 (guided), SSIM from 0.1120 to 0.8068, MAE falls from 0.4384 to 0.2041, and Chamfer distance from 0.0006 to 0.0003.
• Guided initialization also helps texture optimization: Against I_tex, SSIM rises from 0.6718 to 0.7375, DINO from 0.6378 to 0.7328, Edge F1 from 0.7975 to 0.8067, while DreamSim falls from 0.2459 to 0.1744 and LPIPS from 0.4034 to 0.3539.
• Both layers are necessary: Texture-only reaches DINO 0.7328, DreamSim 0.1744, Edge F1 0.8067, Aesthetic 5.0715; structure-only reaches 0.5800, 0.2918, 0.8841, 4.8718; fusion gives the best DINO (0.7372), DreamSim (0.1712), and Aesthetic (5.3507) while keeping a strong Edge F1 of 0.8629.
• Not reported in the main comparison table: Although LPIPS, SSIM, MAE, and Chamfer distance are listed as evaluation metrics, Table 1 reports only Aesthetic, DINO, DreamSim, and Edge F1; LPIPS, SSIM, MAE, and Chamfer appear only in the ablation tables.

Methodology in Plain English

• Build two targets from one image. From the input picture, the system extracts a black-on-white line-art edge map (normalized in line width) as the structural target. Separately, a StyleID-inspired style-transfer module injects reference features into DDIM-inverted content representations, with AdaIN aligning latent statistics, to produce a pencil-sketch-like texture target that keeps the input's layout but adopts the tonal character of the edge map. No user input is needed because the normalized edges are the default reference.
• Place the strokes intelligently. Rather than scattering 700 strokes per branch at random, a fraction ρ is contour-guided (two-thirds placed directly on detected contours, one-third interpolated between adjacent placements, with smoothstep gap filling and dissimilarity below τ = 0.3 skipped). The remainder is attention-guided: a DINO class-token attention map and an edge-confidence map are power-transformed (γ_A = 0.5, γ_E = 0.7), averaged into a single probability map, and sampled to cover salient regions that strong edges do not explain. In the structure branch, strokes are warped to local contour geometry by projecting control-point residuals onto the contour normal with β = 0.3.
• Optimize two branches separately. The structure branch freezes width and color and moves only control points, using a 4×4 patch-grid loss that combines a fine term (stable gradients at sharp edge transitions) with a coarse term on 4×-average-pooled patches (to anchor low-frequency stroke mass where a patch has few foreground pixels). The texture branch frees control points, width, opacity, and color, and minimizes a full-canvas L2 distance to the texture target. Both branches use Adam with separate parameter groups; settings are 700 strokes per branch on a 1024×1024 canvas, 500 iterations, initial width 1.5 pixels (maximum 100), and learning rates of 1.0 for geometry, 0.01 for color, and 0.1 for width.
• Fuse the layers. The texture layer is stacked below the structure layer in the SVG and the structure layer is set to mix-blend-mode: multiply, so dark contours darken and reinforce local structure while white regions let texture show through.
• Evaluate. Baselines are compared on 64 diverse Stable Diffusion illustrations, spanning common daily objects to organic subjects such as animals and human figures, using perceptual (Aesthetic Score, LPIPS, DreamSim), semantic (DINO), structural (SSIM, MAE, Edge F1), and geometric (Chamfer distance on skeletonized stroke paths) measures.

Why This Matters

• For research: The paper reframes raster-to-sketch vectorization as a multi-objective problem, showing that separating sparse-edge geometry from dense-tone reconstruction is a productive decomposition. It also shows that a training-free, backbone-free pipeline built on DiffVG can beat CLIP-supervised and diffusion-prior sketch methods on the authors' benchmark, which is a useful counterpoint to the trend toward large pretrained generators.
• Real-world applications:
  • Vector design and illustration: producing editable, resolution-independent SVGs that designers can adjust stroke by stroke rather than raster layers.
  • Logo and branding adaptation: turning painted or photographic reference art into scalable line work that survives resizing.
  • Storyboarding and previsualization: quickly converting concept paintings into sketch-style frames.
  • Style transfer for content creation: the authors note supports for color-preserving vectorization and reference-guided style transfer, extending the tool beyond plain vectorization.
• Industry relevance: Output SVGs are editable at the stroke level and infinitely scalable, which fits print, web, and motion-graphics workflows where raster assets are a liability. The method's training-free nature lowers the barrier to deployment — no task-specific dataset or fine-tuning run is needed — while the 500-iteration per-instance optimization implies a per-image compute cost that matters for production throughput.

Future Directions

• More efficient optimization. The authors explicitly name efficiency as future work; each image currently costs 700 strokes × 2 branches × 500 Adam iterations.
• Finer user control. Also named by the authors: control over individual strokes and over semantic regions, which would turn the tool from one-shot conversion into an interactive editor.
• Closing the DreamSim gap. The adapted DiffSketcher achieves a lower DreamSim distance (0.5351) than SketchForge (0.5552), and the fusion ablation shows the texture layer's DreamSim (0.1744 against I_tex) is far better than the structure layer's (0.2918) — suggesting the compositing step is where overall appearance fidelity is lost.
• Broadening evaluation. The benchmark is 64 Stable Diffusion–generated illustrations; behavior on photographs, hand-drawn inputs, or line art as the source, and agreement with human preference judgments, are not reported.

Target Audience

Graduate students and researchers in computer graphics and vision working on stroke-based rendering, differentiable vector graphics, or sketch synthesis; graphics engineers building SVG-generation tools for design software; and technical artists or tool developers who need automated raster-to-vector conversion with editable output. Readers without background in differentiable rendering or CLIP/DINO feature spaces will find the method section demanding.

Authors’ abstract

Converting a raster image into a resolution-independent, stroke-editable vector sketch while faithfully preserving both structural contours and tonal texture remains challenging: raster-space rendering pipelines are difficult to edit at the stroke level, while generation-from-scratch vectorization methods struggle to remain faithful to an arbitrary input image. We present SketchForge, a training-free framework that converts an input raster image into a resolution-independent SVG sketch through per-instance test-time optimization of Bezier curves, requiring no additional training or fine-tuning. Strokes are seeded via a contour and attention-guided initialization, then refined by a vector-native differentiable rendering pipeline that decouples structural contour strokes from free-form texture strokes under dedicated objectives, composited via multiply-blend layering. Extensive experiments show that SketchForge consistently outperforms existing vector-based baselines in structural fidelity and generation quality. Project page: https://khangcntn123.github.io/SketchForge