Generating Sketches in a Hierarchical Auto-Regressive Process for Flexible Sketch Drawing Manipulation at Stroke-Level

AI summary

Overview

• Research area: Computer Vision — controllable sketch generation and stroke-level sketch manipulation.
• Technical level: Advanced (requires familiarity with auto-regressive sequence models, LSTM encoders/decoders, mixture density networks and gMLP blocks).
• Scope: The paper proposes Sketch-HARP, a hierarchical auto-regressive generator that produces sketches stroke by stroke so that individual stroke embeddings can be edited at any point during drawing.

What This Paper Is About

Prior stroke-level sketch manipulation methods, notably SketchEdit (Li et al. 2024a), require all edited stroke embeddings to be collected and fed into the generator simultaneously before generation begins, which means the user cannot change anything while the sketch is being drawn. This paper introduces Sketch-HARP, which instead generates each stroke through three stages — predicting a stroke embedding, anchoring it on the canvas, and translating it into drawing actions — in an auto-regressive hierarchy, so stroke embeddings stay exposed and editable throughout the generation process.

Key Contributions

1. Sketch-HARP, a hierarchical sketch generating process that enables flexible sketch drawing manipulation at stroke-level rather than only instance-level.
2. An auto-regressive generator that recursively produces sketch strokes and their canvas positions by turns, so recently drawn strokes and their positions inform the prediction of the current one, placing appropriate strokes at suitable locations.
3. A learned sketch representation pipeline combining a bidirectional LSTM stroke encoder, a fully connected position encoder, a gMLP relationship encoder, and an LSTM sketch encoder, plus a CNN image decoder used as regularization.
4. Experimental validation on stroke replacement, stroke erasion, stroke expansion and in-generation editing, with human scoring showing Sketch-HARP outperforming 7 baselines (including SketchEdit) on stroke replacement and stroke erasion.

Main Findings

• Stroke encoding: Each stroke is encoded into a learned stroke embedding, and each starting position into a position embedding; both are 128-dimensional vectors. A gMLP relationship encoder adds spatial, contextual and semantic relationships between strokes, and the enriched embeddings are combined into a single sketch code of dimension 128.
• Three-stage stroke generation: Generating a stroke means (1) predicting a stroke embedding plus a stop marker, (2) sampling a 2D starting position from a predicted bi-variate Gaussian, and (3) translating the embedding into drawing actions via a mixture of bi-variate Gaussians and a pen-state categorical distribution.
• Automatic stroke count: A two-dimensional marker (with components summing to 1) is predicted together with each stroke embedding and signals whether the current stroke is the last one, so the number of strokes is determined by the model rather than fixed in advance.
• Flexible in-generation editing: Because stroke embeddings are exposed, the model supports erasing the most recently drawn stroke and inserting a chosen stroke before the current stroke is drawn; applying erasion and expansion in a queue realizes stroke replacement.
• Stroke replacement results: The paper reports that Sketch-HARP preserves source-stroke features even though no visual patterns are used in sketch code learning, and that generated sketches stay visually harmonious even when the source stroke and target sketch come from different categories — for example a generated angel whose two wings differ in shape but not in approximate size.
• Stroke erasion results: The model can skip drawing redundant strokes (for example drawing only a single leg on a sheep) while still anchoring strokes correctly, such as keeping bus wheels connected to the bus body and leaving only the leftmost candle on a cake.
• Stroke expansion results: Injected strokes receive their own starting position and drawing actions, and drawing resumes from the breakpoint using the injected stroke's embeddings; the paper highlights a case where hands of a clock are drawn on a face contour borrowed from a pig to produce a pig-head-shaped clock.
• Human perception scores: In a questionnaire of 10 stroke replacement and 10 stroke erasion questions with 51 volunteers, Sketch-HARP scored 2.09 ± 1.06 (replacement) and 2.30 ± 0.96 (erasion), the highest of all evaluated models. The next best, SketchEdit, scored 1.45 ± 1.27 and 1.04 ± 1.14 respectively.
• Lowest-scoring baselines: RPCL-pix2seq scored 0.21 ± 0.59 (replacement) and 0.38 ± 0.76 (erasion); DC-gra2seq scored 0.44 ± 0.90 and 0.24 ± 0.65; SketchHealer scored 0.36 ± 0.73 and 0.53 ± 0.94; SP-gra2seq scored 0.52 ± 0.79 and 0.40 ± 0.78; Lmser-pix2seq scored 0.64 ± 0.93 and 0.65 ± 1.02; RPCL-pix2seqH scored 0.29 ± 0.71 and 0.46 ± 0.76.
• Statistical significance: T-tests against SketchEdit produced p-values of 6.04 × 10⁻¹³ for stroke replacement and 1.23 × 10⁻⁵⁶ for stroke erasion, which the paper describes as a significant improvement.
• Limited error accumulation: The paper argues error accumulation is slight because each decoder generates short sequences — stroke and position decoders generate sequences whose length is the number of strokes per sketch, around 7 for QuickDraw sketches, while the sequence decoder's length is the number of drawing actions per stroke, around 10.
• Not reported in the provided content: The paper's section on sketch representation and reconstruction is truncated, so its quantitative results are not available here.

Methodology in Plain English

Sketches come from QuickDraw in vector form: a sketch is a sequence of drawing actions, each holding a 2D pen coordinate and a one-hot pen state (pen down, pen lifting, end-of-drawing). Because a stroke ends when the pen lifts, a sketch with K strokes splits cleanly into K groups of actions, and the first coordinate of each group is treated as that stroke's starting position on the canvas.

On the encoding side, a bidirectional LSTM turns each stroke into a stroke embedding, a fully connected layer turns each starting position into a position embedding, and a stack of gMLP blocks learns relationship embeddings that capture how strokes relate to one another spatially, in drawing order, and semantically. Each stroke embedding is enriched by adding its relationship embedding, and an LSTM sketch encoder compresses the whole sequence into one sketch code.

Generation runs in the opposite direction, in three nested stages. First, an LSTM stroke decoder takes the sketch code plus the previous stroke's enriched embedding and position, and predicts the next stroke embedding together with a stop marker. Second, an LSTM position decoder predicts the parameters of a bi-variate Gaussian from which the stroke's starting coordinate is sampled. Third, an LSTM sequence decoder (adopted from Ha and Eck 2018) translates that stroke embedding into drawing actions, modelling pen offsets with a mixture of Gaussians and pen state with a categorical distribution. Because each stroke embedding already carries relationship information, individual embeddings can be decoded independently, so multiple strokes can in principle be translated at once.

Training is by sketch reconstruction with five loss terms: a sequence reconstruction loss measuring pen-position and pen-state differences, a position loss anchoring predicted strokes to their true coordinates, a stop loss on the stroke-termination markers, a stroke-embedding loss (weight 5) that pushes generated embeddings toward their targets using a stop-gradient operation, and an image regularization loss (weight 0.5) from a CNN decoder that reconstructs a 128 × 128 × 1 sketch image from the sketch code.

Why This Matters

• Impact on research: It reframes stroke-level sketch manipulation as an interactive, anytime operation rather than a one-shot conditioning problem, and shows that an auto-regressive hierarchy over strokes and positions can keep editable representations exposed without breaking generation quality.
• Real-world applications:
  • Interactive sketch and design tools where users edit a stroke mid-drawing instead of restarting generation.
  • Digital illustration and animation workflows that need to swap, delete or inject strokes while preserving overall composition.
  • Creative mash-up or hybrid-sketch tools that borrow strokes from one drawing to complete or alter another.
  • Sketch-based prototyping interfaces where a partially drawn sketch is completed under the influence of a reference drawing.
• Industry relevance: The method works on vector stroke data rather than only raster images, which fits pen-input devices, tablet drawing apps and any product pipeline that stores drawings as stroke sequences. The released code at https://github.com/SCZang/Sketch-HARP lowers the barrier to reproducing and extending the approach.

Future Directions

• Quantitative representation and reconstruction comparison: The truncated section on sketch representation and reconstruction leaves open how Sketch-HARP compares numerically against RPCL-pix2seq, RPCL-pix2seqH, SketchHealer, Lmser-pix2seq, SP-gra2seq and DC-gra2seq on those measures.
• Extending beyond the evaluated data: The method is evaluated only on QuickDraw-derived DS1 (17 categories) and DS2 (5 categories), each with 70K training, 2.5K validation and 2.5K test sketches per category, so generalization to other sketch collections and to raster-image inputs remains untested.
• Broader and finer manipulation operators: The paper demonstrates replacement, erasion, expansion and in-generation erase/insert; more elaborate editing semantics — partial strokes, stroke deformation at sub-stroke granularity, or multi-stroke group edits — are not explored.
• Parallel decoding in practice: The paper notes that decoding multiple stroke embeddings simultaneously into drawing actions is possible because relationships are already baked into the embeddings; realizing and evaluating that parallelism is left as an open step.

Target Audience

Researchers and graduate students working on sketch generation, controllable generative models and human-computer interaction for drawing tools. The paper's mathematical formulation — mixture density outputs, relationship encoding, multi-term reconstruction losses — assumes comfort with sequence modelling, so practitioners building stroke-based creative applications will benefit most from the manipulation sections and the human evaluation, while newcomers to sketch generation may need background reading on QuickDraw vector representations and auto-regressive decoders.

Authors’ abstract

Generating sketches with specific patterns as expected, i.e., manipulating sketches in a controllable way, is a popular task. Recent studies control sketch features at stroke-level by editing values of stroke embeddings as conditions. However, in order to provide generator a global view about what a sketch is going to be drawn, all these edited conditions should be collected and fed into generator simultaneously before generation starts, i.e., no further manipulation is allowed during sketch generating process. In order to realize sketch drawing manipulation more flexibly, we propose a hierarchical auto-regressive sketch generating process. Instead of generating an entire sketch at once, each stroke in a sketch is generated in a three-staged hierarchy: 1) predicting a stroke embedding to represent which stroke is going to be drawn, and 2) anchoring the predicted stroke on the canvas, and 3) translating the embedding to a sequence of drawing actions to form the full sketch. Moreover, the stroke prediction, anchoring and translation are proceeded auto-regressively, i.e., both the recently generated strokes and their positions are considered to predict the current one, guiding model to produce an appropriate stroke at a suitable position to benefit the full sketch generation. It is flexible to manipulate stroke-level sketch drawing at any time during generation by adjusting the exposed editable stroke embeddings.