Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching

AI summary

Overview

Research area: Computer vision and computer graphics, specifically generative vector sketch synthesis, score distillation sampling (SDS) with diffusion priors, and computational visual illusions.

Technical level: Advanced. The paper assumes familiarity with diffusion models, Score Distillation Sampling, differentiable rasterization, Bézier curve parameterization, CLIP-based metrics, and vision-language-model evaluation.

Scope: The paper introduces "Progressive Semantic Illusions," a vector sketching task in which a single sketch is turned into a different object purely by adding strokes over time, and presents a joint-optimization framework ("Stroke of Surprise") to generate such sketches.

What This Paper Is About

Traditional visual illusions work in space — you tilt, flip, or rotate an image to reveal a hidden second picture. This paper moves the trick into time: a sketch is shown as "a pig," and then more strokes are added until the same drawing reads as "an angel." The goal is to generate vector sketches that are coherent as the first concept with only the prefix strokes present, and coherent as a completely different concept once additional ("delta") strokes are drawn on top.

Key Contributions

1. A new task — Progressive Semantic Illusion. The authors extend visual illusions from the spatial to the temporal dimension, requiring a single vector sketch to reveal distinct semantic interpretations through progressive stroke accumulation. They frame this as a "dual-constraint": prefix strokes must depict object A while simultaneously serving as the structural foundation for object B.

2. A sequence-aware joint optimization method. Rather than freezing the prefix (as greedy sequential methods do), the framework optimizes prefix and delta stroke parameters together using a dual-branch Score Distillation Sampling (SDS) mechanism, dynamically discovering a "common structural subspace" valid for both interpretations.

3. An Overlay Loss for spatial complementarity. A geometric penalty on the normalized inner product of Gaussian-blurred prefix and delta renderings discourages delta strokes from simply occluding the prefix, enforcing structural integration instead.

4. A VLM-based filtering and ranking pipeline, plus scalability demonstrations. GPT-4o scores candidates across four dimensions, and metric- or GPT-based ranking selects the best results. The framework generalizes to K-phase illusions ("A" → "B" → "C") and to alternative representations including B-splines, colored strokes, and general vector graphics.

Main Findings

• Joint optimization beats sequential generation. A sequential baseline that optimizes prefix strokes first and freezes them produces rigid Phase 1 structures that conflict with the final object (e.g., a duck's beak that cannot be repurposed). Joint optimization lets features serve both interpretations (e.g., the beak doubling as the cow's ear).

• The Overlay Loss is necessary for clean illusions. Without it, semantic guidance alone produces redundant strokes stacked on existing ones, causing visual clutter and high intersection artifacts; with it, delta strokes become spatially complementary to the prefix.

• Quantitative superiority across metrics. The method achieves 100% coverage versus Nano Banana Pro's 34.9%. Its top row (Metric-ranking) reports a Phase 1 CLIP score of 30.044, structural concealment of 3.282 (CLIP), 1.237 (IR), and 0.029 (HPS), and semantic concealment of 0.980. The GPT-ranking row reports 29.873, 1.668, 0.839, 0.023, and 0.983 respectively.

• Vector and raster baselines fail in characteristic ways. CLIPasso attains a high Phase-1 CLIP score (32.213) via direct image conditioning but has low concealment (1.690), and ControlSketch shows negative concealment scores (-2.378 CLIP, -0.789 IR, -0.018 HPS). SketchDreamer produces clutter, SketchAgent yields overly abstract results, and Nano Banana Pro relies on destructive editing.

• Our prefixes help other methods — but not enough. When baselines extend a fixed Phase 1 sketch produced by this method, they perform better than when they generate Phase 1 themselves, indicating the optimized prefix strokes embed structural cues for the second concept. Their results still lag behind the full joint-optimization pipeline.

• User studies favor the method. Across two user studies with 143 participants, participants selected this method in 67.7% of GPT-ranking and 87.1% of Metric-ranking cases. The paper reports over 98% overall satisfaction rates for the ranking pipeline in the main text (the Figure 9 caption states a success rate above 97%).

• Stroke budget scales with concept complexity. Simple transformations (e.g., rabbit-to-horse) succeed with 8–16 strokes, while complex subjects like Einstein require 32–64 strokes. The default is 16 prefix strokes and 32 total strokes.

• Initialization matters. Scattered stroke initialization fails to aggregate strokes and produces disconnected artifacts; centered and shifted gathered configurations both work, showing local stroke density outweighs absolute position. Centered gathered initialization is adopted to avoid boundary clipping.

• Stated limitation. The method inherits limitations from pre-trained diffusion priors; weak SDS guidance for complex structures (e.g., "scissors") causes optimization failure.

Methodology in Plain English

The authors represent a sketch as a set of learnable Bézier strokes and split them into two groups: prefix strokes (the first k, e.g., "a rabbit") and delta strokes (the rest, e.g., what turns it into "a horse"). A differentiable rasterizer turns these strokes into images.

The optimization runs two parallel branches. The top branch renders only the prefix strokes and compares them against the first concept using a frozen text-to-image diffusion model via Score Distillation Sampling (prefix SDS loss). The bottom branch renders the full set of strokes and compares them against the second concept (full SDS loss). The total SDS gradient is the sum of both, so prefix strokes receive pressure from both targets at once — that is the mechanism that finds geometry valid for both interpretations.

An Overlay Loss renders the prefix and delta subsets separately, blurs each with a Gaussian to create a soft spatial buffer, and penalizes their normalized pixel overlap. The final objective is the SDS loss plus this overlay term weighted by λ_overlay = 0.1.

For candidate selection, GPT-4o scores outputs on four dimensions (phase recognizability, single-object integrity, illusion quality, and sketch quality), where illusion quality specifically checks that the full sketch is significantly more recognizable than the delta strokes alone. Metric-based ranking combines CLIP, ImageReward, and HPS-derived scores that reward the prefix-target terms and penalize the delta-only term.

Implementation: Stable Diffusion v1.5 provides SDS guidance, training runs on an NVIDIA RTX 4090 for 2,000 iterations with the Adam optimizer and guidance scale 100, taking approximately 13 minutes for two-phase and 15 minutes for three-phase illusions. Evaluation uses a dataset of 64 common objects, and the method extends to K phases by partitioning strokes into disjoint subsets S₁ … S_K and optimizing all cumulative renderings in parallel.

Why This Matters

Impact on research: The work reframes visual illusions as a temporal, additive problem rather than a symmetric spatial transform like flip, rotate, or reproject. It also challenges the greedy "commit each stroke to one target" assumption that dominates sequential sketch generation, showing that strokes must be "primed" for future semantics.

Real-world applications cited by the authors:

• Creative education: Progressive illusions act as spatial reasoning exercises that foster Gestalt perception.
• Brand and logo design: Animated illusions bridge two brand identities in a single vector asset, useful for mergers or motion graphics.
• Physical media steganography: Outputs support thermochromic printing — for example, a mug shows one concept at rest and heat reveals the transformation.
• Dynamic visual storytelling: Native vector output enables arbitrary-resolution rendering and frame-by-frame animation for interactive media.
• Cognitive science: The framework generates controlled stimuli for studying temporal semantic perception with calibrated structural overlap.

Industry relevance: Because results are native vector graphics (resolution-independent, editable SVG-style assets), the approach fits animation pipelines, logo systems, and print/interactive media far better than pixel-based illusion methods. The multi-phase extension suggests a route toward stroke-by-stroke animated reveals from a single asset.

Future Directions

• Overcoming diffusion-prior failures. The authors note that weak SDS guidance for complex structures such as "scissors" causes optimization failure, leaving room for better priors or structure-aware guidance.
• Extending beyond the demonstrated formats. The paper shows preliminary generalization to B-splines, colored strokes, and general vector graphics; scaling color and width variation into the main illusion pipeline is an open direction.
• Longer and more ambitious sequences. K-phase illusions are demonstrated for small K (e.g., apple → sheep → Einstein, apple → rabbit → pig); how far the cumulative-optimization approach scales in phase count and semantic distance is unresolved.
• Reducing generation cost and improving automation. Each two-phase result takes roughly 13 minutes with multi-candidate filtering and GPT-4o ranking, so faster optimization and more reliable automatic candidate selection would make the tool practical for interactive or production use.

Target Audience

Graphics and vision researchers working on generative sketching, vector graphics synthesis, and diffusion-based optimization; SIGGRAPH-adjacent practitioners in animation, logo design, and creative tooling; and cognitive scientists interested in temporal semantic perception and Gestalt closure. The paper is most useful to readers already comfortable with SDS, differentiable rasterization, and CLIP/diffusion evaluation metrics, since the core contribution is a joint-optimization formulation rather than a conceptual primer.

Authors’ abstract

Visual illusions traditionally rely on spatial manipulations such as multi-view consistency. In this work, we introduce Progressive Semantic Illusions, a novel vector sketching task where a single sketch undergoes a dramatic semantic transformation through the sequential addition of strokes. We present Stroke of Surprise, a generative framework that optimizes vector strokes to satisfy distinct semantic interpretations at different drawing stages. The core challenge lies in the "dual-constraint": initial prefix strokes must form a coherent object (e.g., a duck) while simultaneously serving as the structural foundation for a second concept (e.g., a sheep) upon adding delta strokes. To address this, we propose a sequence-aware joint optimization framework driven by a dual-branch Score Distillation Sampling (SDS) mechanism. Unlike sequential approaches that freeze the initial state, our method dynamically adjusts prefix strokes to discover a "common structural subspace" valid for both targets. Furthermore, we introduce a novel Overlay Loss that enforces spatial complementarity, ensuring structural integration rather than occlusion. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines in recognizability and illusion strength, successfully expanding visual anagrams from the spatial to the temporal dimension. Project page: https://stroke-of-surprise.github.io/