Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation

AI summary

Overview

Research area: Generative computer vision and graphics — sketch-conditioned 3D human motion synthesis, specifically multi-human human–object–human (HOH) interaction.

Technical level: Advanced. The paper assumes familiarity with diffusion models, rectified/conditional flow matching, energy-based models, latent VQ-VAE representations, and continuous-time Markov chains.

Scope: The paper introduces a system that converts storyboard-style 2D sketches into controllable, physically plausible 3D motion for multiple people co-manipulating shared objects, evaluated on the CORE4D and InterHuman benchmarks.

What This Paper Is About

Existing text-conditioned motion generators (such as COLLAGE) hand plans to the model but under-specify timing, phasing, and spatial layout, while existing sketch-driven methods (such as Sketch2Anim) handle only a single human with no inter-agent coordination or object contacts. Sketch2Colab closes that gap by treating storyboard keyframes, joint trajectories, and object masks as the control channel for multi-human, object-centric interaction. The goal is motion that follows the artist's sketch precisely — correct keyframes, timings, contacts, and collision-free coordination between people and objects — while sampling faster than diffusion-only approaches.

Key Contributions

1. Diffusion-to-flow distillation. A rectified-flow student is distilled from a strong sketch-conditioned diffusion teacher, so the system retains the teacher's fidelity but gains faster, more stable sampling under complex multi-entity conditioning.

2. Energy-guided rectified flows. Differentiable energies are defined for keyframe pose fidelity, joint/endpoint trajectory tracking, inter-agent and object distances, contact consistency, collision avoidance, foot-skating, ground-plane constraints, and temporal smoothness. These shape the sampling trajectory directly rather than relying on slow posterior guidance loops.

3. A CTMC-based phase planner. A lightweight continuous-time Markov chain over three composite interaction states — {approach or withdraw, contact or release, hold or carry} — proposes contact and handoff schedules, and is coupled to the continuous flow through occupancy-weighted phase mixing and contact-energy gating.

4. Demonstrated multi-entity performance. The method is reported to outperform evaluated baselines on constraint adherence and perceptual quality for multi-human, object-centric tasks, while sampling substantially faster than diffusion-only alternatives.

Main Findings

• Sketch control works for multi-entity scenes. Figure 1 shows generated human–object–human demonstrations in which two people move a table along a sketched path and one lets go, two people carry a box along a prescribed path while adjusting its height, and a specified hand grasps a canister and follows a complex path — all from sparse storyboard keyframes with no text.

• The flow student beats the diffusion teacher visually. Figure 3 (left) reports that given the same storyboard keyframes and joint trajectories, Sketch2Colab follows the sketches more faithfully and matches interaction timing better than the sketch-conditioned COLLAGE Teacher, which often misses storyboard constraints including the handoff, the continued one-person carry, and a required higher lift.

• The CTMC planner produces interpretable phase transitions. In the reported carry scenario, occupancy moves into "hold or carry" during joint transport, briefly returns to "contact or release" when one carrier disengages, and shifts back to "hold or carry" for the single-person carry before final release.

• Hard-case behavior is studied qualitatively. Figure 3 (right) shows difficult cases with heavy sketch noise (approximately 60%, referenced to Table 1(b)), self-intersecting paths, and sparse constraints that would otherwise cause drift, floating, or collisions.

• Inference speed advantage claimed, but no timing numbers are given in the available content. The abstract and contributions state sampling is substantially faster than diffusion-only alternatives; the provided text does not report specific runtime, latency, or speedup figures.

• Quantitative benchmark numbers are not present in the supplied content. The paper states it evaluates on CORE4D and the InterHuman benchmark configuration, but the tables reporting constraint-adherence and perceptual-quality metrics were truncated and are therefore not summarized here.

Methodology in Plain English

The system takes a storyboard as input: a few sketched keyposes per person, optional 2D polylines showing where joints should travel, and rough object masks or silhouettes. An optional text prompt can be added, but it is not required.

Rather than training a generative model from scratch, the authors first train a diffusion "teacher" model built on the COLLAGE latent backbone and decoder, then freeze it. A second model — a "student" using rectified flow, a formulation that moves samples along straighter paths — is trained to imitate the teacher's flow field. Training the student from scratch for this kind of multi-agent scene would be expensive and unstable; distilling from the teacher avoids that.

Both models work in a compressed latent space produced by a hierarchical VQ-VAE, where each entity (each human, each object) is a token. The backbone is a 4-level temporal U-Net with an entity-graph attention block whose attention logits are biased by the physical distances between entity tokens, so nearby agents and objects share information.

To actually obey the sketch, the authors add "energies" — penalty terms that are differentiable and can be differentiated through to latent space. These penalize keyframe misalignment, trajectory deviation, wrong contact margins, too-close spacing, foot skating, leaving the ground plane, and jerky motion. Because guidance is defined on decoded 3D motion but sampling happens in latent space, they learn a low-rank block-Toeplitz Jacobian surrogate to route gradients back to the latent stably. A second set of latent-space energies keeps the sample near the sketch-conditioned manifold, so raw-space precision and latent-space coherence are combined.

Because interaction has discrete events — grabbing, releasing, handing off — which a continuous flow handles poorly, a small CTMC tracks which interaction phase is active and mixes three phase-specific velocity sub-fields according to the current occupancy. The CTMC is fit using a Kolmogorov-residual loss. A learned potential trained by energy-equilibrium matching captures residual preferences the hand-designed energies miss, and a Lyapunov-style loss pushes the student's velocity toward descending directions of the total potential.

Training proceeds in two phases: first the flow and distillation losses, then the full objective with Lyapunov, CTMC, energy surrogate, and latent losses activated. At inference, an ODE is integrated for 30 Heun steps on CORE4D and 60 on InterHuman, with CTMC updates every 3 steps, followed by an optional 3-step micro-Langevin refinement. The final latent is decoded by the frozen COLLAGE decoder into 3D motion.

Why This Matters

Impact on research. The paper offers a template for combining three usually separate ideas — flow distillation, energy-based guidance, and a discrete-state planner — in a single motion generator. It also reframes multi-agent interaction as a hybrid continuous/discrete problem, where smooth pose transport and abrupt contact events are handled by different machinery. This is a meaningful departure from the dominant pattern of adding task-specific control modules or posterior guidance to a diffusion model.

Real-world applications.

• Game development: populating scenes with coordinated NPC interactions from quick storyboard sketches instead of hand-authored animation.
• Film and previsualization: turning director's keyframe sketches into rough multi-character blocking with correct timing and contact.
• VR/AR authoring: rapid prototyping of cooperative object-manipulation scenarios without motion capture.
• Robotics and simulation: generating reference trajectories for two-agent co-manipulation of shared objects.

Industry relevance. Animation and game studios currently pay for long manual cleanup of jitter, collisions, and timing errors in multi-character scenes, or invest in motion-capture pipelines that do not scale across layouts. A sketch-driven generator that respects contacts and timing, and that samples faster than diffusion-only alternatives, targets exactly that bottleneck — and the speed claim matters for interactive authoring tools.

Future Directions

• Quantify the speed advantage. The paper claims substantially faster sampling than diffusion-only alternatives but the provided content contains no runtime numbers; a careful wall-clock comparison across hardware and sequence lengths is an obvious next step.
• Scale the state space. The main experiments use only three composite interaction states. The authors mention ablating coarser and finer granularities in the supplement, but how far the CTMC approach scales to richer interaction vocabularies (multi-object handoffs, tool use, larger groups) is open.
• Reduce reliance on synthesized sketches. The storyboard controls are largely generated by projecting 3D joints and adding perturbations, supplemented by 682 hand-drawn sketches over CORE4D. Robustness to real artist sketches — and to the roughly 60% noise regime shown in hard cases — needs broader study.
• Generalize beyond the two benchmarks. Evaluation is limited to CORE4D (998 sequences) and a 6,022-motion InterHuman configuration. Whether the energy and CTMC machinery transfers to other multi-human interaction datasets, more objects, or dynamic scenes is untested here.

Target Audience

Researchers and graduate students in generative motion synthesis, computer graphics, and human–computer interaction who are working on controllable character animation; graphics and game engineers interested in sketch-driven authoring tools for multi-character scenes; and readers already comfortable with diffusion models and flow matching who want to understand how energy guidance and discrete-state planners can be composed for multi-entity control.

Authors’ abstract

We present Sketch2Colab, which turns storyboard-style 2D sketches into coherent, object-aware 3D multi-human motion with fine-grained control over agents, joints, timing, and contacts. Conventional diffusion-based motion generators have advanced realism; however, achieving precise adherence to rich interaction constraints typically demands extensive training and/or costly posterior guidance, and performance can degrade under strong multi-entity conditioning. Sketch2Colab instead first learns a sketch-driven diffusion prior and then distills it into an efficient rectified-flow student operating in latent space for fast, stable sampling. Differentiable energies over keyframes, trajectories, and physics-based constraints directly shape the student's transport field, steering samples toward motions that faithfully satisfy the storyboard while remaining physically plausible. To capture coordinated interaction, we augment the continuous flow with a continuous-time Markov chain (CTMC) planner that schedules discrete events such as touches, grasps, and handoffs, modulating the dynamics to produce crisp, well-phased human-object-human collaborations. Experiments on CORE4D and InterHuman show that Sketch2Colab achieves state-of-the-art constraint adherence and perceptual quality while offering significantly faster inference than diffusion-only baselines.