OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

AI summary

Overview

Research area: Computer vision and generative modeling, specifically the synthesis of vector animations from multi-modal instructions (text, images, and video) using vision-language models.

Technical level: Advanced. The paper assumes familiarity with autoregressive sequence modeling, vision-language model backbones, vector graphics formats (Lottie, SVG), and standard generative evaluation metrics (FVD, CLIP, PSNR, SSIM, DINO).

Scope: The paper introduces OmniLottie, an end-to-end autoregressive framework that generates editable Lottie vector animations from text, text-image, or video instructions, supported by a new Lottie tokenizer, a two-million-sample dataset (MMLottie-2M), and a benchmark (MMLottie-Bench).

What This Paper Is About

Vector animations are lightweight, resolution-independent, and easily editable, but generating them automatically from human instructions has been difficult because the Lottie JSON format is a deeply nested, verbose structure full of invariant formatting metadata. Prior approaches either decouple appearance from motion (applying priors from reference GIFs or text-to-video models to static vector art) or ask large language models to emit raw Lottie JSON directly, which yields very low success rates. This paper's goal is to make Lottie generation robust and controllable by serializing Lottie files into compact command-and-parameter token sequences, then training a pretrained vision-language model to produce those tokens from multi-modal prompts.

Key Contributions

1. OmniLottie framework: The authors describe it as the first end-to-end framework capable of generating vector animations directly from multi-modal instructions, handling Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie tasks under a single autoregressive model.
2. MMLottie-2M dataset: A large-scale multi-modal dataset of two million Lottie animations with paired text descriptions, reference images, and video demonstrations, collected from platforms including LottieFiles, IconScout, Flaticon, Iconfont, and Icons8, plus SVG-derived animations built from the OmniSVG collection. It has been publicly released.
3. Lottie tokenizer: A parameterization scheme that converts raw Lottie JSON into concise sequences of shape, effect, and animation commands with their control parameters, improving both training efficiency and generation quality relative to direct JSON learning.
4. MMLottie-Bench and evaluation results: A standardized benchmark with a Real Subset (450 samples from artist-designed animations, disjoint from training) and a Synthetic Subset, along with experiments showing state-of-the-art performance in visual fidelity and semantic alignment.

Main Findings

• Higher success rates than direct JSON generation: On the Real Subset Text-to-Lottie task, OmniLottie reports an 88.3% success rate versus 12.7% for GPT-5, 9.3% for DeepSeekV3, 0.0% for Qwen2.5-VL (3B), and 77.3% for the commercial tool Recraft.
• Best visual quality on Text-to-Lottie: OmniLottie achieves an FVD of 202.14, compared with 671.80 (DeepSeekV3), 715.73 (GPT-5), and 300.70 (Recraft).
• Strong motion alignment: OmniLottie scores 5.94 on Motion Alignment for Real Subset Text-to-Lottie, higher than Recraft (4.68), DeepSeekV3 (2.09), and GPT-5 (0.71); its Object Alignment of 4.44 is slightly below Recraft's 4.70 but above the other baselines.
• Robust on Text-Image-to-Lottie: OmniLottie records the best FVD (180.27), Object Alignment (5.10), and Motion Alignment (4.44) on the Real Subset, with a 93.3% success rate, versus 28.0% (GPT-5), 87.3% (AniClipart), 91.3% (Livesketch), and 0.0% (Qwen2.5-VL 3B).
• Substantially faster than optimization-based baselines: For Text-Image-to-Lottie on the Real Subset, OmniLottie averages 88.57 seconds per sample, while AniClipart takes 1212.34 seconds and Livesketch 723.23 seconds. The paper notes these methods also cannot output native vector formats.
• Best video reconstruction fidelity: On the Real Subset Video-to-Lottie task, OmniLottie reports FVD 227.11, PSNR 16.08, SSIM 0.82, and DINO 0.92, with an 88.1% success rate; GPT-5 reaches 9.2% success and Gemini3.1-Pro 0.0%.
• Richer token sequences: OmniLottie produces longer generated Lottie token sequences than all baselines (for example, 36.8k tokens on Real Subset Video-to-Lottie versus 1.1k for GPT-5 and 1.0k for Gemini3.1-Pro), which the authors associate with more expressive and detailed animation output.
• Moderate SVG mixing is optimal: The data-composition ablation finds the best results with 30% SVG data mixed with Lottie. Pure Lottie yields Text-to-Lottie FVD 305.57, Object Alignment 1.85, and Motion Alignment 4.92; an even 50/50 split gives FVD 285.22, Object Alignment 4.12, Motion Alignment 3.38; pure SVG gives FVD 342.61, Object Alignment 4.02, Motion Alignment 2.35; the 70% Lottie / 30% SVG mix gives FVD 269.50, Object Alignment 4.31, and Motion Alignment 5.63.
• Tokenizer ablation confirms the design choice: Training with the tokenizer yields Text-to-Lottie success rate 97.3%, FVD 269.50, CLIP 0.2748, Object Alignment 4.31, and Motion Alignment 5.63, compared with 13.4% success and FVD 459.39 when fine-tuned on raw Lottie JSON, and 0.00% success for the pretrained Qwen2.5-VL. Similar gains are reported for Text-Image-to-Lottie (92.0% versus 15.9% success) and Video-to-Lottie (90.7% versus 10.2% success).
• Failure modes are characterized: OmniLottie failure rates are reported as 2.7% for Text-to-Lottie, 8.0% for Text-Image-to-Lottie, and 9.3% for Video-to-Lottie. About 35% of failures are structural (valid JSON with an empty layers array) and about 65% are rendering-level, subdivided into missing style attributes (~40%), temporal visibility errors (~25%), opacity/scale value errors (~25%), and off-canvas positioning (~10%).
• Baseline failure patterns differ: Qwen2.5-VL reaches 0.0% success across all tasks due to schema hallucination, generating attributes such as "version": "2.0" or "frames" instead of Lottie's expected keys. DeepSeek generates structurally valid JSON but places attributes in incorrect nodes, such as putting color attributes directly in rectangle shape nodes rather than in separate fill nodes, and frequently emits text layers without corresponding font definitions.

Methodology in Plain English

The authors start from the observation that asking a model to write Lottie JSON directly wastes most of its capacity on reproducing boilerplate. Their approach has three parts.

First, they reorganize and tokenize Lottie. The file is split into core metadata (version, frame rate, in/out points, width, height, name, 3D flag, layers) and conditional fields (assets, markers, fonts, characters) that appear only when needed. Each layer is classified as one of five types: Precomposition (ty=0), Solid (ty=1), Null (ty=3), Shape (ty=4), or Text (ty=5). The tokenizer walks the layer tree, emitting command tokens such as layer-type markers and parameter tokens. Continuous values are mapped to discrete tokens with token(x,t) = floor(x * s_t) + o_t, where each parameter type gets its own scale factor and vocabulary offset so different kinds of values do not collide. Text fields are handled by the pretrained text tokenizer, and the decoder inverts the process to rebuild the command sequence and then the full Lottie. The authors describe this as a lossless compression of the JSON into a tree-structured representation.

Second, they build and annotate data. They normalize all animations to a 512×512 canvas and a 0–16 timestamp range, remove non-parameterizable elements (base64 image layers, audio, camera layers, After Effects expressions), and discard files that cannot be parameterized. To compensate for limited native animations, they animate static SVGs from the OmniSVG collection with predefined motions, and they analyze 1M native Lottie files to extract motion signatures (rotation, scale, position, opacity trajectories across keyframes), cluster them into canonical motion templates, and inject those templates into SVG-derived files. Annotations are generated with a vision-language model using a coarse-to-fine strategy: a brief overall caption first, then finer temporal details using cues like "begins with" and "then."

Third, they train a VLM. They start from Qwen2.5-VL, add a randomly initialized set of Lottie vocabulary embeddings, and train with a standard cross-entropy objective to predict each Lottie token given the previous tokens and the interleaved multi-modal instruction.

For evaluation, MMLottie-Bench measures results on two axes: visual quality (FVD; PSNR, SSIM, and DINO for video tasks) and semantic alignment (CLIP similarity, plus Object Alignment and Motion Alignment scored 0–10 by Claude-3.5-Sonnet as an LLM judge). Efficiency is measured by average token length of the generated Lottie JSON (using the Qwen2.5-VL tokenizer) and average generation time per sample.

Why This Matters

Impact on research: The paper reframes vector animation generation as a sequence-modeling problem over a purpose-built discrete vocabulary rather than a JSON text-generation problem. It also provides the field with a two-million-sample dataset and a benchmark containing a 450-sample Real Subset drawn from professional designer work and held out from training, plus a Synthetic Subset designed to reduce contamination risk from future models. These resources give other groups a shared training and evaluation base in an area that previously lacked one.

Real-world applications:

• UI/UX and web design, where lightweight, resolution-independent animations are already standard and where a designer could describe an animation in text or supply a reference image and receive editable output.
• Motion graphics and branding, producing logos, icons, and promotional animations that remain editable in professional tools rather than being locked into raster video.
• Digital publishing and advertising, where compact vector animations reduce file size and render cleanly across screen resolutions and platforms.
• Iterative design workflows, where the model can consume keyframe images or reference videos and produce first-draft animations for a human artist to refine.

Industry relevance: Lottie is widely adopted across modern design workflows, and commercial tools such as LottieFiles Motion Copilot already automate parts of keyframe creation. The paper positions its approach against both commercial generators (Recraft) and general-purpose LLMs (GPT-5, DeepSeek, Gemini), showing that specialized tokenization and training beat general-purpose models on success rate and motion fidelity. The runtime comparison is also practically meaningful: OmniLottie's 88.57 seconds per Text-Image-to-Lottie sample versus 1212.34 seconds for AniClipart and 723.23 seconds for Livesketch speaks directly to whether a tool is usable interactively.

Future Directions

• Constrained decoding: The authors note that autoregressive decoding can still produce invalid sequences and suggest constrained decoding as a way to guarantee structurally valid output.
• Reinforcement learning with renderability rewards: Training with a reward tied to whether the generated animation renders correctly is proposed as a path to greater reliability.
• Agentic integration with professional tools: The paper suggests integrating the model with tools such as After Effects to improve practical applicability.
• Generalization and context length: The authors identify generalization to diverse scenarios and limited context length as ongoing challenges that constrain handling of complex animations.

Target Audience

This paper is most useful to researchers working on generative models for graphics—particularly those studying vector graphics synthesis, autoregressive visual generation, and multi-modal instruction following. It is also relevant to practitioners building design automation tools who need to understand the trade-offs between direct JSON generation and structured tokenization, and to dataset and benchmark builders interested in how the authors constructed MMLottie-2M and the Real and Synthetic Subsets of MMLottie-Bench. Readers

Authors’ abstract

OmniLottie is a versatile framework that generates high quality vector animations from multi-modal instructions. For flexible motion and visual content control, we focus on Lottie, a light weight JSON formatting for both shapes and animation behaviors representation. However, the raw Lottie JSON files contain extensive invariant structural metadata and formatting tokens, posing significant challenges for learning vector animation generation. Therefore, we introduce a well designed Lottie tokenizer that transforms JSON files into structured sequences of commands and parameters representing shapes, animation functions and control parameters. Such tokenizer enables us to build OmniLottie upon pretrained vision language models to follow multi-modal interleaved instructions and generate high quality vector animations. To further advance research in vector animation generation, we curate MMLottie-2M, a large scale dataset of professionally designed vector animations paired with textual and visual annotations. With extensive experiments, we validate that OmniLottie can produce vivid and semantically aligned vector animations that adhere closely to multi modal human instructions.