Manga2Animate: An Agentic Framework for Faithful Manga Animation with Script and Storyboard Planning

AI summary

Overview

Research area: Computer Vision — generative video, manga/comic understanding, and multi-agent (LLM-orchestrated) media production pipelines.

Technical level: Advanced. The paper assumes familiarity with diffusion-based image and video generators, vision-language models, agentic creator–reviewer loops, and fidelity/consistency benchmarking (VBench++, ViStoryBench).

Scope (one sentence): The paper introduces Manga2Animate, a multi-agent framework that converts existing manga pages into multi-shot animation by first building and iteratively reviewing an animation script and a visual storyboard, then using those as a faithful plan for an off-the-shelf video generator.

What This Paper Is About

Turning a manga into animation by hand takes large teams months of storyboarding, key animation, in-betweening, and coloring, even though the manga already fixes the characters, framing, and plot. Existing automatic approaches fail in two opposite ways: text-based storytelling systems describe the manga in words and lose its art style and character identity, while feeding manga pages straight into a video model produces smooth motion but the wrong story, because the model does not know panel reading order or what speech balloons mean. Manga2Animate's goal is to give today's video generators the one thing they lack — a correct understanding of the manga — so their output stays faithful to the source's plot, characters, and art style across shots.

Key Contributions

1. A framework for faithful manga-to-animation. The authors present Manga2Animate, described as the first automated framework that turns existing manga into animation faithful to the source manga, with an explicit panel → script clip → storyboard panel → video clip correspondence maintained throughout.

2. Planning through two intermediate representations with dual-level review. The task is decomposed into an animation script (textual specification: scene, camera, character positions and actions, expressions, dialogue) and a visual storyboard (appearance and scene composition per shot). Creator agents produce candidates; paired reviewer agents check them against the original manga panels, with representation-specific selection (clip-wise for scripts, whole-image for closed-source storyboards) and feedback that flows from storyboard-level issues back into the script.

3. An evaluation protocol for the task. MangaEval adapts ViStoryBench and VBench++ metrics to manga input and multi-shot output, covering storyboard Visual Consistency, Content Alignment, Aesthetic Quality and Cost, plus video Manga Alignment, Video Quality, Aesthetic Quality and Cost.

4. Empirical and human validation. The framework outperforms storyboard and video baselines even when those baselines receive the same refined script and character references, works with both open-source and commercial backends, and is preferred by users in a 37-participant study.

Main Findings

• Better manga fidelity than video baselines. With Gemini for planning and Seedance 2.0 for video, Manga2Animate reaches I2VS 0.791, I2VB 0.768, SC 0.931, BC 0.958, Shadow-v2 0.518 and Waifu-v3 5.090, versus AniMaker (0.775, 0.749, 0.922, 0.950, 0.410, 2.694), ViMax with Seedance 2.0 (0.753, 0.747, 0.927, 0.948, 0.385, 4.764) and StoryMem (0.728, 0.731, 0.885, 0.922, 0.275, 3.602).

• The gain is not tied to one model combination. The FLUX.2 + MiniMax-H3 configuration also performs well, with the highest I2VS in Table 1 at 0.794 and the highest Shadow-v2 at 0.549, while Gemini + MiniMax-H3 gives the highest I2VB at 0.791. Because the framework guides existing models rather than replacing them, it works with open-source and commercial generators.

• Higher motion scores do not mean correct motion. Direct manga-to-video with Seedance 2.0 and MiniMax-H3 posts the highest Dynamic Degree (13.941 and 11.787) and fast runtimes (3m29s and 5m54s), but the paper argues DD only measures how much movement exists, not whether it is the right movement — such outputs may animate the page layout or show manga panels as video frames.

• Storyboard fidelity improves under the same script. On storyboard metrics, the FLUX.2 backend gives the best CSD (0.622) and OCCM (73.5), and the Gemini backend gives the best CIDS (0.808) and PA (3.50), each beating the best baseline. StoryDiffusion (Shadow-v2 0.697, Waifu-v3 5.789) and Story-Iter (0.615, 6.308) score higher on anime-specific aesthetics but considerably lower on fidelity (CIDS, PA, OCCM), showing that single-image aesthetics and adaptation fidelity are distinct.

• Every reviewer matters. Removing the Script, Storyboard, or Video Reviewer drops the full model's I2VS from 0.791 to 0.766, 0.783, and 0.768 respectively, and lowers alignment and consistency overall. Removing the Script Reviewer causes the largest I2VS drop; some ablated variants score higher on MS or Shadow-v2 while scoring lower on alignment, consistency, or DD.

• Clip-wise script selection beats whole-script selection. Composing the best clip description per panel raises Waifu-v3 from 4.423 to 5.090, Shadow-v2 from 0.473 to 0.518, and DD from 5.951 to 9.457, with only small decreases in I2VB (0.768 vs. 0.772) and MS (0.960 vs. 0.963) — both gaps within 0.004.

• Script-level and storyboard-level feedback are complementary. Withholding either lowers CSD (0.605 to 0.597 without script feedback, 0.588 without storyboard feedback), CIDS (0.808 to 0.786 and 0.770), and PA (3.50 to 3.33 and 3.39); removing storyboard feedback also lowers both aesthetic metrics more than removing script feedback.

• Strong human preference. Thirty-seven participants judged eight storyboard and eight video cases with method names hidden and outputs randomized; Manga2Animate achieved the highest preference on every storyboard criterion (Character Identity, Manga Plot, Absence of AI Hallucination, Overall Performance) and every video criterion (the same four plus Visual Comfort and Consistency), and was selected as best overall in over 90% of responses for both storyboards and videos.

• Cost profile. For each story, the closed-source pipeline makes 34.5 calls on average in the script and storyboard stages (28.5 Gemini text calls, 6 image-generation calls) and 3 in the video stage, costing about US$5.65 per story, of which US$4.55 is Seedance. End-to-end wall-clock time for the main setting is 24m49s, versus 8h9m for AniMaker and 2h16m8s for StoryMem.

Methodology in Plain English

The design mirrors how a real animation studio works, with AI agents playing the roles of staff and supervisors.

Pre-processing. Pages are first broken into panels, ordered into the correct reading sequence, and OCR'd using MAGI, a manga-specific detection model. The text is merged into a single transcript. A Character Description Creator then writes down stable, identity-defining traits (facial features, body build) while deliberately excluding variable things like clothing and accessories, so downstream agents share one consistent notion of each character. Reference images come from official color artwork when available, otherwise from clean crops of the manga. Each input set is six ordered panels.

Two intermediate representations. Instead of handing raw pages to a video model, the pipeline builds a script (one clip per panel, in structured JSON) and a storyboard (one image per clip). Everything keeps an explicit one-to-one-to-one link back to the source panel, so any generated element can be traced and checked.

Creator–reviewer loops for two rounds. In round one, the Script Creator proposes 3 candidate scripts; the Script Reviewer selects compositionally — picking the best character and clip description at each panel position across candidates and reassembling them, because scripting errors are local and different candidates make different mistakes. The remaining weaknesses become feedback. The Storyboard Creator then proposes 3 storyboards from the selected script; the Storyboard Reviewer compares candidates pairwise, and in later rounds compares each new candidate against the previous best, judging each storyboard against the script that produced it. A storyboard that does not win leaves the previous best intact, keeping the final script–storyboard pair consistent. Closed-source storyboards are selected as whole images because their panels share style, color, and lighting, so recombining panels would introduce visual inconsistency; the open-source FLUX.2 backend instead generates and scores panels individually before compositing a 3×2 grid.

Dual-level feedback. Script-level and storyboard-level feedback go jointly to the next-round Script Creator along with the best script and the source panels. Storyboard problems are treated as symptoms of ambiguous or incomplete script text, so the Script Creator is told to locate the responsible line (an ambiguous spatial description, a missing detail) and revise only what reviewers flagged, leaving everything else untouched. When reviewer feedback conflicts, the source panel decides.

Video generation. Two modes are supported. In reference-conditioned mode, the full best script is the text prompt and the whole storyboard grid is the reference image, and 2 complete multi-shot candidates are generated in one pass for the Video Reviewer to choose from. In first-frame mode, each storyboard panel becomes the first frame of its clip, 2 candidates are generated per clip, and the Video Reviewer picks greedily clip by clip using the script, the panel, and previously selected clips, which enforces cross-clip continuity at selection time. Main results use reference-conditioned mode.

Evaluation. Eight stories of six ordered panels, drawn from Manga109 plus additional cases, are run against storyboard baselines (StoryDiffusion, Story-Iter, AnyStory) and video baselines (AniMaker, ViMax, StoryMem), plus direct manga-to-video with Seedance 2.0 and MiniMax-H3. Every script-based baseline receives the same refined script, and character references are supplied wherever a baseline accepts them, so the comparison isolates generation quality from manga understanding.

Why This Matters

Impact on research. The paper reframes manga animation as a grounding problem rather than a raw generation problem: today's video models already animate well, they just cannot read a manga. It also shows that consistency can come from reviewed intermediate state rather than from model architecture, and it contributes a reusable evaluation protocol (MangaEval) plus evidence that automatic fidelity metrics and human preference agree in direction on plot and character identity.

Real-world applications:

• Studio pre-visualization. Animators could turn a manga chapter into a rough animatic — script, storyboard, and timed shots — before committing artist hours to key animation.
• Fan and independent adaptation. Small creators and fan communities could produce faithful motion adaptations of titles that would never receive a full studio budget.
• Localization and publishing. Publishers could generate animated previews, trailers, or motion-comic editions of catalog titles while preserving the original art style.
• Accessibility and archive work. Motion versions with dialogue and audio can serve readers who struggle with dense page layouts, and could help animate archival material from collections such as Manga109.

Industry relevance. The pipeline treats image and video generators as interchangeable backends, demonstrated with Gemini 3.1 Flash Image, FLUX.2-dev, Seedance 2.0, MiniMax-H3, and Index-AniSora V3.2. That means gains from future video models can be absorbed without redesigning the system — relevant to studios and platforms that want to upgrade models without rebuilding their production tooling. The reported cost of roughly US$5.65 per six-panel story also makes the economics of automated pre-production concrete.

Future Directions

• Robustness to difficult art styles. The authors name complex or roughly sketched art as an open problem; highly sketchy, cluttered, or compositionally complex pages can still be misinterpreted, so extending the pipeline to rougher sources is a natural next step.

• Source-grounded evaluation of story events. Current metrics measure appearance, subject, and background fidelity and content adherence, but not whether specific story events are correctly rendered — the authors call for evaluation that grounds narrative events in the source.

• Longer-range narrative understanding. The pipeline works over a small set of ordered panels; extending understanding across pages and chapters, including scenes that depend on context from elsewhere in the story, is explicitly listed as future work.

• Reducing stochastic and policy fragility. Since the pipeline is agentic and built on stochastic models, outputs vary across runs, and API-based models may refuse copyrighted or policy-sensitive content — an important open question for any production deployment.

Target Audience

Researchers and practitioners in generative video, comics/manga understanding, and multi-agent systems; animation and pre-production tool developers evaluating automated storyboarding; and benchmark designers interested in fidelity-oriented evaluation of story-to-visual generation. The paper is most useful to readers comfortable with diffusion video models and LLM agent pipelines, though the high-level framing — plan first, review against the source, then generate — is accessible to animation technologists and studio R&D leads.

Authors’ abstract

Adapting existing manga into coherent animation that stays faithful to the source is a valuable yet challenging task. Existing methods either access the manga mainly through text, losing its characters and art style, or feed manga pages directly into video generation models, which misinterpret the manga's structure and produce incorrect storylines. In this paper, we propose Manga2Animate, a multi-agent framework that grounds every stage of generation in the source manga. We decompose the task into two intermediate representations, an animation script and a visual storyboard. Each representation is produced by a creator agent and reviewed against the source panels by a paired reviewer agent. We tailor representation-specific selection and dual-level feedback to progressively refine both before synthesizing the final animation. Extensive experiments and a user study demonstrate that Manga2Animate produces animations more faithful to the source manga and more consistent across shots than state-of-the-art approaches. Project page: https://annchenn.github.io/Manga2Animate/