Panel-by-Panel Souls: A Performative Workflow for Expressive Faces in AI-Assisted Manga Creation

AI summary

Overview

Research area: Human-Computer Interaction, with a focus on human-AI co-creation tools for creative work — specifically AI-assisted manga (Japanese-style comic) production.

Technical level: Intermediate. The paper is a systems and workflow paper that builds on existing pre-trained models (insightface, LivePortrait, DALL·E 3) rather than proposing new generative architectures, and it reports a qualitative, author-led case study rather than quantitative benchmarks.

Scope: The paper presents and analyzes a three-stage interactive pipeline for preparing faces, mapping nuanced expressions, and recompositing them into AI-generated manga panels.

What This Paper Is About

Text-to-image models such as Stable Diffusion and DALL·E can produce visually polished manga-style art, but they cannot reliably render the subtle, coordinated facial expressions that sequential storytelling depends on — a "knowing glance," a flicker of suspicion, or a smile tinged with regret. Artists are left to redraw faces manually panel by panel, which is slow and risks inconsistent character appearance, while existing AI tools (facial reenactment software, node-based editors like ComfyUI) do not fit a manga artist's layer-based drawing workflow.

The paper's goal is not a new generative model, but a new interactive workflow that lets artists specify expressions performatively and precisely, then hand the result back to the artist for final polishing.

Key Contributions

1. A dual-hybrid, three-stage pipeline for expressive faces in manga panels: (1) Automated Face Preparation, (2) Interactive Expression Mapping, and (3) Layered Composition and Refinement, designed to be repeated across multiple characters and panels.

2. A hybrid face preparation stage combining landmark-based auto-detection — using a state-of-the-art face analysis model from the insightface library that detects 106 facial landmarks — with a manual framing mode in which the artist draws a bounding box directly on the source image.

3. A hybrid expression mapping stage that uses the pre-trained LivePortrait engine for performative video input (live webcam or uploaded video), with an interactive timeline for scrubbing to a keyframe, followed by numerical sliders for fine-grained eye and lip retargeting.

4. An author-led end-to-end case study on a multi-character manga draft generated with neutral expressions using DALL·E 3, documenting practical findings and limitations, supported by an expert interview with a professional manga artist.

Main Findings

• Auto-detection works for primary faces but fails at scale and complexity: The landmark-based detector successfully identified and framed the primary male characters in the case-study panel, but failed on a small, distant face (highlighted in red in Figure 1) and required manual override for characters with complex hairstyles or accessories.

• Generous crops outperform tight crops for reenactment: Including a character's hair in the crop worked better for the subsequent reenactment stage than cropping tightly on facial features alone, because it provides more contextual information to the model and improves perceived coherency between face and hairstyle.

• A temporal offset exists between performance and result: The most aesthetically pleasing or narratively correct expression on the manga character often appeared a few frames before or after the artist's seemingly best driving-video frame. This validates the interactive timeline slider over a one-shot "perform-and-generate" interaction.

• Direct composition leaves predictable artifacts: Pasting the 512x512 reenacted face back at its original coordinates and size produces visible seam artifacts, including geometric misalignments at the neck and hair and minor hue and lighting shifts. The authors intentionally hand this composited draft to the artist rather than automating the fixup.

• Artifacts inherited from LivePortrait persist regardless of motion mode: Using either relative or absolute motion modes, the authors observed hair and ears remaining static during head rotation, and photorealistic features such as lips and teeth being introduced into a monochrome aesthetic.

• Head pose sensitivity: Faces turned more than 45 degrees away from the camera produced less reliable expression mapping.

• Expert concern about stylistic homogenization: In an expert interview, a professional manga artist noted that widespread digital tool adoption (for example, pen tablets with line smoothing) has already led to a perceived loss of the individual "pen stroke." The authors draw a parallel between this and the "expressive homogenization" of generic AI-generated faces.

• No quantitative evaluation is reported: The case study is described as formative and expert evaluation, performed by the authors, who have formal training in fine arts (visual communication design, painting) and human-computer interaction. The paper does not report benchmark numbers, timing measurements, or a formal user study.

Methodology in Plain English

Using an AI model, the artist first produces a manga panel with neutral character expressions. The system then walks through three stages for each character face.

Stage 1 — Face preparation. The system runs an automatic face detector that finds 106 facial landmarks (eyes, brows, nose, mouth, facial contour) and builds a tight, padded bounding box from them. This excludes non-facial elements like hands and torsos better than a general object detector would. If the auto-detector fails — distant faces, complex hairstyles, face-obscuring accessories — the artist can draw a square frame manually, and that box is treated exactly like an automatically generated one.

Stage 2 — Expression mapping. The artist performs the desired expression on a webcam or uploads a pre-recorded video reference. The LivePortrait engine transfers that performance onto the prepared manga face. The artist scrubs through the video on a timeline and picks the frame that produces the best result, then adjusts dedicated eye and lip sliders to correct things that are hard to perform, such as gaze direction independently of head pose, or lip curvature matching a character's established design.

Stage 3 — Composition. The reenacted face is resized back to its original dimensions and pasted at its original coordinates in the panel. Because this leaves visible seam artifacts, the authors deliberately stop here and hand the draft to the artist, who uses familiar tools such as a smudge or airbrush brush in their own software to polish the result.

The authors then ran the whole pipeline end to end on a DALL·E 3-generated multi-character, multi-panel draft, using written character prompts for an older mentor, a young trainee, and an observing peer, and recorded where the pipeline succeeded and where it broke down.

Why This Matters

Impact on research: The paper argues that the bottleneck in creative AI is not image quality but the interface between artistic intent and model execution. Its central technical insight — the temporal offset between an artist's best performance frame and the model's best output frame — reframes expression control as an interactive exploration problem rather than a one-shot generation problem. It also positions the work as a counter-movement to expressive homogenization, and treats the remaining seam artifacts as a known post-processing problem deliberately left to the artist rather than hidden.

Real-world applications:

• Manga and comic production, where an artist needs to shift a scene's emotional tone across many existing panels without redrawing every face.
• Character consistency management across sequential panels, using the same prepared face crops and expression controls.
• Storyboarding and pre-visualization for animation or film, where rapid changes in a character's emotional state must be iterated quickly.
• AI-assisted drafting workflows in layer-based drawing software such as Clip Studio Paint, where the system acts as an assistant rather than a replacement.

Industry relevance: The paper is explicitly aimed at professional manga artists who work under tight deadlines in layer-based drawing applications rather than visual programming environments. It positions itself against node-graph interfaces like ComfyUI and against facial reenactment tools built for video production, arguing that neither matches the iterative, panel-by-panel nature of manga work. The expert interview reinforcing the homogenization concern gives the work grounding in professional practice.

Future Directions

1. Integrate 3D-aware models to fix geometric inconsistencies — such as static hair and ears during head rotation — potentially using ControlNet-like mechanisms for pose control, which would let artists manipulate entire head orientation and posture, not just facial muscles.

2. Fine-tune the reenactment model on a curated dataset of manga-style art, or incorporate style-preserving techniques, to prevent photorealistic lips and teeth from breaking a stylized monochrome aesthetic.

3. Conduct a formal user study with professional manga artists, moving beyond the author-led evaluation, to quantitatively assess workflow efficiency and gather qualitative ratings of expressive control. The temporal offset and the necessity of the manual framing tool are proposed as primary hypotheses and design probes for that study.

4. Extend beyond human-like faces. The current landmark-based auto-detector is not designed to handle the full range of non-human characters common in manga, leaving this as an open problem. Automated seam-blending is mentioned as a possible future improvement to composition, though it was intentionally treated as out of scope.

Target Audience

This paper is most useful to HCI researchers working on human-AI co-creation and creative tooling; developers building artist-facing interfaces on top of generative and facial reenactment models; and manga artists, illustrators, and comics professionals curious about where AI can realistically fit into a panel-by-panel workflow. It is also relevant to anyone studying the trade-off between automation and creative control, and to researchers interested in qualitative, expert-evaluation methods for creative AI systems. Readers looking for new generative architectures, quantitative benchmarks, or statistical user-study results will not find them here.

Authors’ abstract

Current text-to-image models struggle to render the nuanced facial expressions required for compelling manga narratives, largely due to the ambiguity of language itself. To bridge this gap, we introduce an interactive system built on a novel, dual-hybrid pipeline. The first stage combines landmark-based auto-detection with a manual framing tool for robust, artist-centric face preparation. The second stage maps expressions using the LivePortrait engine, blending intuitive performative input from video for fine-grained control. Our case study analysis suggests that this integrated workflow can streamline the creative process and effectively translate narrative intent into visual expression. This work presents a practical model for human-AI co-creation, offering artists a more direct and intuitive means of ``infusing souls'' into their characters. Our primary contribution is not a new generative model, but a novel, interactive workflow that bridges the gap between artistic intent and AI execution.