No Pixel Left Behind: Filling Gaps in Anime Colorization

AI summary

Overview

• Research area: Human-Computer Interaction (HCI) / Creativity Support Tools, intersecting with applied computer vision and professional animation production workflows. Published at the 2026 CHI Conference on Human Factors in Computing Systems (Barcelona, Spain, April 13–17, 2026); arXiv:2609.00800v1 [cs.HC].
• Technical level: Intermediate. The paper combines a domain-specific deep learning method (a U-Net-based prediction pipeline) with an interactive tool and a professional user study, but explains both the model and the interaction design in accessible terms.
• Scope (one sentence): The paper presents GapFill, a tool that detects small unpainted regions left behind by the paint bucket in anime line-art colorization, suggests fill colors using a deep-learning region-correspondence model, and lets professional colorists inspect, correct, or batch-apply those suggestions.

What This Paper Is About

In anime production, colorists fill flat colors into clean line art. The widely used paint bucket (flood fill) tool frequently leaves small enclosed regions — called "gaps" — unpainted, typically at sharp corners such as hair tips or between fingers, and these gaps are hard to spot visually. Because professional pipelines treat even a single unpainted pixel as a defect requiring a costly retake, colorists spend significant time manually detecting gaps, zooming in, and choosing colors. GapFill aims to reduce that repetitive detection–zooming–color-selection cycle while keeping the human in control of the final result.

Key Contributions

1. A formative study bridging anime production and research: a two-stage study (questionnaire S1 with 20 professional colorists, R1–R20, and semi-structured interviews S2 with 4 experienced professionals, I1–I4, plus a screen recording V1 from the studio) that documents real colorization workflows and identifies unpainted gaps as an overlooked challenge.
2. GapFill, a specialized gap-filling tool designed to integrate into existing professional pipelines, with automatic gap detection and circular highlights, temporary AI-based color fills, hover-activated 5× pop-up magnification, in-circle color-pick correction, and sweep-to-apply plus an Apply-All button.
3. A domain-specific deep learning method that predicts colors for unpainted regions by leveraging the flat-color nature of anime images. Rather than regressing color values directly, it predicts a spatial likelihood map identifying the neighboring region most likely to share the target region's color, and infers the color from that region correspondence.
4. An evaluation with 13 professional colorists comparing GapFill against conventional tools on two tasks (coloring line art from scratch, and detecting/filling gaps in partially colored images), measured by completion time, number of overlooked gaps, surveys, semi-structured interviews, and feature-level analyses.

Main Findings

• Gaps are a widespread, high-impact problem: In S1, small unpainted gaps were the most frequently cited difficulty, voluntarily mentioned by 11 out of 20 participants. 65% reported often needing to address unpainted gaps (M=4.8, SD=1.7), and 85% agreed that deciding the appropriate color for gaps is important for completing an animation (M=6.0, SD=1.2).
• Gap filling is time-consuming: 60% agreed that addressing unpainted gaps is time-consuming (M=5.0, SD=1.8). One interviewee (I3) described detecting gaps by zooming in very close and moving the canvas bit by bit, taking up to several minutes per frame in the worst case for complex drawings.
• Paint bucket dominates the workflow: Among the 24 participants referred to across S1 and S2, all used Clip Studio Paint (CSP). The Paint Bucket was most frequently used (90%), followed by brush tools (50%) including the Leftover Pen, AI-based auto-coloring tools (40%), and lasso-like tools (20%) such as Enclose and Fill.
• Input modality diversity: In S1, input devices were pen tablets (80%) and pen displays (70%), with 35% additionally using a mouse and keyboard as auxiliary inputs.
• Current AI auto-coloring underperforms in practice: The studio's internal AI tool transfers colors from adjacent frames and achieves 70–80% accuracy for minor movements, but frequently misrecognizes regions in dynamic scenes. Interviewees said they often have to check its work and would rather paint manually to avoid doing it twice, though one expressed willingness to adopt better-performing tools.
• Coloring decisions rely on local context: In S1, 90% of participants referenced surrounding colors when deciding gap colors, 75% used official references such as model sheets, and 45% checked adjacent frames.
• Color prediction accuracy: The proposed method achieved 81.68% accuracy on an unseen dataset, and its outputs were subjectively rated as valuable in production contexts.
• GapFill improved performance and usability: In the study with 13 professional colorists, the system improved performance and usability in gap-filling tasks over conventional methods, with significant improvements particularly in the task of detecting and filling gaps in partially colored images.
• Accuracy alone does not drive usability: The study suggested that prediction accuracy is not the primary factor for usability; appropriate colors can be contextually ambiguous, and participants also valued clear visual aids and controllability of AI suggestions. Adoption of AI-powered assistance depended on users' trust.
• Complementary rather than replacing: GapFill can complement existing tools depending on users' trust in new AI-powered assistance.
• Per-frame time varies widely: According to I1 and I4, colorization time per frame ranges from under a minute to 15–30 minutes depending on complexity. One colorist described a batch strategy of coloring the same part across multiple frames before switching colors.

Note: The provided paper content is truncated at Section 5.1.1 (Participants), so the detailed quantitative user-study results tables and per-task statistics are not available in the supplied text.

Methodology in Plain English

The researchers first studied how anime colorists actually work. They ran an online questionnaire with 20 professional colorists and then 30-minute semi-structured interviews with 4 experienced professionals at the same studio, plus obtained a screen recording of a real colorization session. This revealed that paint bucket filling leaves small unpainted gaps, that finding them is done manually (often using the "Black Light Method," which temporarily renders all colors black so transparent gaps appear as bright dots), and that colorists decide gap colors mainly by looking at surrounding colors.

Based on these findings, the team built GapFill as an on-demand toggle inside the standard painting workflow. When activated, it detects enclosed transparent regions below a user-adjustable pixel threshold using BFS-style traversal, marks each with a circular highlight, and temporarily fills each with a suggested color. Hovering over a highlight shows a fixed 5× magnified pop-up for inspection without zooming the canvas. Dragging inside a highlight activates a color-picker-like mode so the user can correct the suggestion without zooming, and a dotted line shows which gap is being changed. Dragging outside the highlights sweeps across multiple circles to apply suggestions in batch, and an Apply-All button fills everything at once.

For color prediction, the team avoided predicting color values directly. Instead, they trained a compact U-Net to solve a region-correspondence problem: given a two-channel binary input of a line-art mask and an unpainted-gap mask, the model outputs a likelihood map indicating how likely each pixel in a local patch belongs to the same color region as the target gap. The suggested color is then taken from the painted region with the highest average predicted likelihood.

Training data was generated synthetically. Line drawings were segmented into enclosed regions via BFS-based fill, regions with pixel counts below a threshold of 10 were treated as potential gaps, and for each of the resulting 1,807,977 targets from complete professional anime episodes, a 32×32 pixel patch centered on the gap was extracted. Rotation and flipping augmentation was applied. The model learned to map the line-art mask and gap mask to the closest large (above the pixel threshold) same-color region mask computed from the ground-truth colored image.

The tool was then evaluated with 13 professional colorists recruited from the same anime studio, using a custom painting software that kept the full colorization environment identical between conditions except for the gap-handling tools: the Baseline condition included conventional tools (Leftover Pen, Enclose and Fill, and the Black Light Method toggle), while the Ours condition replaced them with a single GapFill mode toggle.

Why This Matters

This work brings a specific, practical, and previously understudied production problem into HCI research. Most automatic line-art colorization research assumes fully colorized reference images or user hints, and most inpainting research targets natural photos with continuous tones rather than flat, discrete anime regions. GapFill instead targets tiny regions under production constraints where a single unpainted pixel is a defect, and it contributes a lightweight region-correspondence formulation suited to flat-color images. It also adds empirical evidence about how creative professionals weigh trust, controllability, and contextual ambiguity when deciding whether to adopt AI assistance.

Real-world applications:

• Anime and cel-animation colorization (shiage/finishing): reducing retakes and labor cost by catching and filling gaps before frames move to the composition stage.
• Digital illustration and comics: the same detection, magnification, correction, and batch-apply interactions apply to flat coloring of line art in Clip Studio Paint-style tools.
• Other flat-color digital painting pipelines: workflows with separated line, shadow-guide, and highlight-guide layers, where gaps form from unintentional line intersections.
• Creativity-support tooling more broadly: the design pattern of showing discrete suggestions as highlights, allowing per-item correction, and permitting batch acceptance is transferable to other human-in-the-loop AI suggestion systems.

Industry relevance: the anime industry broadcasts over 200 titles per year, and colorization remains largely manual, with per-frame work ranging from under a minute to 15–30 minutes. Even a single unpainted pixel can trigger a costly retake. GapFill was designed from interviews with a commercial studio, tested with 13 professionals, and is intended to fit into existing pipelines rather than replace them. The paper's code is available at https://marc2825.github.io/GapFill.

Future Directions

• Trust and adoption in professional settings: The study found that usability was not driven by prediction accuracy alone and that adoption depends on users' trust in AI-powered assistance. How to design for and measure that trust remains open.
• Handling contextually ambiguous colors: With 90% of colorists referencing surrounding colors but 75% also consulting model sheets and 45% checking adjacent frames, cases where the correct color cannot be inferred from local context alone are unresolved.
• Accuracy improvement and generalization: The method achieved 81.68% accuracy on an unseen dataset. Whether it generalizes across studios, differing guide-color conventions (noted to vary, with green used in some studios), and more complex drawings is an open question.
• Better region estimation: The paper notes that methods such as trapped-ball segmentation could be used to more strictly estimate enclosed regions by accounting for line discontinuities, suggesting a path to more robust gap detection.
• Integration with existing AI tools: Given that the studio's internal frame-transfer auto-coloring tool reaches 70–80% accuracy for minor movements but struggles in dynamic scenes, how GapFill should combine with such tools is left open.

Target Audience

• HCI and creativity-support-tool researchers interested in professional creative workflows and human–AI collaboration.
• Computer graphics and computer vision researchers working on line-art colorization, region correspondence, or domain-specific inpainting for flat-color images.
• Animation and illustration production professionals, technical directors, and tool developers evaluating AI assistance for finishing/colorization pipelines.
• Interaction designers building suggestion-and-correction interfaces where users must inspect, adjust, or batch-accept automated outputs.

Authors’ abstract

Animation production workflows often involve digital colorization of line art, where small unpainted regions ("gaps") frequently occur and remain an underexplored challenge. We conducted a formative study in Japanese animation (anime) pipelines and found that while the paint bucket tool is widely used for base coloring, tiny enclosed areas are frequently overlooked, resulting in time-consuming manual detection and filling. We introduce GapFill, a tool grounded in professional practices that reduces the effort of gap detection, zooming, and color selection. Our deep-learning method suggests appropriate fill colors by referencing surrounding regions, leveraging the flat-color nature of anime-style images. In a user study with 13 professional colorists, our system improved performance and usability in gap-filling tasks over conventional methods. The study also suggested that prediction accuracy alone is not the primary factor for usability, that appropriate colors can be contextually ambiguous, and that GapFill can complement existing tools depending on users' trust in new AI-powered assistance.