Color3D: Controllable and Consistent 3D Colorization with Personalized Colorizer

AI summary

Overview

Research area: Computer vision — 3D scene reconstruction and colorization, covering both static radiance fields (NeRF / 3D Gaussian Splatting) and dynamic scene representations (4D Gaussian Splatting).

Technical level: Advanced. The paper assumes familiarity with Gaussian Splatting, spherical harmonics, CIE Lab color space, diffusion-based generative augmentation, and adapter-based parameter-efficient fine-tuning.

Scope (one sentence): Color3D is a framework that colorizes monochromatic static and dynamic 3D scenes by colorizing a single key view with any off-the-shelf 2D colorizer and then fine-tuning a per-scene personalized colorizer to propagate that color consistently to all other views and time steps.

What This Paper Is About

Reconstructing a scene in 3D normally requires color photographs, so a scene captured only in grayscale cannot be turned into a colorful 3D model. The obvious fix — running a 2D colorization model on every frame or view and then reconstructing — produces severe color shifts between views, because 2D colorizers are not designed to be consistent across viewpoints. Color3D instead colorizes only one carefully chosen key view and then fine-tunes a "personalized colorizer" for that specific scene to carry the same colors to every remaining view and video frame, so the complex 3D colorization problem becomes a much simpler color-propagation problem.

Key Contributions

1. A unified controllable 3D colorization framework (Color3D) that handles both static and dynamic 3D scenes, enabling user-guided colorization by tuning a personalized colorizer for each individual scene.
2. A customized key view selection strategy plus a single view augmentation scheme that together improve coloring richness and generalization when fine-tuning the personalized colorizer from only one colored image.
3. A Lab Gaussian representation for Gaussian Splatting that decouples luminance and chrominance into separately optimized spherical-harmonic coefficient sets, improving color reconstruction fidelity and preserving scene structure.
4. Demonstrated superiority across benchmarks, with the authors reporting more vibrant, controllable, and consistent results than alternatives on static and dynamic 3D colorization datasets.

Main Findings

• Static scenes (DL3DV-140, LLFF, Mip-NeRF 360). Under automatic colorization, Color3D reaches FID 37.48 / Colorful 32.65 / ME 0.084 / TC 0.017 on DL3DV-140 (140 static 3D scenes), versus FID 63.56 / Colorful 28.15 / ME 0.146 / TC 0.038 for 3DGS+ImageColorizer and FID 77.89 / Colorful 22.38 / ME 0.128 / TC 0.031 for 3DGS+VideoColorizer. On LLFF, Color3D attains FID 35.10 / Colorful 33.99 / ME 0.056 / TC 0.007 under automatic setting, and FID 62.84 / CLIP Score 0.6544 / ME 0.051 / TC 0.009 under language-guided control. On Mip-NeRF 360, Color3D reports FID 68.23 / CLIP 0.6246 / ME 0.058 / TC 0.012 for language-guided colorization and FID 48.62 / Ref-LPIPS 0.6823 / ME 0.079 / TC 0.015 for reference-based colorization.

• Dynamic scenes (DyNeRF, HyperNeRF). On DyNeRF, Color3D reports FID 58.62 / CLIP 0.6271 / ME 0.041 / TC 0.007 for language-guided colorization, compared with FID 112.35 / CLIP 0.6082 / ME 0.062 / TC 0.011 for Instruct 4D-to-4D, and FID 89.39 / ME 0.124 / TC 0.025 for 4DGS+ImageColorizer. On HyperNeRF, Color3D reports FID 63.28 / CLIP 0.6257 / ME 0.045 / TC 0.008 under language guidance and FID 58.63 / Ref-LPIPS 0.6610 / ME 0.055 / TC 0.008 under reference-based control. The authors state that ME is reduced by 0.83 and 0.70 for language-guided and reference-based settings respectively.

• Averaging-based consistency costs color. The paper finds that methods such as ColorNeRF, which enforce multi-view consistency by averaging inconsistent 2D predictions, improve consistency but degrade FID and task-specific metrics and sacrifice controllability; Color3D reports that it preserves vividness while still being consistent.

• Ablation on Mip-NeRF 360 (language-guided). Starting from a baseline of FID 115.66 / CLIP 0.6145 / ME 0.078 / TC 0.018, adding Key View Selection gives FID 100.28 (delta +15.38) / CLIP 0.6175 (+0.0030) / ME 0.074 / TC 0.017; adding Single View Augmentation gives FID 85.25 (+15.03) / CLIP 0.6196 (+0.0021) / ME 0.069 / TC 0.016; adding the Fine-Tuning-Based Colorizer gives FID 75.48 (+9.77) / CLIP 0.6225 (+0.0029) / ME 0.064 / TC 0.013; adding Lab Gaussian gives the final FID 68.23 (+7.25) / CLIP 0.6246 (+0.0021) / ME 0.058 / TC 0.012.

• Feature-level consistency. In a PCA-based diagnostic comparing two camera viewpoints, features for the same 3D content (flower clusters, pillars, stone paths) remained consistent and stable across viewpoints, which the authors present as evidence for why the personalized colorizer produces view-consistent colors.

• Real-world applicability. The method is also applied to in-the-wild monochrome multi-view images and historical monochrome video (old movies), producing colorizations the authors describe as realistic and consistent across viewpoints and time.

Methodology in Plain English

Stage 1 — train a personalized colorizer from one image.

• Pick the best key view. Features are extracted from all monochrome candidate views with CLIP, L2-normalized, and compared via pairwise cosine similarity. Each view's similarity distribution is turned into an entropy value (Eq. 1); the view with maximum entropy (Eq. 2) is chosen because it relates most evenly to all others and thus encapsulates the broadest visual information.
• Colorize that one view using any off-the-shelf colorization model. The paper evaluates ControlColor for language-guided, DDColor for automatic, and UniColor for reference-based colorization, with ColorMNet used as a video colorization reference.
• Augment the single view. Generative augmentations include outpainting (image split into a 2×2 grid, each region extended with Stable Diffusion, with an LLaVA-generated caption as the text prompt), image-to-video with Stable Video Diffusion to simulate appearing objects and motion, and novel views from Stable Virtual Camera along a predefined orbital trajectory. Traditional augmentations are then applied: rotation, flip, grid shuffle, and elastic transform. Images are converted to CIE Lab, with L as input and ab as the target.
• Fine-tune the colorizer. The architecture is a frozen DDColor (ConvNeXt) encoder plus trainable adapters and a lightweight CNN decoder initialized from scratch. The pre-trained decoder is deliberately avoided because its built-in color priors are a major source of multi-view inconsistency. Training uses an L1 loss between predicted and target ab channels (Eq. 3).

Stage 2 — colorize and reconstruct the 3D scene. The personalized colorizer predicts consistent chrominance for all remaining views or frames, which are used to optimize the scene with Lab Gaussian Splatting built on 3DGS (static) and 4DGS (dynamic).

• Lab Gaussian: the three spherical-harmonic coefficient sets are reformulated from RGB to {SH_L, SH_a, SH_b} so luminance and chrominance are parameterized separately. Rendered channels are normalized to [0,1] — L divided by 100, and a and b shifted by 128 and divided by 255 (Eq. 4).
• Losses: the L channel uses the 3DGS L1 and D-SSIM terms plus an edge loss based on the Laplacian operator (Eq. 5, Eq. 6), with β = 0.2; the ab channels use only L1 and D-SSIM (Eq. 7), since chromatic information is low-frequency.
• Warm-up: during the first half of training iterations all three SH sets represent L and render a three-channel luminance image; in the second half two sets are reassigned to a and b for color modeling.

Implementation. PyTorch on NVIDIA RTX A6000 GPUs; Structure-from-Motion initializes Gaussian points from monochrome inputs; all settings other than the Lab Gaussian representation and the new objectives are kept identical to original 3DGS/4DGS; the personalized colorizer tuning adds about eight minutes. Metrics used are FID, CLIP score (language), Colorful Score (automatic), Ref-LPIPS (reference), plus the authors' proposed Matching Error (ME), which uses a dense image matching model to find pixel correspondences and measure average color difference, and TC, an average of short-term and long-term optical-flow-based consistency scores.

Why This Matters

The work reframes a hard 3D problem — consistent, controllable colorization of an entire scene — as a tractable single-image task, and does so for dynamic scenes, which the authors state were previously unaddressed in controllable 3D colorization. Because only one view must be controlled, users can steer an entire 3D scene or video through one image, which is a simpler and more predictable interaction than per-view prompting. It also means arbitrary 2D colorization models, including language-guided and reference-based ones, can be plugged in without modification.

Real-world applications:

• Digital art and artistic creation, where an artist colors one key frame and the rest of the 3D scene follows.
• Cultural heritage preservation and legacy restoration, demonstrated here on in-the-wild monochrome multi-view images and historical monochrome video.
• Archival film and old-movie revitalization, using the dynamic-scene path.
• Monochrome capture pipelines (for example, grayscale camera footage or historical photography) being upgraded to colorful, view-consistent 3D content.

Industry relevance: the framework is built on top of widely used open representations (3DGS and 4DGS) and off-the-shelf colorization models, so the added cost is a per-scene fine-tuning step that the authors report as roughly eight minutes — a practical increment for content production, virtual production, and immersive media.

Future Directions

• Extending beyond the studied scene categories. The evaluation covers DL3DV-140, LLFF, Mip-NeRF 360, DyNeRF, and HyperNeRF; whether the per-scene personalized colorizer generalizes to other capture conditions is not established here.
• Reducing the per-scene tuning cost. Each scene requires its own fine-tuned colorizer (about eight additional minutes), so amortizing or sharing colorizers across scenes is an open question.
• Loosening the reliance on a single key view. The approach depends on one colorized reference view and a heuristic entropy-based selection; behavior when no view is representative, or when the key view's colorization is poor, is not analyzed in the provided content.
• Broadening controllability modes. Only language-guided, automatic, and reference-based control are demonstrated; other interaction styles for steering the single key view remain unexplored.

Target Audience

Researchers and graduate students in computer vision and graphics working on novel view synthesis, radiance fields, Gaussian Splatting, or image and video colorization. It is also relevant to practitioners in VFX, digital art, film restoration, and cultural heritage digitization who need controllable, view-consistent colorization of 3D or video content, and to readers specifically interested in parameter-efficient per-scene fine-tuning and Lab-space scene representations.

Authors’ abstract

In this work, we present Color3D, a highly adaptable framework for colorizing both static and dynamic 3D scenes from monochromatic inputs, delivering visually diverse and chromatically vibrant reconstructions with flexible user-guided control. In contrast to existing methods that focus solely on static scenarios and enforce multi-view consistency by averaging color variations which inevitably sacrifice both chromatic richness and controllability, our approach is able to preserve color diversity and steerability while ensuring cross-view and cross-time consistency. In particular, the core insight of our method is to colorize only a single key view and then fine-tune a personalized colorizer to propagate its color to novel views and time steps. Through personalization, the colorizer learns a scene-specific deterministic color mapping underlying the reference view, enabling it to consistently project corresponding colors to the content in novel views and video frames via its inherent inductive bias. Once trained, the personalized colorizer can be applied to infer consistent chrominance for all other images, enabling direct reconstruction of colorful 3D scenes with a dedicated Lab color space Gaussian splatting representation. The proposed framework ingeniously recasts complicated 3D colorization as a more tractable single image paradigm, allowing seamless integration of arbitrary image colorization models with enhanced flexibility and controllability. Extensive experiments across diverse static and dynamic 3D colorization benchmarks substantiate that our method can deliver more consistent and chromatically rich renderings with precise user control. Project Page https://yecongwan.github.io/Color3D/.